Chunyan Song

Check out the latest in our "Debuggability" blog series! This blog is authored by Kevin Downey Jason Johl and me and talks about how we built a Debuggability Paved Road at Intuit using Ephemeral Containers and Argo Workflows. We'd love to hear your thoughts on the topic. Also, what tools do you build as-a-platform to help debug service, compute and network issues in Kubernetes at scale? #intuit #kubernetes #debug

23

Hey everyone, I am an investor and advisor to Katanemo.... They are reinventing the infrastructure primitives for GenAI and launched Arch - https://lnkd.in/gCrbbkEV - an intelligent prompt gateway. Give the project a look on Github, and star the project if it resonates

88

3 Comments

I just realized: we should start saying "build memoization" rather than "build caching." We aren't really talking about least-recently-used or time-to-live algorithms, we're talking about "don't bother recompiling this piece if none of its inputs changed." (It has been puzzling me that some people are so deeply fearful of "build cache" as a concept, and honestly I think it's mostly because most build systems have non-hermetic state scattered all over the place. Maybe a slightly more precise term for the type of caching we're concerned with would clarify the semantics in question.)

83

9 Comments

Project update! My application tracking extension is now doing fuzzier matching and injecting new information into LinkedIn (and other sites) to keep me from doubling up on applications. Next up: local storage and bulk import/export operations...

12

2 Comments

🍜Anybody up for some ramen? I am excited to share that I am back in the garage, building again (it’s actually a house in the Berkeley Hills). We are taking the wraps off of Supercog AI, a new startup focused on solving Application Integration using LLM powered agents. My co-founder in this new venture is a great friend of mine (and a former co-founder partner), Alex Osborne. Alex and I have worked around the application integration space for a long time, and we are unrealistically excited about applying the GenAI stack to this problem. Not many people relish the task to get some bits to move from system A to system B. But it’s a critical job to be done, and that’s led to a lot of brittle scripts, a myriad of inflexible “no code” tools, and a lot of hand crafted SQL. But the power of the Large Language Model, trained on a huge corpus of information, offers a remarkable tool to solve this problem. The LLM you use today already knows the APIs of hundreds of popular systems. It knows the SQL dialects of every major database. It understands the semantics and data schemas of many popular SaaS systems. It understands much of the specific domain in which your business operates. I like to call the AI revolution “the last platform shift”. That may turn out to be hyperbole, but I definitely believe we are in the very early stages of seeing what this new stack can enable. I know we are hardly the first folks to claim that “amazing stuff is coming!”. But this is why Alex and I have decided to focus on a real and hard problem. If we can prove that LLMs can power a new way to solve this problem, it will be the first successful *new* approach in 15 years.. We will have much more to share about this new platform soon. In the meantime, if you’re interested in getting a peek - or even better if you have application or data integration tasks that you would like some help with - please reach out because we want to help. #genai #startups #backinthegarage

164

37 Comments

Want a preview of what I'm going to be talking about in my keynote for RubyConf? You can catch a glimpse of it here!

10

LLMs are excellent for that first draft and refactoring tasks that used to take hours of tedious effort. David Crawshaw captures their utility perfectly in his latest essay on programming with LLMs. https://lnkd.in/e5ZVYs9u

10

1 Comment

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

I am excited to share the latest blog, co-authored by myself, Junhwi Kim and Vyas Sekar, detailing how we managed to successfully localized and improved LLM by combining Fine-Tuning and RAG. https://lnkd.in/gKDkgGPq

70

3 Comments

I've dusted off my personal blog after years of neglect and published a new post: How to add natural language AI data filters to your app. Take a look! https://lnkd.in/gxT3NfRr

22

1 Comment

The afternoon Keynote @ Stripe Sessions was Fireside Chat between Patrick Collison (CEO of Stripe) and Jensen Huang (CEO of NVIDIA). Patrick was an excellent moderator — asked Jensen really probing, nuanced questions that Jensen answered with his usual sharp wit and unique (and impressively well rationalized) perspective. A couple of quotes among many that stuck with me: - “Current CPU installed market is $1 trillion of hardware, all of which will be upgraded to GPUs over the next 5 years. And I would estimate about another $1 trillion will be spent on top of that during the same time period as companies try to do more with AI. Add those together — and I think at least $2 trillion will be spent on GPUs in the next 5 years.” - “I like to go after Zero-Billion Dollar Markets — nonexistent markets that I envision & can rationalize well, but will be very hard, expensive & time-consuming to achieve.”

1

This is the final part of the 3 part series of posts that shows how to Evaluate your LLM based RAG application. In the last part what I would like to show us how to Evaluate the Answer (Actual Response) faired against the Contexts Provided after Retrieving it from a Vector Database. The only metric that is available for this is Faithfulness. Where the Claims made in the Answer (Actual Response from the LLM) are checked against the Claims present in the Contexts Provided to the LLM to answer the user query. The way the Claims are obtained is by asking the same LLM or another LLM to present all the Claims made in the Answer, and then get all the Claims made in the Contexts that was provided to the actual LLM to generate the Answer. Then calculate the ratio between the two. #llm #evaluation #metric #ai

4

In the AI ecosystem, LLMs act like CPUs, crucial yet dependent on complementary tools such as Chatbot and AnythingLLM, which function similarly to I/O systems. Alongside, tools like MemGPT and CrewAI serve roles akin to RAM and network interfaces. The challenge arises as these tools often evolve independently, aspiring to be standalone solutions. This siloed development leads to a lack of standardization, complicating their integration and limiting their utility in professional settings. Without unified protocols, integrating these diverse AI tools with LLMs demands significant customization, placing an undue burden on developers and hindering broader innovation. Just as the adoption of standardized interfaces like USB revolutionized computer usability, AI tools would benefit from similar standardization to enhance functionality and ease of use. Promoting common standards for AI tools is crucial for maximizing the capabilities of LLMs and ensuring their practical application across various domains. Without such efforts, the AI field may continue to struggle with fragmentation, undermining the potential of these advanced technologies.

3

2 Comments

Come join us tomorrow in SF to learn more about the latest updates to run AI workloads on Google Cloud!

11

1 Comment

Check out the latest episode of our Tech Teams Today podcast with Yongsheng Wu, VP of Engineering at Granica - now available on all major podcast platforms. #AI #EngineeringLeadership #DataScience #MachineLearning #TechTeamsToday #Revelo

4

I am incredibly excited to announce Arch gateway https://lnkd.in/ePAf-VK8, an open source project for developers to build genai apps faster. It is engineered with purpose-built LLMs, Arch handles the critical but undifferentiated tasks related to the handling and processing of prompts, including detecting and rejecting jailbreak attempts, intelligently calling “backend” APIs to fulfill the user’s request represented in a prompt, routing to and offering disaster recovery between upstream LLMs, and managing the observability of prompts and LLM interactions in a centralized way - all outside business logic. Here are some additional key details of the project, * Built on top of Envoy and is written in rust. It runs alongside application servers, and uses Envoy's proven HTTP management and scalability features to handle traffic related to prompts and LLMs. * Function calling for fast agentic and RAG apps. Engineered with purpose-built fast LLMs to handle fast, cost-effective, and accurate prompt-based tasks like function/API calling, and parameter extraction from prompts. * Prompt guardrails to prevent jailbreak attempts and ensure safe user interactions without writing a single line of code. * Manages LLM calls, offering smart retries, automatic cutover, and resilient upstream connections for continuous availability. * Uses the W3C Trace Context standard to enable complete request tracing across applications, ensuring compatibility with observability tools, and provides metrics to monitor latency, token usage, and error rates, helping optimize AI application performance. This is our first release, and would love to build alongside the community. We are just getting started on reinventing what we could do at the networking layer for prompts. Do check it out on GitHub at https://lnkd.in/ePAf-VK8 Please leave a comment or feedback here or in GH and I will be happy to answer. #gateway #envoy #rust #genai #developers #ai cc Salman Paracha | Muhammad Ikramul Haq | Syed Anas Hashmi | Fahim ul Haq | Farheen Omar | Umar Farooq | Arslan Ahmad | Abdul Rafay Abbasi |

60

9 Comments

The session unfortunately wasn't recorded, but here are the slides from my talk at LFMS on quantifying the ROI of contributing to OSS https://lnkd.in/gwdQa3UZ

66

5 Comments

Anthropic's latest Claude AI model is taking human-AI interaction to a whole new level! 🖥️🤖 The new Claude 3.5 Sonnet can now use computers just like a human would - by looking at the screen, moving the cursor, clicking buttons, and typing text. This is a major leap forward in AI capabilities. Key features: Operates based on screenshots, emulating human computer use Can perform complex multi-step tasks across different applications Self-corrects and retries when encountering obstacles2 Accessible via API for developers to build with Why it's game-changing: This opens up incredible possibilities for AI assistance in data entry, form filling, software testing, and automating repetitive computer tasks13. Imagine having an AI assistant that can seamlessly work alongside you, handling tedious computer work while you focus on higher-level tasks. While still in beta, this technology demonstrates how AI is rapidly advancing toward more human-like interaction with digital interfaces. As it develops further, it could revolutionize how we work with computers and boost productivity across industries. The future of human-AI collaboration just got a lot more exciting! What do you think about this new capability? #AIInnovation #FutureOfWork Here is a video about it: https://lnkd.in/gh8aNYEu

6