Storm6’s Post

What does the decreasing LLM latency curve mean for developers? Last week we added support for Groq in Freeplay (details in Ian’s Loom below 👀). After the initial shock of the lightning-fast inference times on Groq wore off, it got me thinking about what the rapidly decreasing latency curve means for developers. 👇 Here are some of the unlocks I’m seeing across our customer base and the industry more broadly as inference times drop. 1️⃣ Realtime Guardrails Model-graded evaluation has become a cornerstone of any team’s LLM evaluation toolkit. As these evals start to become more reliable, people naturally ask the question, “Why can’t I run them in real-time and prevent my customers from seeing a bad generation in the first place?” The answer to date has often been latency. As latency drops, developers can start thinking about how to strategically implement these real time guardrails without hindering the customer experience. 2️⃣ Large Scale Batch Processing Some of the most promising early LLM use cases I’ve seen are backend workloads that involve classifying and annotating large swaths of unstructured data. These tend to be large batch processes that need to run over large volumes of data. While there isn’t a customer waiting on the other end, the aggregate latency of LLMs to date have made the idea of processing 10s of millions of items infeasible. One customer we worked with at Freeplay had a task like this. While we were able to hit a solid quality and cost bar, the idea ultimately fizzled because it was going to take 100s of hours to process a day's worth of data. 🤯 Given what we see from Groq, that workload could be a few hours today using an equally powerful model. As latency drops these kinds of large async workloads become possible. This will be a boon for LLM adoption because often these kinds of tasks only require a model to be right on average. 3️⃣ Latency-Sensitive User Experiences While the intelligence of modern LLMs has been impressive for a while now, builders have been chronically underwhelmed by their speed. As latency drops, the opportunity space grows for how LLMs can be used throughout customer-facing software products. 4️⃣ Use Multiple Queries to Increase Quality LLMs are stochastic processes, meaning the same input is not guaranteed to produce the same output. This can be frustrating for developers, but it is also a necessary component of the model’s intelligence. One of the simplest ways to improve quality is to query an LLM multiple times and look for consensus across the outputs. This strategy has been functionally difficult to implement to date due to latency constraints. But as latency (and cost) drops, developers can start to think about where their LLM applications may benefit from this multiple query strategy. 😎That’s everything on my mind – what am I missing? What are you excited about with lower latency models? Shout out to Groq for getting our creative juices flowing!

Several people have requested more information about the Rust Readiness Assessments that I posted about a few days ago. Scroll through the pages of this deck to get an idea of how your team might benefit. p.s. I've updated the name to Rust Readiness Review

🚀 For the second class of Cohort 5, we're thrilled to have Steph Orpilla, DevRel Lead at Nillion! Steph will dive into what it takes to be the pioneering DevRel hire at a company, setting the stage for success. 🌟 Discover: 🛠️ Practical steps for bootstrapping a developer community 🤝 Launching successful collaborations with engineering teams 📚 Mastering the art of creating impactful documentation and robust examples from scratch ...and much more!

We hardly ever talk about it because it’s not “sexy.” If you are trying to grow your business, you have had a few flops. If you are trying to move into a leadership role, you have had your flops. If you are a CEO, constantly trying to keep the organization moving forward, you have seen your share of flops. I’ll tell you, the best learning experiences have been when I tried something, threw a whole bunch of energy into it, but it didn’t work for whatever reason. Some may call it failure, I called it a flop I can learn from. One of the best flops happened while being the CEO of Willis HR. My flop taught me that just because you see a hole in the market, a solution to the problem, doesn’t mean it’s the right thing for me or my business to focus on. I had to start asking myself, - Does this align with my long-term goals? - Is this something that I want myself or my team focused on? - What are the reasons why someone else hasn’t shown up to solve the problem? - Is now the right time to be spending time, money, and energy on this? By answering a few questions, I was able to stay focused and consistent in building a business model that was profitable and connected to my greater WHY.

Here's the story behind the First Place Housing Stabilization Hackathon team and their solution. Plans are in the works to see where this idea can go to keep families stable and housed. https://lnkd.in/gAFvsqjv

Exciting news indeed! Thank you Hack the North for this amazing experience, Aly Shariff and @Vaishvi Shah for the stellar performance on frontend (once again), and Krish Modi for the all-nighter on the backend! Without their help, our computer vision models, fine-tuned with YOLOv5 and YOLOv8 to detect road signals, and car plates, would have never been implemented in such a comprehensive algorithm to assess a driver's capabilities on the road! Looking forward to maybe implementing a classification model in addition to the CV models which may further enhance the algorithm's accuracy! Here is the GitHub repository for our project : [https://lnkd.in/gBQUnDVQ] Xin Lei

Building Planwyze greatly improved my software engineering & business understanding skills. I'd like to thank Lloyd Reshard and Vincent Quarles for all the requirements and business understanding coaching they are providing through the Kukua Institute Inc. black founder programs. Still have a lot to learn! PS: All data in the home view and the detail view in this demo are mock data. The live version will have actual data using the Yelp Fusion API upon approval of the Yelp Fusion Enterprise. Please share your thoughts! #swift #swiftui #swiftprogramming #ios #iosdevelopment #iosengineer #buildinginpublic

Did you know? Research shows expert commitment typing is crucial for reliable title services. At TitleBees, we don't leave it to chance. Learn how our experienced team ensures accuracy and consistency in every transaction! #titlebees #commitmenttyping #researchbacked

It's about to start comrades

Here's a framework of common tech stacks you can use when developing RAG applications. All of these are tech stacks I use frequently when building RAG prototypes for clients or personal projects. 👇 Drop a like if I should make a repo containing replicable code containing boilerplates for retrieval, augmentation generation, and UIs whether you need to build a prototype fast or have a hackathon project. CC: WeCracked #calhacks #hackathon #rag