Quotient AI

We were excited to sponsor the Duke University AI Hackathon last weekend!

We're working on intelligent prompt engineering. 🧪 PromptLab + 🧠 IQ Rapidly prototype with your favorite models. Evaluate. Build production-ready AI products. Out in beta. (Grab your laptop for this one!) 🔊 on 👀 more coming soon 👀

👻 No tricks, only treats! 👻 Join us for a fun night of AI projects with AI Tinkerers London. We're teaming up with Monzo Bank and Zilliz to bring engineers, AI enthusiasts, and tinkerers together for a hands-on meetup focused on everything AI. Whether you're working on the next breakthrough or just curious, you'll have the chance to share ideas, network, and enjoy pizza, swag, and refreshments! Interested in Speaking? Got an exciting project or experiment in AI? We’d love to hear about it! No need for polished slides—just bring your work-in-progress ideas and connect with a community of experts from across the AI space. Sign up in comments 👇

Quotient AI's new SMELL framework, now AI developers can better align LLM evaluators with human feedback. 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗦𝗠𝗘𝗟𝗟 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 • Subject-Matter Expert Language Liaison for AI evaluation • Combines human expertise with LLM capabilities • Generates structured feedback, rubrics, and evaluation metrics 𝗞𝗲𝘆 𝗙𝗲𝗮𝘁𝘂𝗿𝗲𝘀 • Integrates domain-specific knowledge into AI assessment • Leverages LLMs to process and analyze vast amounts of data • Bridges the gap between expert input and AI capabilities 𝗜𝗺𝗽𝗮𝗰𝘁 𝗮𝗻𝗱 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 • Enhances enterprise AI development, especially in customer support • Improves AI performance through targeted feedback • Offers scalable evaluation across various domains and use cases Quotient AI's SMELL framework is revolutionizing how we approach AI evaluation, ensuring more human-aligned and reliable AI systems.

John Berryman and I recorded a podcast.. and we took ourselves very seriously! Listen to the first episode of Two-Shot to learn more about prompt engineering, LLM evals, and our takes on the is-RAG-new-or-is-it-just-search debate. Check out the full episode of Two-Sho with {John Berryman} {Principal Consultant, Arcturus Labs} {ex-GitHub Copilot} on all the podcast places: - YouTube: https://lnkd.in/gNjq4KGg - Apple: https://lnkd.in/gHxV7jhu - Spotify: https://lnkd.in/g7vFPND9 More episodes are coming with some very cool people! Each episode invites a guest to a straight-to-the-point, no-softballs, deep(er) dive about the areas they're most passionate about. DMs are open for speaker & topic proposals.

Everyone I talk to is at least considering developing with open LLMs, but they often have to "show that it works" with a proprietary model first. This is changing fast. I'm very confident that open LLMs are going to play a huge part in AI development moving forward. There's a ton of very compelling tooling being developed that's lowering the adoption barrier for open LLMs and making the open vs. proprietary decision more of a tossup. I write about this shift and the tools I expect to be part of the permanent core dev stack: Together AI, Fireworks AI, Groq, OpenPipe, Martian, unstructured.io, LangChain, LlamaIndex, Pinecone and Qdrant. Check out the article here: https://lnkd.in/ev2vRXc6

Do your LLMs pass the SMELL test? 👃 Introducing SMELL (Subject-Matter Expert Language Liaison): a new framework for aligning LLM evaluations with human feedback. SMELL integrates subject-matter expertise into AI evaluations, ensuring more accurate and domain-specific assessments. Traditional LLM evaluations often miss the subtle judgments that humans bring to specific tasks. SMELL captures a small set of expert feedback, synthesizing it into rubrics for LLM judges—allowing AI developers to evaluate domain-specific use-cases at scale. Inspired by Databricks Mosaic Research's Grading Notes framework, SMELL refines LLM evaluation by incorporating concise, task-specific grading guidelines. By adapting judge rubrics from human feedback, SMELL creates a scalable, nuanced system that aligns closely with human assessments. SMELL achieved outstanding results on various preference datasets, with a Cohen's Kappa of 0.92, 96% accuracy, and a 0.96 F1 score on the preferences from UltraFeedback—all using just 5 in-sample data points. SMELL's ability to align with human judgments, even with limited feedback, highlights its efficiency and strong predictive power. Curious about integrating SMELL into your AI workflow? 📓 Explore our detailed Jupyter notebook for insights into SMELL's implementation and results. https://lnkd.in/eN3UfJMF 🔗 Try SMELL in your own projects with our preview API: https://lnkd.in/eJX8YV-7 🤝 Want to build a custom judge for your specific use case or contribute to our research? Share your datasets with us at [email protected], and we’ll publish the results with full attribution.

After months of research, we are droppin’ 10 companies from The #AIHot100 Market Report (East Coast)....curated by The AI Furnace 🧨🔥 ft. 100 of the hottest AI startups (pre-seed to series A) 🧯💥 We curated the Hot 100 list from over thousands of submissions, using a committee of AI, technical, startup and industry experts and brought them together at The #AIHot100 Conference in NYC last week. Sharing our first 10 below 👇 ➡️ Carbon Arc - Gather and structure raw data into on-demand, consumption-based insights  ➡️ Arcanna.ai- Decision Intelligence Platform for Enterprise Critical Ops ➡️ Quotient AI - State-of-the-art LLM evaluation and synthetic data generation platform ➡️ Multimodal - Automates complex middle and back office workflows in financial services ➡️ Penguin AI- Leverage AI to retarget your high-intent visitors via email and Linkedin ➡️ Goldcast - AI-powered B2B Video Campaign Platform ➡️ AutoPCB - AutoPCB automates circuit design ➡️ Metaphysic.ai - Visual effects with AI ➡️ VoyceMe - Building the future of StoryTech ➡️ EZ Newswire - Turn your news into performance Some hidden 💎s here - this is the culmination of one year of building grassroots community with The AI Furnace 🧨🔥, getting to know founders, seeing under the radar startups, and some who cannot demo due to being in stealth. The report has Sector View, Layer View, List View and more. 👉 If interested in seeing the full list,  sign up for our newsletter to see the full PDF version https://lnkd.in/eGpzsTQr #AIHOT100 #aifurnace Kirk McKeown, Petrica Ruta, Julia Neagu, Ankur Patel, John DiLoreto, Ethan Ding, Palash Soni, Caspian Chaharom, Jo Plaete, Dylan Telano, Caitlin Kelly, Neel Shah

The #AIHot100 Conference is tomorrow and we're launching The AI Hot 100 Market Report (East Coast) curated by The AI Furnace 🧨🔥, ft. 100 of the hottest AI startups (pre seed to series A) on the East Coast. We had thousands of submissions and we have curated this list using a committee of AI, technical, startup and industry experts. We are bringing these founders to the #AIHot100 Conference in one space in Manhattan tomorrow. The report has Sector View, Layer View, List View and more. Some hidden 💎s here - this is the culmination of one year of building grassroots community with The AI Furnace 🧨🔥, getting to know founders, seeing under the radar startups, and some who cannot demo due to being in stealth. 👉 Comment your email below and I'll send the full PDF version one week from now Some of the startups include: Nexa AI Lazarus Ohai.ai Moonhub 🚀 Enkrypt AI Reality Defender Lyzr AI Thumbprint Furniture Flock AI CalypsoAI Thinkverse lytix ai Wild Moose Alinea Invest Topline Pro #AIHOT100 #aifurnace

Big news! Quotient AI is featured in Qdrant’s new RAG Evaluation Guide!  This guide provides in-depth insights into assessing Retrieval-Augmented Generation models, and we’re honored to be recognized for our role in the evaluation landscape. Discover how our framework contributes to more accurate and effective RAG evaluations! Check out the Guide here: https://lnkd.in/ezXvDden