Last week, we brought together our entire company (50-strong and growing!) for a fun and productive 3-day offsite in the beautiful Healdsburg wine country.
Building a startup can be all-consuming, and events like this give us an opportunity to slow down, reflect, and share new ideas in a distraction-free environment.
It was an intense couple of days but I came away feeling super energized and inspired by the team’s creativity and passion. At the end of the day, it’s all about the people — and I’m incredibly proud of the team we have built at Contextual AI.
We’re only getting started — join us on our mission to change the way the world works through AI! 🚀🚀🚀
🌞 As summer officially comes to a close, we want to take a moment to spotlight the outstanding work of our 2024 summer interns! 👏
Your passion for AI has led to significant contributions to projects here at Contextual AI as well as to the broader AI research community. Your work has helped us develop novel extraction techniques, improve our model evaluation and testing framework, create more efficient feedback loops for better model alignment, and much more.
We’re incredibly proud of everything you’ve accomplished and hope you made some amazing new friends and new memories along the way.
Though we’re sad to see you go, we know this is just the beginning of your incredible careers and we're glad to be part of the journey.
🚀 Upward and onward!
🙌 Well done Rohan Shah, Vignav Ramesh, Ali Hindy, Andrey Risukhin, Nandita Shankar Naik, Gurnoor Singh, Jon S., Maharshi Gor, Karel D'Oosterlinck, Dan LaBruna
Thank you also to our fantastic mentors Shikib Mehri, Casey A. Fitzpatrick, William Berrios, Winnie X., Sheshansh Agrawal, Soumitr Pandey, Pranjali Basmatkar, Akash Mahajan, Aditya Bindal, Amanpreet Singh, Douwe Kiela, Jenny Bae
Join us tomorrow for Transformers at Work 2024 in Berkeley! Douwe Kiela, Founder and CEO of Contextual AI will present "RAG on the Edge: GRIT and OLMoE for Hyper-Efficient Retrieval and Generation."
Discover cutting-edge techniques for deploying RAG on edge devices, including GRIT for faster caching and OLMoE, an open-source MoE model that outperforms larger models with superior efficiency! #taw2024
Check out the workshop and hear Douwe, the OG of RAG, as well as seven other top experts in the field.
https://lu.ma/2k566ig2
I had a great time speaking with Jacob Effron this week on Redpoint's Unsupervised Learning podcast. It was a fun conversation and we covered a ton of ground, like the role of RAG, real world challenges with enterprise AI, and of course the exciting work we’re doing at Contextual AI.
Some key points we covered:
- Enterprises have specific problems that need highly specialized AI for their business context — not AGI.
- Systems over model. This has been our point of view when we started Contextual AI, and we are now seeing this play out across the industry. OpenAI’s latest o1 model is essentially a system compressed into a LLM.
- Alignment is huge opportunity to improve the usefulness of AI in the real world, based on user feedback. KTO (Kahneman-Tversky Optimization), and more recently CLAIR (Contrastive Learning from AI Revisions) and APO (Anchored Preference Optimization) are promising techniques we’ve developed to solve the underspecification problem in alignment.
- Multimodal data represents a wealth of untapped data to improve AI understanding, has far-reaching applications in the enterprise, and we’re only scratching the surface of this opportunity.
- Other hot takes about AI Agents, synthetic data, the future of AI startups, alternate endings to GoT, and my old boss Mark Zuckerberg.
Our mission at Contextual AI is to change the way the world works with AI, and we’re only getting started.
Please reach out if you’re interested in learning more.
Douwe Kiela's (CEO, Contextual AI) contributed foundational parts to the AI ecosystem. He co-wrote the first paper on RAG and has raised over $100 million to help enterprises build contextual language models that fit their use cases. Before Contextual he was the head of research at Hugging Face, and worked on the Facebook AI research team. He remains an adjunct professor at Stanford.
On Unsupervised Learning, Douwe was incredibly open with his take on AI’s recent history and where he thinks it’s going. Some takeaways:
🤝 Alignment is exciting
Douwe is most excited about working on alignment – or making sure AI does what it's intended to do. One of the questions he asks himself is, ”how do we make systems maximally useful for the end users?” There are a lot of approaches, but one he’s excited about is Anchored Preference Optimization, or APO. It’s a way to cut out a lot of manual data tagging and make sure the model is using training data in the most optimal way.
🏗️ Infrastructure is fickle and hard
“When you’re not in a startup you think the infrastructure things are easy…turns out having a high end research cluster that actually works is incredibly hard,” Douwe says. The latest Llama paper had stats about the “amazing” number of hardware failures they had with the need to swap out GPUs or even entire nodes.
🧑💻Douwe’s Take on Open source vs Closed
Douwe’s bias is slightly towards open source. He thinks of models as a kind of triangle. The frontier models are at the apex – interesting but the most expensive. The bottom is full open source, where anyone can do anything. “The most interesting part is the middle of the triangle, where you get the right trade offs… if you start from open source, and then you have an amazing post-training capability, you can end up in the sweet spot.”
👏 Reaction to o1
Douwe’s reaction to the o1 model highlights a significant shift in AI from focusing solely on models to embracing broader system-level thinking. He explains how the o1 model compresses chains of reasoning, creating a more sophisticated system that enhances reasoning capabilities. Douwe finds this approach encouraging, as it aligns with the work his team is doing, particularly around retrieval mechanisms. He notes that while this model's system-centric approach is powerful, its future adoption will depend on deployment needs, particularly latency considerations during test time.
Check out the full conversation below:
YouTube: https://lnkd.in/g4YhH2ny
Spotify: https://spoti.fi/4daJxl5
Apple: https://apple.co/4d8ndIR
Douwe Kiela's (CEO, Contextual AI) contributed foundational parts to the AI ecosystem. He co-wrote the first paper on RAG and has raised over $100 million to help enterprises build contextual language models that fit their use cases. Before Contextual he was the head of research at Hugging Face, and worked on the Facebook AI research team. He remains an adjunct professor at Stanford.
On Unsupervised Learning, Douwe was incredibly open with his take on AI’s recent history and where he thinks it’s going. Some takeaways:
🤝 Alignment is exciting
Douwe is most excited about working on alignment – or making sure AI does what it's intended to do. One of the questions he asks himself is, ”how do we make systems maximally useful for the end users?” There are a lot of approaches, but one he’s excited about is Anchored Preference Optimization, or APO. It’s a way to cut out a lot of manual data tagging and make sure the model is using training data in the most optimal way.
🏗️ Infrastructure is fickle and hard
“When you’re not in a startup you think the infrastructure things are easy…turns out having a high end research cluster that actually works is incredibly hard,” Douwe says. The latest Llama paper had stats about the “amazing” number of hardware failures they had with the need to swap out GPUs or even entire nodes.
🧑💻Douwe’s Take on Open source vs Closed
Douwe’s bias is slightly towards open source. He thinks of models as a kind of triangle. The frontier models are at the apex – interesting but the most expensive. The bottom is full open source, where anyone can do anything. “The most interesting part is the middle of the triangle, where you get the right trade offs… if you start from open source, and then you have an amazing post-training capability, you can end up in the sweet spot.”
👏 Reaction to o1
Douwe’s reaction to the o1 model highlights a significant shift in AI from focusing solely on models to embracing broader system-level thinking. He explains how the o1 model compresses chains of reasoning, creating a more sophisticated system that enhances reasoning capabilities. Douwe finds this approach encouraging, as it aligns with the work his team is doing, particularly around retrieval mechanisms. He notes that while this model's system-centric approach is powerful, its future adoption will depend on deployment needs, particularly latency considerations during test time.
Check out the full conversation below:
YouTube: https://lnkd.in/g4YhH2ny
Spotify: https://spoti.fi/4daJxl5
Apple: https://apple.co/4d8ndIR
🔧 All about Generative AI: Join the Transformers at Work 2024 Workshop on September 20 📅
Looking for a top-notch event about the latest breakthroughs in AI here in Silicon Valley? We've got you covered! On September 20, Zeta Alpha is hosting it's fifth edition of the 𝘛𝘳𝘢𝘯𝘴𝘧𝘰𝘳𝘮𝘦𝘳𝘴 𝘢𝘵 𝘞𝘰𝘳𝘬 workshop in Berkeley. The workshop will focus on what is happening in the world of AI, specifically focusing on LLM applications for enterprise 💭
The workshops' content lies at the nexus of cutting edge research, and building practical applications that deliver value for clients. The program will feature AI-rockstars including world-renowned researchers and engineers that are moving the needle on Neural Search, RAG, LLMOps, Prompt Optimization, Agents and AI Hardware, including:
🤖 Rama Akkiraju - VP AI/ML for IT at NVIDIA
⚙ Douwe Kiela - CEO and Co-Founder of Contextual AI
👨💻 Fernando Rejon Barrera - CTO at Zeta Alpha
🧠 Zhuyun Dai - Research Scientist at Google DeepMind
🔧 Julia Kiseleva - Engineer at MultiOn
👨🎓 Michael Ryan - Research Intern at Snowflake
💻 Natalia Vassilieva - VP & Field CTO ML at Cerebras Systems
📖 Raza Habib - CEO and Co-Founder at Humanloop
The workshops will close with networking + bites and music in the evening 🥂
You won't want to miss out on this unique event, sign up using this link 👉 https://lu.ma/2k566ig2#AI#GenerativeAI#SiliconValley
Wrote an article on how Contextual AI is advancing large language models with innovations like RAG 2.0, OLMoE, and GRIT to revolutionize enterprise AI solutions. These advancements are driving better performance, scalability, and real-world impact across industries. Check out the details on my latest Medium post! #AI#MachineLearning#RAG2#ContextualAI#Innovation
Thank you NVIDIA for featuring us on your blog today! Contextual AI is proud to partner with NVIDIA to develop and bring the next generation of LLMs, powered by RAG 2.0, to market.
If you're trying to build an enterprise use case using RAG, you are no doubt familiar with issues limiting production-readiness such as:
1. Hallucination
2. Poor data freshness and auditability
3. Frankenstein RAG systems, with no end-to-end optimization between retriever and generator
4. Off-the-shelf extraction methods offering limited support for complex enterprise documents with tables, diagrams, and symbols.
5. Complex deployment, maintenance, security measures
We're already working with some of the largest companies in the world to eliminate these problems in their most complex and highly specialized Gen AI use case. Join us!
Read the blog here: https://lnkd.in/gCmVPE_dElias WolfbergAshutosh JoshiJ.Vikranth JeyakumarThomas GburekDouwe KielaAmanpreet SinghChloe HoAditya BindalJay ChenMike K.Jenny Bae
Enterprise AI systems need to be precisely aligned for each use case. We found that conventional alignment methods are underspecified, making this challenging.
Today, we share solutions that tackle both alignment data and algorithms, resulting in a ~2x performance boost for the same amount of data ($$).
With our contrastive data solution, AI developers precisely control *what* a model learns. With our tailored algorithm solution, AI developers precisely control *how* a model learns.
We’re also sharing a paper and blogpost discussing underspecification in alignment and our proposed solutions.
Read more in our blog post here: https://lnkd.in/eg2q83gW
Or, check out the paper: https://lnkd.in/eftdNZkN
