Ali Al-Ghaithi, MSc

Our code for RAG evaluation is now publicly available at Amazon Science website: https://lnkd.in/g-HKUNZZ. This is the implementation of our evaluation mechanism performed by scoring the RAG on an automatically-generated synthetic exam composed of multiple choice questions based on the corpus of documents associated with the task. #OpenSource #OpenSourcing #AmazonScience #ICML2024 #Evaluation #LLMEvaluation

213

6 Comments

Being an amazing data scientist != you can lead the team! You can't just: - use more tools - put in more hours - learn more models - fulfill more requests - juggle more projects - respond faster to email - complete more "story points" (whatever the hell that means) You're already doing this, and your boss is telling you "you're not ready." They won't tell you why. They won't tell you what you're missing. And even if they did, there's no guarantee they'll have a role or budget to promote you. 99% of the time, they don't even know what you're missing, anyway. So you think you'll have better luck elsewhere. But no one will give you your first shot. You're stuck. And it's maddening! Ever seen a job that says "looking to hire an IC into their first manager role?" Me neither. Being an amazing data scientist does not mean you can lead the team! Why? Because... A DS leader's job is NOT to DO data science! Read that last line again. "Doing" data science is the data scientist's job. The DS leader's job is to ADVANCE the FUNCTION of data science. Advance towards what? - AI-driven cost savings (and) - AI-driven revenue growth You do this by 1. building partnerships across the business (in order to) 2. embed AI throughout the operating model (and) 3. launch AI-driven products That's it. That's your job. Print this out and put it on your wall if you want. This is the correct order of operations for all data science functions that aren't AI-first startups. Sometimes this means writing more code. Sometimes this means reading more papers. Sometimes this means learning more models. Often this means more MLOps. Often this means more meetings (for you). Often this means more focus on filling the team's skill gaps. If "doing" data science is the best way RIGHT NOW to advance the function, then you MUST do it. But often you're just doing it because it's in your comfort zone. To lead is to embrace discomfort. To lead is to Advance the Function. Never forget! BTW, if you liked this, a repost would mean the world to me. #thinksenior

82

5 Comments

Watching Gradient drop a Fortune 100's Databricks jobs costs by >50% in a few days is crazy. Everyone was shocked. 🔥 🔥 Best part was - it was all automatic. 🔥 🔥 While everyone was in meetings, building other pipelines, or just grabbing lunch - Gradient was at work dropping costs with no overhead at all. 💰 Gradient literally prints money for our customers ⚡ What's even crazier is Gradient isn't even done yet, it's still early on its journey in optimization The power of AI/ML in the cloud infrastructure world is, in my opinion, the killer app. And it's amazing to see this new technology in action helping real people. But don't take my word for it. Try it out for yourself. Book a demo here if you want help automating and managing your Databricks jobs Sync Computing

45

4 Comments

Here is some highlights of how to use #SKLearn library in #Python for #machinelearning #ml #ai #data

4

🔍📉 Ever feel like your data is just too much to handle? Here are the top 10 techniques to keep your data nice and neat: ✨📦 1. PCA (Principal Component Analysis): This linear technique captures data variance in orthogonal components. It's like getting a bird’s eye view of your data—seeing the forest for the trees. 🌳🦅 2. LDA (Linear Discriminant Analysis): Focuses on maximizing class separability. Think of it as the ultimate bouncer, keeping your data points in their proper VIP sections. 🕺🚪 3. t-SNE (t-Distributed Stochastic Neighbor Embedding): Non-linear and all about preserving local data relationships. It’s like a high school reunion, bringing old friends closer while keeping the frenemies at bay. 🎓👯 4. ICA (Independent Component Analysis): Transforms data into independent components, often used in signal processing. Imagine unmixing a cocktail to understand each ingredient—cheers to clarity! 🍹🔬 5. Random Projection: Uses random matrix projection to reduce dimensionality. It's like rolling the dice and still ending up with a good approximation of your data. 🎲🎯 6. Isomap: Non-linear method preserving geodesic distances in data. Perfect for keeping your data’s travel route as the crow flies, no detours allowed! 🗺️✈️ 7. Autoencoders: Neural networks that compress and then reconstruct input data. Think of them as the data whisperers, understanding and recreating the essence of your information. 🤖🗣️ 8. Factor Analysis: Captures data variance using latent factors. It's like uncovering hidden talents in your data—America's Got Talent for stats! 🌟🎤 9. MDS (Multidimensional Scaling): Represents pairwise data distances in reduced-dimension space. Picture it as the ultimate matchmaker, finding the perfect pairs among your data points. 💞🔢 10. UMAP (Uniform Manifold Approximation and Projection): Non-linear technique preserving both local and global data structures. The zen master of dimensionality reduction, achieving balance and harmony. 🧘‍♂️🌏 Curious how these techniques can make your data more manageable and insightful? Let’s connect and explore the possibilities! 📊💡 Image credit : data science dojo #DataScience #DimensionalityReduction #MachineLearning #BigData #TechHumor #AI #Statistics #LinkedInLearning

43

Dear companies, take care of your AI talent, especially the researchers. Researchers? Really? Aren't researchers the least influential people in the company? - They are the cost-centers. They do not directly sell to customers & so, do not generate revenue. And company has to fund them from profits - They live far away from reality. They do not know the "ground-truth". - They are not enterprising or charismatic like the #marketing & #sales' dashing & street-smart folks. - They cannot converse in normal language but are deep inside their narrow, un-understandable world. - Even if they do not exist in the company, the current product portfolio can be enough to sustain sales for years, or even decades. Look at ROLEX. Their watches are as mechanical as they were decades ago! Clearly their R&D budgets for adopting consumer-side emerging technologies are near-zero. - They help achieve company's Reduction in Force (RiF) targets. This is probably the only reason a company & its other employees may want them - for own survival. As they get hit first. So, why should companies care about their AI researchers, if at all they have them? The world has changed. They are one of the most influential people in the tech world and its only likely to increase with time. Why? We do not need every researcher to come up with a newer neural architecture or a different LLM, VLM or LMM. Right? That's true. But their role has just expanded and we still do not know. They are some of the best equipped people to manage the speed of current AI changes, naturally. Look, the AI world has been so rapidly advancing in the past 2 years that leaders are finding it difficult to keep track of them. Even if they succeed to keep track of them, their AI developments & implementations are becoming obsolete within 6-9 months! Ask a leader, and you'll invariably hear this. So? How will AI researchers help? They know the next in-thing (or likely to be one) at least a year before the world. And so, they can guide the next #AI #development or implementation within an organization that can sustain itself for a little over an year and a half, at least. Even if they are not implemented, at least the leaders & organizations can plan their 3-year strategies (Short Range Planning) based on their inputs. Are there any examples? AI Agents. They have been in existence for over an year. But organizations are still implementing LLMs. In spite of #Devin going viral! So tell me, what's the next in-thing that companies should think of? While there are nuanced answers to this question, some of the prominent ones, in general, would be - Natural Conversations, Reinforcement Learning & #Humanoids. But not every future in-thing actuates in future. No? True. And hence, we even need a Discriminator & not just a Generator AI researcher ^⁠_⁠^ #GAN #Future #HLM

20

3 Comments