Quilee Simeon

Do you really understand why the multi-class cross-entropy loss works? The multi-class cross-entropy loss is an extension of the concept of entropy in information theory, applied to the context of machine learning classification tasks. Its formulation is deeply rooted in probability theory and statistical mechanics, providing a robust metric for evaluating the performance of classification models. Let’s dive deeper into the mathematical underpinnings and derivation of this loss function, along with a thorough analysis of its components. #math #statistics #machinelearning #artificialintelligence https://lnkd.in/g2rRK_Wm

2

Every 1% progress in AI research while simulating biological systems represents a significant amount of hard work done by researchers. The complex nature of integrating biological data into AI models means that even minor advancements are the result of extensive experimentation, troubleshooting, and refinement. Yet, armchair critics dismiss these achievements, overlooking the tremendous effort and dedication required to make such progress. Research, by its very nature, involves working on things that are “not yet there.” These critics often downplay these efforts with demands like “Show me where it works,” which not only reveals a superficial understanding but also hinders the spirit of scientific inquiry. Imagine if this attitude had prevailed in theoretical physics — we might have missed out on groundbreaking advancements. #Biology #ArtificialIntelligence #criticism https://lnkd.in/g2N8tbWr

2

A clever attempt to assess depth of LLMs. Designing these benchmarks is not easy and defining what is right in exploration tasks like experiments could be what is actually wrong with a benchmark. This line of research on how to properly design benchmarks that can reveal causal depth of models is fruitful.

14

Ted Chiang and Terence Tao: art, math and AI There is much I dislike about Ted Chiang's manifesto in The New Yorker two weeks ago (https://lnkd.in/gdXhCQ2K). But his narrow understanding of #AI, and specifically #genAI and #LLMs, is a topic for another time. 👉 A more positive take relies on a striking parallel between Chiang's (deeply flawed, in my view) main point that art is characterized by the myriad decisions required at all scales, and Terence Tao's observation (https://lnkd.in/gAjs2N7K) that getting to the solution of a really hard math problem requires a lot of decisions about where to go next -with so many dead ends that avoiding them is a key survival skill. The big difference between the two is that Tao can see the potential of AI to support the decision-making, even if the critical forks in the road are defined and decided by the talented human. Chiang thinks it removes meaning. I think the two should talk. 💎 This parallel is not as far-fetched as one might think. The Library of Babel, the quintessential Jose Luis Borges short story from his 1941 collection "The Garden of Forking Paths" (a perfect title for our purpose here!), bridges the two worlds. In the Library of Babel, all books already exist (up to a certain length), buried in an ocean of gibberish. So the decisions Chiang is talking about are akin to routing decisions in the Library: which book, which shelf, which section, which octogonal chamber, which floor, ... at all scales. Similarly, if a solution to a mathematical problem "pre-exists", the right sequence of decisions will lead to it. In both cases, the expert (or artist) is the scout, guided by experience, context, intuition, emotions (even in math). I know I am being naughty, here is a depiction of the Library of Babel generated by chatGPT4o.

1

1 Comment

Western scientists more likely to get rejected papers published – and do it faster   Teplitskiy and his colleagues worked with data provided by IOP Publishing (IOPP), a company based in Bristol, UK, that publishes more than 90 English-language journals and is owned by the Institute of Physics. They examined around 203,000 manuscripts that were submitted to 62 of IOPP’s physical-sciences journals between 2018 and 2022. Some 62% were rejected. The researchers scoured a bibliometric database to see whether the same (or similar) work was subsequently published elsewhere. They then sorted these publications by the geographical region of the corresponding author — the researcher who is usually in charge of a study’s publication process — and compared the outcomes for authors from the West (which they define as North America, Europe and Oceania) with those from the rest of the world. The analysis showed that corresponding authors from Western countries are 5.7% more likely to publish a manuscript after rejection than those from other regions. In a process that often takes up to 300 days, they did so 23 days faster, on average. These authors also revised the abstract of their manuscript — a proxy for the overall paper — 5.9% less often, as defined by a computational ‘edit distance’ metric. And, ultimately, they published in journals with 0.8% higher impact factors. #researcher #article #publication #journal #deviation #review #impact_factor #technologycenter   https://lnkd.in/dmkgw5x7

4

After decades of experience in building foundational AI models, the simplest rule of thumb I've observed that works empirically as a framework for hybrid modeling AI platforms, when applied in the following strategic order, is: 1) Start by extracting simple local features, prioritizing multi-scale extraction early on. 2) Then, focus on broad-range (dilated) interactions, ensuring high resolution by incorporating skip or residual connections. 3) Next, recalibrate features by modeling inter-dependencies (squeeze and excite). 4) Follow this by capturing long-range dependencies through some form of state management (e.g., multi-head attention). 5) Enhance expressivity through non-linear transformations (such as non-linear feedforward networks). 6) Finally, condense the representation using global pooling or dense layers. By applying these layers in multiple blocks through stacking, you will generally achieve effective numerical conditioning rapidly, minimizing the need for extensive ablation studies. This approach has consistently proven to be reliable. I stand by this rule all the time and works for me in general. #AI #ArtificialIntelligence #Engineering

4