Ntropy

Our CEO Naré Vardanyan shares why bringing the costs down for LLM inference while guaranteeing performance and reliability is still one of the biggest problems to solve in this decade. Using Ntropy API today to classify billions of transactions, you are getting SOTA model performance, reliability and cost-efficiency like no where else. We are talking 3-4 orders of magnitude. It is not about being the best model any more. It is about delivering you the best output at any point in time, reliably and fast, at a cost that makes economic sense.

This year there have been a lot of reports and a ton of chatter re the ROI for the compute spend. So at Money20/20 with NVIDIA and Amazon Web Services (AWS) we are going to debunk the myths and talk real use cases, run demos, show end to end results and obviously sprinkle some magic ! Join us at the Pavilion ! Ntropy and Ntropians will be at the booth from 9am daily! Obviously we could not make this happen if our customers didn’t build awesome products on top of our APIs , Amazon Web Services (AWS) and NVIDIA Accelerated compute! Cascading AI (YC S23) Scribe Novo Airbase 🚀🚀🚀🚀

Building finops and for CFOs? Here’s what we have learnt

Surprise surprise: You AI agent needs great quality financial data! If I do say so myself !

Prompt caching, just launched by Anthropic, is a must-have if you are running high-throughput tasks with LLMs. Lets unpack: - it is effectively the same as Gemini's "Context caching" that was launched 2 months ago - makes fine-tuning obsolete for all, but some niche cases - unlocks up to 10x reduction in cost for high-throughput applications where the ratio between (static input tokens) vs. (dynamic input tokens + output tokens) is large - we can call prompt caching (or "context caching") LEVEL 2. By only requiring *parts* of the prompt to be the same as a previous query, it goes one step further than the most basic LEVEL 1 input caching, where the *full* prompt has to be exactly the same as a previous query. - at Ntropy we have had an even more advanced LEVEL 3 caching setup in production since early this year, which has enabled us to scale to massive LLM volumes at an even lower cost than running smaller specialized models in-house. - with LEVEL 3 caching, even queries that at first glance look different, but which have fundamentally similar structure can share the same cache bucket. This requires figure out structural similarities between the *variable* parts of different prompts. We use an approach similar to locality-sensitive hashing with hashing models that are specialized for each individual task. - LEVEL 2 and LEVEL 3 caching can be combined to achieve orders of magnitude cost and latency reduction over raw model queries and are key to make the largest LLMs viable for high-value, high-throughput tasks in the real world. So much more to come 🚀🚀🚀

There are many reasons why code generation and creating a software engineering agent is the holy grail for so many companies. Software engineering is a classic example of a high leverage activity, but is very expensive. Most code is buggy, gets old, needs to be updated and maintained, is hard to audit. The consequences of bad code are far reaching and can even result in serious disruptions. So now we have three components that matter: - Very high utility, ie directly leads to a lot of value being created - Expensive and growing demand for it with low supply, meaning there is not enough humans to write all the code that needs to be written. - Needs to be constantly maintained and updated. Often cost of maintenance is higher than the design of the solution itself. Because of the above, software as a service took off in the last decades. Instead of hiring humans to write expensive code, then audit and debug it for every single activity you want to perform via software, you could suddenly pay for the code and its constant deliverability in a certain narrow area. This is how we end up with so many vendors and SaaS indices. Now, if the cost of writing code goes down and instead of 30M developers, we have hundreds of millions of software development agents, imagine the amount of process that can be automated and efficiency that can be introduced. The second order implications of this are wild. But I am not writing about that today. If you are a founder, trying to find the right "skills" or "tasks" to disrupt with LLMs the framework behind SE is very viable. - Needs to have enough utility, yet a lot of human scarcity behind it - Needs to be expensive to perform with humans. The demand needs to be growing vs shrinking, which means cost drops will activate further demand, following Jevons paradox - Maintenance and update take equal amount of time and cost, making the demand continuous If your task contains these properties and the person you are augmenting / automating processes for has a gun to their head to get things done, you are "golden". Software engineers, recruiters, GTM roles, accountants and finally lawyers are obvious places to go . Lawyers are interesting. At first glance it seems they can bill less hours hence their business model will be challenged. On the other hand, the entities that can suddenly afford first class legal advice will grow drastically in number, multiplying the billable hours, even if the business model never changes. Accounting and financial workflow automations are a no brainer here. Finance - Has high utility and is very expensive. You cannot do anything, if you do not do the finances - Shrinking supply of financial professionals and rising demand. - Bringing the costs down means the demand will rise further, allowing anyone to have access to real-time intelligence. - Are repetitive, hard to maintain and audit If you are building financial automations, talk to us at Ntropy.

Understanding unstructured data for SME-s has been impossible at scale making them highly unprofitable to lend to and invest in . This is changing thanks to better data + LLM-s. Check out what our partners at Validis have to say about it

How are you evaluating vendors , how long does it take you, what matters? As the sales stack is changing, so is procurement and how we manage the many vendors. Guess what you need to streamline a vendor management solution and get on top of it? Yes, the Ntropy API, and reliable, clean and accurate transaction information. Our CEO shares her thoughts below re AI native sales vendors and how the space is shaping up!

Recenlty there have been many thought pieces on LLMs and what AI can and cannot do and where it is useful . If you are building with AI, breaking through the noise is key. Join Ilia Zintchenko at Fintech devcon today, where he shares notes on deploying LLM-s at scale in financial services and touches upon bottlenecks such as cost, speed, reliability, compliance and how we are working to overcome them. One of the fascinating things about this technology is that its parity with the ability of humans in certain areas pushes many to apply it to things it is not necessarily fit for yet. There is a lot of economic value lying just in front of us where it is a perfect fit and is going to change anything. Even if these models never got better from this spot, that value is up for grabs. But they are getting better. The cycles will take longer with the data centre buildout, but the reality is that we have not seen anything make a bigger difference to the quality of the models than scale. So we shall scale. In the meantime, let us push some intelligence to prod !

Years ago, the first Macintosh had the ambition of talking to us and we found it cute. Hello world ! Now we have machines that can actually talk back and do commands in English. And guess what is one of the first areas that is going to be disrupted by this? Accounting, Tax and audit. This is a very unique opportunity where we are seeing a ton of adoption with LLM-s. One of the main reasons is the declining talent pool of human professionals and the increasing demand in these services. So many companies are re-thinking the whole space. We have event had inbound outreach from government bodies trying to address tax credit fraud with automation because they do not have enough humans to audit. Accounting data is a core source of truth to understand businesses. LLM-s have unlocked what you can do with this data , from CPA assistants and co-pilots to automated tax filing and loan underwriting and finally delivering real-time intelligence for your finance team. But first, you need to get the data and make sure it is of high quality. Data normalization and ingestion from bank ledgers into accounting platforms is still very manual, context dependent and demands high precision. We are solving this. Once you have the better data, your further output with LLMs can be more reliable and grounded. Today we are partnering with Validis , one of the top providers of accounting data, to bring bank and accounting information to the LLM age. Companies like Barclays and Santander, Grant Thornton (US), Withum and Deloitte already rely on Validis for access to accounting feeds. The announcement blog with more details is in the comments. If you are building in the space, do reach out ! Super excited for this chapter and getting to work with Ronan O'Dea, Michael Turner and the rest of the Validis team !