Joao Pedro Afonso Cerqueira

Tomorrow, Thomas Wood will be presenting the Harmony project at PyData London (organisers: Ian Ozsvald, Prashant Tiwari). 🤷 What does Harmony do? 👉 Psychologists and social scientists often have to match items in different questionnaires, such as "I often feel anxious" and "Feeling nervous, anxious or afraid". 👉 This is called harmonisation. 👉 Harmonisation is a time consuming and subjective process. 👉 Going through long PDFs of questionnaires and putting the questions into Excel is no fun. ✅ Enter Harmony, a tool that uses natural language processing and generative AI models to help researchers harmonise questionnaire items, even in different languages. 🌐 https://lnkd.in/drXNbRK #machinelearning #dataanalytics #analytics #artificialintelligence #ml #fastdatascience #naturallanguageprocessing #nlp #predictiveanalytics #bigdata #datascience #ai #neuralnetworks

2

Checkout our latest blog from Samuel Favarin. A real-world example of using LLMs to help automate a rather time consuming task of manual entry. Opening up time for professionals to focus on their value-add: driving new business! #DataEngineering #DBC #Azure #LLMs

5

Best data news? It's right here 🎺 https://lnkd.in/ecYCuxYD Check it out: Medium 🧠 🧠 The BigQuery Qualify clause @matteoBennio 🧠 How Notion handles 200bn data entities Vu Trinh 🧠 A scalable ingestion framework in MS Fabric Piethein Strengholt 🧠 Migrating and testing out pyairbyte Felix Gutierrez 🧠 Snowflake vs Databricks; a useful comparison for 2024 Hugo Lu 🧠 People do not like this latest article “The rise of the notebook engineer” (sorry Daniel Beach!) **Editors Pick** 🧠 How Pinterest do Analytics Kapil Bajaj LinkedIn🕴 🕴 BigQuery release continuous Queries and real-time rELT (also see Richard Shenghua (升华) He's excellent post on this recently) 🕴 Difference between Junior and Senior engineers 🕴 Trio of Leaders leave Open AI #dataengineering #roundup #dataroundup #bigquery #streaming

14

3 Comments

It's quite often a challenge to maintain governance over dynamic data schemas. Schema drift, characterised by changes in the structure, distribution, or quality of data over time, can have far-reaching implications on downstream systems and processes. By using Apache NiFi as our data movement and orchestration tool, with a touch of custom code, we’ve created a comprehensive schema drift detection and monitoring solution, allowing us to mitigate the potential impacts of schema changes and enabling prompt remediation when necessary. Read all about it in this week's blog post! https://hubs.la/Q02RbLHx0 #data #analytics #apache #nifi #cloudera #schema #dataquality #datagovernance

12

3 Comments