- The Loop
- Posts
- 🔆 AI Decisions Exposed: Trading Mystery for Trust
🔆 AI Decisions Exposed: Trading Mystery for Trust
Generative search tools don't give the correct answers, big funding for biomedical breakthroughs, and explaining trained AI models.
Allegra Guinan & Emma G Moller
April 03, 2025
Was this email forwarded to you? Sign up here
🗞️ Issue 64 // ⏱️ Read Time: 7 min
Hello 👋
When an AI system denies your loan application (say adieu to your new apartment), approves a medical treatment (is that second round of antibiotics even good for your body?), or flags content for removal (shadow-ban, anyone?), should you be entitled to know why?
As AI increasingly influences high-stakes decisions in our lives, the divide between explainable and "black box" AI models raises profound questions about transparency, trust, and accountability.
In this week's newsletter
What we’re talking about: The fundamental distinction between explainable AI models with transparent decision-making processes and black box models whose internal workings remain largely opaque, even to their creators. Recent interpretability research is beginning to reveal how modern AI systems actually process information.
How it’s relevant: As AI systems make more consequential decisions in healthcare, finance, hiring, and beyond, the ability (or inability) to understand how these systems reach their conclusions directly impacts user trust, regulatory compliance, and ethical implementation. New interpretability tools are helping researchers detect when models might be fabricating explanations or susceptible to manipulation.
Why it matters: The tension between model performance and explainability presents one of the central challenges in responsible AI deployment. Organizations that effectively balance these competing priorities will be better positioned to build sustainable, trusted AI systems that deliver value while mitigating risks. As AI capabilities advance, understanding these systems becomes even more critical for ensuring safety and alignment with human values.
Big tech news of the week…
🔡 AI search engines are generally bad at declining to answer questions they can’t answer accurately, and generative search tools fabricates links and cites syndicated and copied versions of articles, according to a Report from Tow Center/Columbia Journalism Review. In other words, consider carefully if you really trust the information you get from search engines like Perplexity.
🩺 Apple is developing an AI-powered health coach and virtual "doctor." This initiative is focused on the Health app with personalized recommendations and insights derived from users' health data. The model is currently being trained on data from staff physicians and specialists across various medical fields.
👋 Joelle Pineau, Meta's head of AI research, has announced her departure from the company. Pineau was instrumental in advancing Meta's AI ethics and transparency efforts and championed the importance of sharing research findings and tools with the broader scientific community. Time will tell how her departure will impact Meta’s AI narrative.
🫠 OpenAI removed the free-tier access to its GPT-4o image generation tool just 24 hours after launch due to a combination of legal, ethical, and technical concerns. The decision followed a viral surge of AI-generated images mimicking the distinctive style of Studio Ghibli. CEO Sam Altman stated that demand was so high it caused their GPUs to "melt."
💰️ Isomorphic Labs, an AI-first drug design and development company, raised $600 million in its first external funding round. They aim to apply their pioneering AI drug design engine to deliver biomedical breakthroughs. Their breakthrough model, AlphaFold 3, was developed and released in May 2024 together with Google DeepMind, with the ability to accurately predict the structure and interactions of all of life’s molecules.
The main tension: Performance vs Explainability
Exciting developments in AI interpretability research are offering unprecedented glimpses into how these systems actually "think", meaning that we can understand the reasons as to why one outcome is prioritised over others. This represents a significant leap forward in our ability to understand AI systems that have historically been opaque black boxes.
This week, we're exploring the critical balance between model performance and interpretability that organizations must navigate.
Here's the central tension: The most powerful AI models today, those capable of remarkable feats in language understanding, image recognition, and pattern detection, are often the least explainable.
This creates a fundamental dilemma for organizations: Do you choose a more transparent, explainable model that might offer less impressive performance? Or do you opt for a state-of-the-art black box model that delivers superior results but can't explain its reasoning? Let’s start from scratch and look at the main differences between explainable models and non-explainable models.
Explainable vs. Non-Explainable AI: Back to Basics
Not all AI is created equal when it comes to transparency. To understand the landscape, we need to distinguish between two broad categories: Explainable AI Models and Black Box Models. The distinction isn't binary. There's a spectrum of interpretability, with trade-offs at each point. Generally, as model complexity and performance increase, explainability tends to decrease.
Explainable AI Models
These are systems whose decision-making processes can be understood, interpreted, and explained in human terms. See the table below for some examples of explainable models.
