- AI Fire
- Posts
- ⚠️ AGI is a God We Can't Control (Yet)
In partnership with
Read time: 5 minutes
Claude just got a serious power-up, and it might finally close the gap with ChatGPT. Two new features quietly rolled out this week, just like plugins and API. Finally you can connect to your favorite apps, and back up its answers with REAL sources.
What are on FIRE 🔥
IN PARTNERSHIP WITH HUBSPOT
Want to get the most out of ChatGPT?
ChatGPT is a superpower if you know how to use it correctly.
Discover how HubSpot's guide to AI can elevate both your productivity and creativity to get more things done.
Learn to automate tasks, enhance decision-making, and foster innovation with the power of AI.
AI INSIGHTS
Even with perfect conditions, oversight may already be failing. According to new research on “Scaling Laws for Scalable Oversight,” weaker AIs - or even humans - may only correctly oversee stronger AIs about 52% of the time, even under ideal circumstances. As this intelligence gap grows, oversight effectiveness plummets.
And the urgency is clear: we might be building systems we simply can't monitor. One of the author’s assessment is that the 'Compton constant', the probability that a race to AGI culminates in loss of control of Earth, is >90%.
1. Simulation-Backed Oversight Limits
Researchers ran numerical simulations to model the probability of oversight success.
The study shows that Nested Scalable Oversight - a setup where multiple overseers watch each other in layers - often performs better than a single overseer.
Still, even in ideal conditions, when there's a 400 Elo-point gap, weaker agents only succeed ~52% of the time.
2. Real-World Estimates Using Elo Ratings
The team applied Elo curve data (typically used in chess rankings) to measure capability gaps.
Predicted oversight success rates:
Debate-style oversight: ~51.4%
Wargames setup: only ~9.8%
→ These rates highlight that oversight quality collapses as model capability increases.
If we're serious about building aligned and safe AI, this points to an urgent need for:
Smarter oversight systems that scale with model capabilities
Automated auditing tools trained alongside frontier models
Slow-down mechanisms as capability gaps grow beyond reliable supervision
We may already be at the edge of what we can safely oversee. And if we lose that grip, the consequences will be existential.
Why It Matters: As we race to build smarter and more capable AI systems, oversight is one of the last safety levers we still control. But this research shows that even the best current oversight strategies may fail when models get significantly more capable than their overseers, whether human or AI.
TODAY IN AI
AI HIGHLIGHTS
🔒 Anthropic just upgraded 2 major features for Claude: Integrations to connect more external apps & Advanced Research to cite real sources (differentiate itself as a “trust-first” AI company) - much like ChatGPT plugins.
🧠 Anthropic CEO publicly admitted that we have no idea how AI works, even at the companies building it. This “black box” is dangerous and unprecedented. He plans to build a sort of “MRI for AI”.
💭 People have “awakened” the AI, treating ChatGPT like a god. A thread titled “ChatGPT induced psychosis” went viral. For vulnerable individuals, this creates a dangerous feedback loop of delusion.
🕵️ You can now forward any WhatsApp message to Perplexity for an instant fact-check - especially helpful in a group chat where lots of misinformation is flying.
💸 Meta’s standalone AI app is planning a premium tier and ads with product recommendations being next. With nearly 1B users, Meta AI can go toe-to-toe with ChatGPT in reach.
👶 Google is letting kids under 13 use its Gemini AI through Family Link. It’s meant for homework help, but experts warn AI could confuse or even harm young minds.
💰 AI Daily Fundraising: Anthropic raised $3.5B at a $61.5B valuation to boost AI research, scale globally, and improve Claude - now cutting tasks like report writing from 12 weeks to just 10 minutes.
AI SOURCES FROM AI FIRE
NEW EMPOWERED AI TOOLS
💥 Meet PodSnacks*: A personalized, AI-powered email delivering the smartest insights from your top podcast. No scrolling, no fluff. Just the news you care about. Like Morning Brew, but built just for you. Try for Free!
🤖 omiGPT connects 100+ tools to ChatGPT and make it 5x more personal.
📈 LLMrefs tracks keyword rankings & optimizes your brand's SEO performance.
📱 Postiz is an ultimate AI social media tool with MCPs, 20+ available socials.
🛠️ GoCodeo is an open-source AI agent that builds full-stack SaaS apps.
*indicates a promoted tool, if any
AI QUICK HITS
🖼️ Trump faces backlash over AI generated picture of him dressed as pope.
🦉 Is Duolingo the face of an AI jobs crisis?
🎮 Google’s Gemini 2.5 Pro has beaten Pokémon Blue (with a little help).
🍏 Apple and Anthropic reportedly partner to build an AI coding platform.
🤔 AI is just as overconfident and biased as humans can be, study shows.
AI CHART
One of the world’s most iconic painters, Raphael, might not have painted every brushstroke in his famous “Madonna della Rosa”. That’s not a human art critic saying so, it's an AI discovering that the face of St. Joseph in the painting may have been painted by someone else.
The Mystery: The face of St. Joseph (top left) appears stylistically out of place compared to the rest of the painting.
Tech Behind the Scenes:
Core tech: Modified ResNet50 (a Microsoft-created deep convolutional neural network) - trained on verified works by Raphael.
It focused on elements like:
Brushstroke patterns
Color usage
Shading and composition
The system compared these traits to known Raphael features, down to minute details the human eye might miss.
What Did It Reveal? St. Joseph’s face, confirming suspicions long held by some human art historians, was possibly done by an assistant or added later.
Still, it does not replace human expertise, it offers a scientific second opinion, not a final verdict. If Raphael didn’t paint St. Joseph, it doesn’t mean fraud, it’s just collaboration - which was common in Renaissance workshops.
As museums digitize their collections, we expect AI-assisted authentication to become part of the standard workflow. And the next frontier? Not just identifying artists but discovering lost ones, mapping workshop networks, or even spotting forgery trends across centuries.
AI JOBS
We read your emails, comments, and poll replies daily
How would you rate today’s newsletter?Your feedback helps us create the best newsletter possible |
Hit reply and say Hello – we'd love to hear from you!
Like what you're reading? Forward it to friends, and they can sign up here.
Cheers,
The AI Fire Team
Reply