“AI agents will replace all data analysts.” “AI analysts will never be trusted.” Two extreme takes. Both are wrong.

Why This Newsletter Episode Matters

The debate over AI analyst agents is heating up, and for good reason.

If you work in data - whether you’re an analyst, engineer, scientist, or CDO - you’ve probably heard claims that AI is about to replace all data analysts. Or, on the flip side, that AI analysts are useless because of hallucinations and unreliable outputs.

So who’s right?

My take: Neither. 

The truth is, AI analysts can work. But only under very specific conditions.

And if you don’t understand those conditions, you’ll either waste months building a system that breaks... or you’ll dismiss the entire opportunity and get left behind.

This newsletter walks you through my lessons from testing several conversational analytics tools, what makes most of them fail and what one setup is actually showing promise.

Let’s dig in.

The Story: A Tale of Two Extremes

A few weeks ago, something interesting happened on LinkedIn.

The CEO of Voi (the scooter company) announced that they’d ripped out Tableau and replaced it with chatbots in Slack and Google Sheets. The post went viral.

It also polarized the data community:

• One side was cheering, “Finally! The death of dashboards!”
• The other was furious, calling it naive and dangerous.

At first, I also criticized it but then I dug deeper into the case.

Most people (including me, at first) missed what actually mattered: the strategy and foundations behind the setup.

Those are the critical components to make the tech work.

At the same time, I’ve been testing a number of conversational analytics tools myself and speaking with a few vendors in the space. And here’s what I’ve learned:

• Many vendors are building text-to-SQL systems.
• Many of these systems are designed to be used by non-technical users.
• And almost all of them are running into the same wall: non-determinism and hallucinations.

The Problem: Why AI Analysts Mostly Fail

Here’s the root of the problem:

LLMs are probabilistic. SQL is deterministic. That’s a mismatch.

Let’s break it down.

When an LLM writes SQL:

• It doesn’t “know” the right query.
• It makes an educated guess.
• That guess might be right. Or it might be off by just enough to look convincing - but return incorrect results.

If a strong data engineer is in the loop (for example for the "vibe coding" use case), this isn’t as much of a huge deal, compared to the conversational analytics use case. The engineer will spot the issue, tweak the query, and move on.

But if the end user is:

• A marketer
• A product manager
• A finance manager

…they’ll have no idea that the output is flawed. They’ll make decisions based on broken logic and they won’t even know it.

That’s why AI analysts as they’re being built today are often worse than useless. They're dangerous.

But that doesn’t mean it can’t work.

It just means you need to flip the approach.

The Solution: A Setup That Actually Works

Let’s go back to Voi’s example.

They didn’t just plug a chatbot into BigQuery, Slack and Google Sheets and hope for the best.

They built their system on two non-negotiable foundations:

1. Strategic Pillars

a) Strong Data Governance

• A solid semantic layer (defined in yml and not SQL!)
• Well-defined metrics and dimensions
• Controlled vocabulary

b) Skilled Business Superusers

• Not SQL experts, but data-savvy
• Understand how to use metrics to drive decisions
• Know the logic behind the metrics they’re querying

I have built a whole masterclass that teaches all my frameworks and blueprints about setting up these crucial foundations . 

These foundations change the entire game.

Instead of relying on the chatbot to “guess” what the user wants and generate SQL on the fly, it becomes more like an interface for an already well-structured system.

2. Technical Design: The Semantic-Layer-First Stack

Here’s the stack that’s starting to show real potential:

➡ Semantic Layer (e.g. Cube, dbt, Connecty) Defined in YAML or JavaScript. Not in SQL.

➡ Chatbot that natively connects to the Semantic Layer (e.g. Magnowlia AI) Sits on top of the semantic layer, not on top of the raw database.

➡ Strict interaction rules

• The chatbot only uses defined metrics and dimensions from the semantic layer
• If the user uses a synonym, the chatbot asks for clarification (e.g. User: “CTR” → Bot: “Do you mean click-through rate?”)
• No direct SQL generation. No guessing.

Key Framework: How to Design a Trustworthy AI Analyst

To summarize, here’s the framework to follow if you’re considering conversational analytics:

1. Build a Robust Semantic Layer

• Define all metrics and dimensions explicitly
• Use YAML or JavaScript, not SQL
• Think of it as the source of truth for your analytics logic

2. Restrict Access to the Semantic Layer Only

• The chatbot should not have access to the raw warehouse
• No open-ended SQL generation
• All queries go through the semantic layer’s definitions

3. Enable Clarifying Interactions

• Use synonyms? Bot asks for confirmation
• Ambiguous query? Bot prompts the user to refine
• Make it a guided experience, not a black box

4. Train Business Superusers

• Ensure users know what metrics exist in the semantic layer
• Help them understand the logic behind those metrics
• Don’t expect them to write SQL, but do expect them to know what they want

What’s Next

I’m now setting up a POC using this exact stack:

• Semantic Layer: Cube
• Chatbot: Magnowlia AI

This setup mirrors the architecture used at Voi and it’s the only one I’ve seen so far that has a chance of scaling safely.

I’ll be testing this with real users and real queries in the coming weeks and will share my findings soon.

I also launched a community of AI-first data team leaders. Inside we discuss and share real-life best practices about conversational analytics and other approaches to create outsized business impact in the AI-era.

Bottom Line

AI analysts can work, but only if we stop pretending they’re magic.

They’re not here to “replace” data teams. And they’re not useless, either. 

But if you want to build a conversational analytics agent that people can actually trust, here’s what you need:

• A controlled semantic layer
• No raw SQL access
• Clarifying dialogue
• Business users who understand their data

Anything less is a liability.

So if you're experimenting with AI in your analytics workflow, don’t fall for the hype -or the hate.

Build it right - or don’t build it at all.

More updates soon.

Cheers,

Sebastian

P.S.: If you want to get updates really regularly and want to be part of a group of data leaders who build tomorrow's AI-first data teams, check out our 10X Data Team Collective .

P.P.S.: I'm not affiliated with any vendors mentioned in this email. You are reading my unbiased views and opinions.