The Part Nobody Explains: How AI Agents Decide What To Do

In the last post, we saw this:

AI can use tools.

It can:

• search the web
• run code
• open files
• call APIs

Cool.

But something still feels… missing.

Because there's one question almost nobody properly explains:

How does the AI decide which tool to use?

---

The Hidden Step Most People Never See

When you type something like:

"What's the weather in London?"

the AI doesn't just magically "know" what to do.

Under the hood, something more structured is happening.

Instead of replying normally, the model generates a structured tool call — essentially an instruction that says:

"Use this tool with these inputs."

The exact format varies by model and framework, but the idea is always the same: instead of plain text, the model outputs a structured action for the system to execute.

And honestly, this is the moment AI agents started making more sense to me.

Because suddenly it stopped feeling like magic.

---

At First, I Assumed One Giant Model Did Everything

My original mental model was basically:

"Okay… GPT probably handles everything itself."

Reasoning. Planning. Tool selection. Responses. Memory. Everything.

And to be fair, a lot of modern systems actually do work like that.

Large models from companies like OpenAI, Anthropic, and Google Gemini can often decide which tools to call directly.

But then I discovered something interesting.

Some systems use smaller, specialized models just for tool calling.

And that's where FunctionGemma comes in.

---

Meet FunctionGemma

FunctionGemma is a specialized version of Google's Gemma model built specifically for function calling.

Not chatting. Not storytelling. Not writing essays.

Its main job is:

Take user intent → convert it into structured tool calls.

That's it.

And honestly, I think that idea is fascinating.

Because instead of trying to make one giant model do everything…

you can split the system into smaller, focused parts.

---

And Here's the Wild Part

FunctionGemma is tiny compared to modern large language models.

It's built on the Gemma 3 270M parameter model.

Which sounds ridiculously small in today's AI world.

But the important realization is this:

Tool calling is actually a much narrower problem than open-ended conversation.

The model doesn't need to:

• write novels
• explain philosophy
• debate politics

It mainly needs to:

• understand intent
• choose a function
• generate structured outputs correctly

That constrained problem makes smaller specialized models surprisingly effective — especially after fine-tuning.

And the size has another huge advantage: FunctionGemma is specifically designed to run on-device — on laptops, phones, or edge hardware — without needing a server. That's a big deal for privacy and offline use.

---

This Completely Changed How I See AI Agents

Before this, I imagined agents like:

"One super-intelligent AI doing everything."

But now I think of them more like systems made of layers:

Main model         → reasoning and conversation
Tool-calling layer → converts intent into actions
Tools / APIs       → actually perform the work
Orchestration      → manages the loop, retries, and flow

Almost like:

Brain → Decision Layer → Hands

And suddenly agents feel a lot less mystical. And a lot more understandable.

---

So What Actually Happens in an Agent?

At a very basic level, the loop is surprisingly simple.

User says:

"Search AI startups"

The system generates a structured tool call, something like:

tool: search_web
query: "AI startups"

Then:

1. The backend executes the tool
2. Gets the results
3. Sends them back to the model
4. The model continues

Think → Call tool → Get result → Continue

That's the core loop.

---

Real Agents Add More Layers

Of course, production agents usually become much more complex.

They may include:

• memory
• retries
• validation
• planning
• permissions
• context management
• error handling

But the important thing is: the core idea is still understandable.

And honestly, that realization made AI feel way more approachable to me.

---

One Important Thing Most People Miss

FunctionGemma is not really meant to be dropped in "as-is" as a universal agent model.

Google actually positions it as a foundation for fine-tuning.

Meaning:

• You define your own tools
• Train it on your own examples
• Improve its reliability for your specific use case

So instead of:

"one AI that knows everything"

you get:

"a small, specialized model trained for your specific workflows"

That's a very different philosophy. And the benchmark numbers back it up — the base model scores 58% on Mobile Actions tasks. After fine-tuning? 85%.

---

And Honestly… I Think This Is Where AI Is Heading

The more I learn about agents, the more it feels like modern AI systems are becoming modular.

Instead of one massive model doing everything, we're starting to see:

• routers
• planners
• memory systems
• verification models
• specialized tool-callers
• local edge models

Smaller parts working together.

And weirdly… that makes the whole field feel less intimidating.

---

What I'm Doing Next

So instead of only reading about agents…

I want to build one myself.

A small one. From scratch. Nothing insane.

Just:

• a few tools
• a simple loop
• tool-calling logic
• structured outputs
• backend execution

And I'll document the whole process here as I learn.

---

One Line Worth Remembering

AI agents are not just "one giant AI." They're systems made of smaller parts. And sometimes… the part deciding what action to take next can be surprisingly small.