Counting people in a 24/7 stream with a giant language model works, but it’s using a sledgehammer to crack a nut: you pay for capability you don’t use, and the result is worse than what a purpose-built model delivers. That thought ran through the talk our colleagues Javier Carnero and Rodrigo Cabello, from the R&D team at Plain Concepts, gave on Wednesday 30 September at Espacio RES in Seville, at an event organised by the IA Generativa Sevilla community in collaboration with SevillaDotNET. They showed how to build a team of AI agents that work together.
Here’s a recap of what they covered: what an agent really is, when you need a coordinator, three demos in computer vision and 3D, and how to test all of it without losing your mind. And if you’d rather hear it first-hand, we invite you to watch the full talk.
An agent is less mysterious than it sounds
An agent is a model, system instructions, tools and memory. The tools are the data it can query (through RAG, for example), the actions it can run and the connectors to other systems. The model doesn’t have to be a text model either: it can be a vision-language model that looks at an image and reasons about it.
A simple agent with its own skills takes a few lines of Python with Microsoft Agent Framework, which is open source, connected to a local model running on Ollama. Then you plug it into whatever app you have, in Python or C#.
Now the uncomfortable part. Agentic systems are not deterministic, they hallucinate, and costs in production climb fast if you build them carelessly. Classic automation is still perfectly valid, and the sensible move is to combine it with agents instead of retiring it.
Language models can’t do everything
When the task is visual (detecting, segmenting or counting objects, or pulling a 3D object out of a photo), purpose-built computer vision models deliver better quality at a much lower cost. Even Karpathy and LeCun acknowledge that LLMs built on text tokens fall short when it comes to understanding space.
LLMs remain the right tool for what they do best: holding a conversation, interpreting instructions and deciding what to do next. We split the work so each model does the job it’s best at. A small model, such as a classifier or a JEPA-type model, fits triage and orchestration decisions very well, since nobody needs an essay at that step.
Three ways to coordinate a team of agents
As soon as you have several agents, someone has to decide who’s in charge. These are the three patterns we’ve tested.
- Sequential. One agent’s output is the next agent’s input. It’s deterministic and easy to follow. Our number one recommendation is to always start here and add complexity only once the happy path works.
- Group chat with a central coordinator. You get more control, but the coordinator reads everything and burns more tokens.
- Handoff. Each agent decides who gets the baton next, with no coordinator. On the same task it used clearly fewer tokens than group chat, and we measured it with telemetry and traces. The trade-off is that it’s harder to control, sometimes calls for more powerful models, and works best when each agent’s responsibilities are cleanly separated.
A practical tip: build the flow as a group chat first and switch to handoff afterwards, because in code the change is small. There are more patterns (network, supervisor, hierarchical or custom), but these three are enough to get started.
One detail that saves money. The context the framework shares between agents is not the same as what gets sent to the LLM, so send only what’s needed. You save tokens, context window and budget.
Three demos, from video to 3D worlds
The first one is about video. We built a handoff flow with a triage agent, a video understanding agent (a vision-language model on Azure Foundry) and a video manipulation agent that runs a small local model on Foundry Local, with an FFmpeg skill and access to Blob Storage. The job was to pick the most representative frame of a video and save it as a still. It worked: the agent chose the moment, called FFmpeg with the right parameters and left the image in storage. For debugging we used the framework’s development web UI, which draws the agent graph.
The second starts from a customer case. They send a photo of a sofa from their catalogue and want to see it in 3D on their website. Triage hands off to the segmentation agent: Grounding DINO finds “a sofa” from a natural-language prompt and returns the box, and SAM turns it into an exact mask. That service runs as an API on Azure Container Apps with a T4 GPU and is described with OpenAPI, so the agent uses it like any other tool. Next comes the 3D agent, running SAM 3D with light fine-tuning for several objects, also on Container Apps, with a volume for the weights and scale to zero, so it shuts down when nobody is using it. A web agent closes the loop, with Gemma running locally through Ollama, compatible with the OpenAI contract. It generates a static site with the 3D viewer and uploads it to storage, so you can walk around the sofa.
The result is more faithful and far cheaper than asking a general-purpose LLM for the GLB directly. What that model does is write a script that generates the mesh; it doesn’t understand the 3D asset.
The third one is about worlds. With Open Code, skills and subagents, we start from an image and generate a navigable 3D world with World Labs (Gaussian splatting), detect the most representative objects, generate them with SAM 3D and add collisions: chairs, benches, street lamps, bins, streets, rooms. Today’s LLMs understand an image well enough for one-off cases, but for fine-grained results the specialised vision models still lead. And in very specific domains, such as industrial machinery or a particular street lamp model, you need to fine-tune with examples, because generic SAM 3D struggles.
Tests first, instructions second
In an agent, part of the behaviour is programmed in text, and hand-writing the instructions and tools gives worse results than starting from behaviour tests. You ask for something like “give me the first frame of this video”, check that it comes back, and keep iterating on the instructions until the test passes, TDD style. Watch for the model cheating by slipping in instructions written just for that test.
These tests come with a bonus: you can switch models or framework versions without fear, and you can test entire workflows too.
For everything else, go hybrid. Add human or team approval points (human in the loop), hooks before any access to sensitive data, and automated tests after code generation. And never give an agent an unrestricted sandbox.
Divide and conquer will keep working
We believe a team of specialised agents performs better than a single giant model, just as you send different people to study different disciplines. We see it in cost, in context limits, in security (your own models for your own business, with data that stays in-house) and in energy use.
Models will keep improving and sometimes one will replace two: Grounding DINO plus SAM became SAM 3 on its own. A piece changes and the architecture stays.
If you want to try it, start with a sequential flow of two or three agents and one behaviour test that defines what “it works” means. Once the happy path passes, measure token spend, and only then consider a handoff.
