How to Track Whether AI Agents Choose Your Developer Tool Over Competitors

Where do AI agents choose developer tools?

Agents make tool choices in three different situations, and each needs its own tracking:

The second situation is the one most developer tools miss. When a developer asks an agent to "add payments", the agent may choose, install and write code for a provider without presenting options. The developer may keep whatever works.

How do AI agents read product documentation and pricing to recommend tools?

When an agent is unsure or needs current details, it searches the web and fetches documentation pages, much as a developer would. It then relies on what it finds:

• Quickstarts and code samples it can adapt directly, ideally for the exact framework in the project.
• Current package names, versions and API signatures, since outdated examples produce broken code.
• Pricing and limits, especially whether there is a free tier that suits a prototype.
• Clear statements of what the tool does and does not support, which help it decide fit.

Documentation that is rendered only with JavaScript, split across many tabs, or hidden behind a login is harder for agents to read. Many AI fetchers do not run scripts. Some developer tool companies also publish plain markdown versions of their docs or an llms.txt index for AI tools. That can help tools that look for them, although Google says its Search does not use such files.

How do you build an agent choice benchmark?

1. Pick 15–30 tasks where your category is a natural choice, written as a developer would phrase them. Include tasks that do and do not mention your category by name.
2. Prepare small sample projects in the frameworks your users use, so agents work in a realistic context.
3. Choose the agents to test: the coding assistants and agentic tools your users mention, plus general assistants with web search.
4. Run each task three to five times per agent in a fresh environment, since choices vary between runs.
5. Record the outcome: which tool was chosen, whether it was installed and used, whether the code worked, and whether alternatives were mentioned.
6. Save the transcripts and diffs so you can see why each choice was made.
7. Repeat monthly, and after major releases of your docs or of the agents.

A results table might look like this:

"Code worked" matters as much as "chosen". An agent that picks your tool but writes code against an old API version creates a poor first experience.

What signals show agents are choosing your tool in the wild?

• Documentation logs: requests from AI fetchers and agent user agents to your docs, and which pages they read.
• Sign-up patterns: accounts created through the CLI or API, API keys generated shortly after install, or projects created with agent-typical names and settings.
• Package installs: download trends for your SDKs, especially for framework-specific packages.
• MCP server usage, if you publish one: installs and tool calls.
• Support tickets and community posts mentioning code written by an AI agent, including errors caused by outdated examples.
• Self-reported attribution: add "an AI coding assistant chose or suggested it" to your onboarding survey.

None of these is complete on its own. Together with the benchmark, they show whether agents are driving adoption and where they go wrong.

Why do agents choose competitors?

• Training data familiarity. Agents default to tools that were widely used and discussed when they were trained. Newer tools start at a disadvantage.
• Clearer examples. A competitor with a copy-paste quickstart for the exact framework is easier for an agent to use.
• Free tier and simpler setup. Agents prefer tools that work with minimal configuration and no payment step.
• Existing stack. If the project already uses a related service, agents often stay within that ecosystem.
• Third-party content. Tutorials, comparison posts and GitHub examples that feature a competitor feed both training data and search results.

Reading the transcripts usually shows which of these applies.

What improves your odds of being chosen?

1. Framework-specific quickstarts for the stacks your users build with, each complete and copy-pasteable.
2. Current, versioned examples, with deprecated patterns clearly marked.
3. Plain, crawlable docs: server-rendered pages, stable URLs and, if you like, a markdown version for AI tools.
4. Transparent pricing and limits, including what the free tier covers.
5. Clear capability statements: what you support and what you do not.
6. An MCP server or well-described API for agents that call tools directly.
7. Helpful error messages that tell an agent, or a developer, how to fix a problem.
8. Public example repositories and tutorials that show your tool in common stacks.
9. Presence in comparison articles and community answers for your category.

Avoid trying to manipulate agents with hidden instructions in documentation. That is a form of prompt injection: agents and their developers treat it as an attack, and it damages trust. Inaccurate competitor claims in AI answers are a separate problem, covered in how AI search tools can falsely report that competitors support a feature. Manual prompt tracking for chat assistants is covered in how to manually track AI visibility by testing prompts across AI engines.

What do people ask about AI agents and developer tools?

Can I see which agent sent a developer to my docs?

Sometimes, from user agents and referrers in your logs, but many agent requests are not labelled consistently. Treat log data as a partial signal.

Does publishing an llms.txt file help coding agents?

Some AI tools look for llms.txt or markdown docs, and it costs little to provide. Google Search does not use it. The evidence for its effect is thin, so prioritise clear, crawlable docs first.

How often do agents' tool preferences change?

With each model update and as new content appears. Rerun your benchmark after major agent releases.

Should we build an MCP server?

If agents are likely to use your service directly, yes. It gives them a reliable, described interface rather than guessing from docs.