How to Batch-Test 20 AI Visibility Queries With the OpenAI API Instead of Buying a Tool

When does a DIY script make sense?

A script is a good fit when:

• you track a small set of questions, roughly 10–50, in one or two markets;
• someone on the team can maintain a short Python or JavaScript script;
• you want the raw answers and full control over method;
• you mainly need ChatGPT-style answers, possibly adding other engines later.

A paid tool makes more sense when you need many locations, many engines captured from their consumer interfaces, client-ready reports, or when nobody can maintain code.

What does the script need to do?

Six parts:

1. Read the questions from a file, each with an ID and exact wording.
2. Call the API with web search enabled, several times per question.
3. Save every raw response with date, question ID, run number and model.
4. Extract citations: the URLs the answer cites.
5. Detect brands: your brand and competitors, using name variants.
6. Write a summary per question: runs that named each brand, and cited domains.

What does a minimal example look like?

A minimal Python sketch using the OpenAI SDK's Responses API with the web search tool. Model names and tool options change, so check OpenAI's current documentation before running it:

import csv, json, re, datetime
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment
MODEL = "your-current-model"  # a model that supports the web search tool
RUNS = 5
BRANDS = {"us": [r"\bAcme Payroll\b", r"\bAcme\b"], "rival_a": [r"\bPayWise\b"]}

def run(question):
    resp = client.responses.create(
        model=MODEL,
        tools=[{"type": "web_search"}],
        tool_choice="required",  # search on every run
        input=question,
    )
    urls = []
    for item in resp.output:
        if item.type == "message":
            for part in item.content:
                for ann in getattr(part, "annotations", None) or []:
                    if ann.type == "url_citation":
                        urls.append(ann.url)
    return resp.output_text, urls

today = datetime.date.today().isoformat()
with open("questions.csv") as f, open(f"answers-{today}.jsonl", "w") as out:
    for row in csv.DictReader(f):
        for i in range(RUNS):
            text, urls = run(row["question"])
            named = [b for b, pats in BRANDS.items()
                     if any(re.search(p, text, re.I) for p in pats)]
            out.write(json.dumps({"date": today, "id": row["id"], "run": i + 1,
                                  "model": MODEL, "named": named,
                                  "urls": urls, "answer": text}) + "\n")

The output is one line per answer, which you can load into a spreadsheet or database. Keep the raw answers: they are the evidence behind every number and let you re-classify later.

How do you turn raw answers into a report?

From the saved answers, calculate per question and overall:

• Mention rate: runs naming your brand ÷ total runs, such as "3 of 5".
• Competitor rates on the same questions.
• Cited domains: counts per domain, and whether your domain appears.
• Changes since last period, flagged only when larger than the usual run-to-run swing.

Then read a sample of answers each month to judge whether mentions are recommendations, neutral mentions or negative. Automated name detection cannot tell the difference reliably. How to classify is covered in the difference between being cited and being recommended.

How do you run it every day to check citations?

Schedule the script with cron, a CI pipeline such as GitHub Actions, or a cloud function. For daily checks:

• run fewer repeats per question each day, for example two, and combine them into weekly totals for reporting;
• alert only on specific events, such as your domain disappearing from citations on a key question for several days in a row, or a known wrong fact appearing;
• store results in a small database such as SQLite, so trends are easy to query.

Daily single runs are noisy on their own; the value comes from combining them.

What does it cost to run?

Costs come from two parts: model tokens for each answer, and a separate charge for each web search tool call, both priced by OpenAI and changing over time. The volume is easy to calculate: 20 questions × 5 runs × 4 weekly runs = 400 calls a month. For a set this size, costs are usually modest compared with a subscription, but check the current price of both tokens and search calls before you commit, and set a spending limit on the API account.

How do you extend it to other engines?

The same pattern works with other providers' APIs, each of which returns sources in its own format:

• Anthropic's API offers a web search tool for Claude, with citations in the response.
• Google's Gemini API supports grounding with Google Search, but its terms forbid storing, analysing or monitoring grounded results, so it cannot be used for this kind of batch tracking.
• Perplexity's API returns answers with their source links.

Add one small function per provider that returns the answer text and cited URLs in the same shape, and keep the rest of the script unchanged. Keep API keys in environment variables or a secrets manager, never in the script, and respect each provider's rate limits and terms. Capturing consumer apps by automated scraping usually conflicts with their terms, which is one reason tools that do it charge more.

Where do API results differ from what users see?

This is the most important limitation to understand:

• No personal context. API calls have no memory, chat history or custom instructions.
• Different instructions. The consumer apps add their own system instructions and features on top of the model.
• Search behaviour. In both the app and the API the model decides whether to search and what to search for. In your script you decide only whether search is available; set tool_choice to "required" if you want every run to search, and record whether each run searched, so answers without sources are not counted as lost citations.
• Location. Local answers depend on location. Pass an approximate location where the API supports it, or name the place in the question, and still expect differences from the app.
• Model versions. The app may use a different model or routing from the one you call.

So treat API results as a consistent, repeatable measurement, not as an exact copy of the ChatGPT app. Check a sample of questions by hand in the app each month; if the two diverge badly for your category, especially for local questions, a tool that captures the app interface may be worth paying for. For combining approaches, see how to track AI visibility without checking every model manually.

What else should you get right?

• The questions. A script is only as good as its question set; see how to choose the right prompts to track.
• Brand detection. Include product names, abbreviations and common misspellings, and check matches by hand at first to avoid false positives from similar words.
• Versioning. Record the model name and any change to questions or settings, so trends stay comparable.
• Errors and retries. Log failed calls and retry them, so gaps do not look like drops in visibility.

What do people ask about DIY AI visibility testing?

Is the OpenAI Batch API a good fit for this?

It can cut costs for large volumes of requests that do not need an immediate answer. Check whether the features you need, such as the web search tool, are supported in batch mode before relying on it.

Can I test Google AI Overviews with an API?

Not directly through a Google API for Overviews. Tools that track them usually capture search results pages. For a small set, manual checks are often enough.

How many runs per question should the script use?

Three to five per engine per period is a practical minimum. Use more for questions you report on individually.

Should I build this or buy a tool?

Build if the question set is small, you have someone to maintain it, and API answers are close enough to the app for your category. Buy if you need scale, many locations or consumer-interface capture.