Should I Use Temporary ChatGPT Chats for an Unbiased Brand Visibility Test?

What is memory personalization in ChatGPT, and why does it matter for businesses?

ChatGPT can remember things about a user in two ways. Saved memories are details the user asks it to remember, or that it decides are useful, such as a job, a company or preferences. It can also refer to past conversations. Memory is available on free plans as well as paid ones, so ordinary customers' answers can be personalized too. For businesses, this matters in two directions:

• For your tests: if you or your team often ask ChatGPT about your own company, it may bring your brand into answers because it knows your interest. Your tests then look better than what customers see.
• For your customers: a buyer who has used your product or discussed it before may see you more often than a new buyer does. Personalized answers mean there is no single "true" answer, only typical ones.

Same conversation versus fresh session for testing AI answer repeatability

Use a fresh session for every run. In a single conversation, earlier questions and answers shape later ones. If you ask "best CRM for agencies", then "tell me about [your brand]", then repeat the first question, the brand is now in context and more likely to appear. Earlier answers also anchor later ones, so repeated runs in one conversation understate how much answers vary. Fresh sessions, one prompt each, match how a new buyer would ask. Keep multi-turn conversations for deliberate journey tests, where following one conversation is the point.

Can Custom Instructions make an AI audit unreliable?

Yes. Custom instructions tell ChatGPT about the user and how to respond, such as "I work in marketing at [company]" or "prefer UK suppliers". They can change which brands appear and how answers are framed. A tester with instructions mentioning their employer is effectively running a branded test. Turn custom instructions off in the account used for testing, or use unpersonalized temporary chats. Also check for connected apps or plugins that might add context.

How do you set up a fair ChatGPT visibility test?

1. Use a dedicated test account with memory, chat history reference and custom instructions switched off, or use unpersonalized temporary chats.
2. Start a fresh chat for each run of each prompt.
3. Fix location and language. ChatGPT may estimate location from your IP address, so test from the region your customers are in.
4. Record the model and whether web search was used.
5. Run each prompt several times, since answers vary between runs even with no personalization.
6. Use the same prompts each time and keep branded prompts separate from category prompts.
7. Automate through the API if you need scale. OpenAI's terms prohibit automated extraction from ChatGPT itself, and API answers can differ from the app, so spot-check the app by hand.

How many runs are needed is covered in how many times to test a prompt in ChatGPT or Perplexity to verify visibility.

Are logged-out tests better?

Logged-out ChatGPT has no memory or instructions, which removes personalization. But logged-out users may get different models or features from logged-in users on paid plans. Use logged-out tests as one reference point, alongside tests on the plans your customers typically use.

What might an illustrative case look like?

Consider an illustrative project management software company whose marketing team reports that ChatGPT names the company in 80% of "best project management tool for agencies" answers. The team ran the tests in their own accounts, where saved memories note their employer and they had asked about their product many times. Rerunning the same prompts in unpersonalized temporary chats, with fresh sessions and ten runs each, gives an appearance rate of 35%. The team adopts the unpersonalized method for all reporting, and uses its own accounts only to see how returning users might experience ChatGPT.

What do people ask about testing ChatGPT fairly?

Are temporary chats private?

They do not appear in history or create memories, and they are not used to improve OpenAI's models. OpenAI may keep a copy for up to 30 days for safety.

Does the paid plan change results?

It can, because plans may give access to different models and features. Record the plan and model with each test.

Should I also test other assistants this way?

Yes. Gemini, Claude and others have their own memory and personalization settings. Use clean sessions and fixed settings in each.