What Should a Software Company Do When ChatGPT Invents Product Features?

Why does ChatGPT invent features?

The presupposing question is especially common in support. A customer asks ChatGPT how to do something, gets a confident set of menu steps that do not exist, then opens a ticket saying the product is broken.

Which invented features cause the most harm?

Some invented features are merely annoying. Others create real risk:

• Compliance and certifications, such as an answer saying the product is HIPAA compliant or holds a certification it does not. Buyers may rely on this in procurement.
• Integrations with tools the customer already uses, which often decide a purchase.
• Data residency and security controls, such as regional hosting or single sign-on on a plan that lacks it.
• Mobile apps and offline use, which field teams depend on.
• Pricing-tier features that exist only on a higher plan.
• AI features that competitors have and assistants assume everyone has.

Prioritise fixes by harm. A wrong compliance claim deserves immediate attention, including a clear statement on your security or trust page.

How do you collect the evidence?

• Tag support tickets that mention ChatGPT or another assistant, and ask the customer to share the conversation or a screenshot.
• Ask sales to log prospects who arrive expecting features you do not have.
• Run the questions yourself in ChatGPT, with web search on and off, plus Gemini, Perplexity and Copilot. Use neutral questions ("Does [product] have [feature]?") and presupposing ones ("How do I use [feature] in [product]?").
• Record the sources cited for each wrong claim.
• Separate the error types: an answer citing a wrong source needs a source fix. An answer with no source, or one that misreads a correct source, needs clearer content on your own site.

What should you publish?

Public, crawlable documentation

Documentation behind a login is invisible to AI crawlers and web search, so it cannot inform the answers prospects see. A signed-in customer's own assistant can reach it through an authenticated docs connector they install, such as an MCP server, or a browser agent working in their signed-in session. Publish at least a public overview of each major feature, with plan availability. Put key limits in HTML text rather than screenshots.

A feature and plan matrix

A table of features by plan, in text, answers "does [plan] include [feature]?" directly.

A changelog that marks removals

Record features added, changed and removed, with dates. When you retire a feature, say so plainly and point to the alternative.

A roadmap that is clearly a roadmap

Label planned features as planned, with no implied dates unless you mean them. Avoid "coming soon" language on feature pages.

Integration pages that credit the right product

Where a feature comes from a partner integration, say so: "Time tracking is available through the [partner] integration, which is sold separately."

Why add a “what it doesn’t do” section to prevent AI from inventing product features?

Models fill gaps by inference. If nothing says your product lacks a feature, and most products in the category have it, an answer may assume you do too. An explicit statement gives retrieval-based answers a direct, quotable response to "does it do X?".

A good "what it doesn't do" section:

• Covers the features people commonly assume, based on support tickets, sales calls and the errors you found. It does not need to list everything.
• Is specific: "[Product] does not include built-in payroll" rather than "some features may not be available".
• Offers the alternative: "Most customers connect [payroll provider] through our integration."
• Sits in public documentation and on relevant feature pages, not only in a help centre behind a login.
• Is kept current, so it does not become the next source of outdated information.

It also helps buyers qualify themselves, which reduces churn from mismatched expectations. It will not stop every invented feature. Answers drawn from a model's training data may still guess. But answers based on web search can quote it.

How do you fix third-party sources?

• Review platforms: correct feature checklists on your profiles, and ask the platform to fix errors you cannot edit.
• Integration marketplaces: make sure listings name the partner providing each capability.
• Comparison and affiliate sites: send corrections with evidence, such as a link to your feature matrix.
• Old press and blog posts: ask for updates where they describe features you retired.

The opposite problem, where assistants say you lack a feature you have, is covered in what sources make ChatGPT claim software lacks a feature it has.

How should support handle it?

Create a standard reply that is polite and avoids blaming the customer:

AI assistants sometimes describe features that [product] doesn't have. [Product] doesn't currently include [feature]. Here is what customers usually do instead: [alternative]. Our feature list is here: [link].

Track how many tickets use this reply each month. A falling count is a practical measure of whether your fixes are working.

Also report wrong answers through ChatGPT's feedback option, with a short correction. It will not change answers immediately, but it adds a signal.

What might an illustrative case look like?

Consider an illustrative scheduling software company that sees 40 tickets a month from customers asking how to enable "automatic payroll export". ChatGPT, asked "How do I export payroll from [product]?", gives menu steps that do not exist. Asked neutrally, it says the product "includes payroll export", citing an integration marketplace listing that does not name the partner. The company updates the listing and publishes a public feature matrix. It adds a "what it doesn't do" section stating that payroll export requires a partner integration, and gives support a standard reply. Two months later, search-based answers mostly describe the feature correctly, citing the new matrix, and related tickets have fallen by more than half.

What do people ask about invented features?

Can we make ChatGPT stop inventing features completely?

No. You can reduce it by publishing clear, specific information and fixing wrong sources. Some answers will still guess, especially without web search.

Is it risky to publish what our product doesn't do?

It can feel uncomfortable, but it builds trust and reduces churn. Frame it with alternatives rather than as a list of weaknesses.

Should documentation be public?

At least the feature overview, plan matrix and key limits. Detailed configuration guides can stay behind a login if needed.

How often should we recheck?

Monthly, and after major releases or retirements.