Generative AI depends on an information ecosystem it may also disrupt. Publishers, researchers, programmers and other creators produce the material that helps make models useful. Yet AI services can compete with those same sources for attention and revenue, while flooding the web with generated content that may enter future training datasets.

That tension sits at the heart of David Gerard’s critique of the AI industry. His warning connects three issues often discussed separately: model collapse, copyright liability and the economics of original work.

The connection deserves attention. But it also requires distinctions. An odd chatbot response is not proof of model collapse, a copyright ruling is not a universal verdict on AI training, and rapid revenue growth does not establish a sustainable business.

What model collapse actually means

Model collapse describes a failure mode in which repeatedly training models on model-generated data can erode the quality and diversity of what they learn. Uncommon patterns can disappear, errors can become reinforced, and successive generations can drift away from the original data distribution.

Imagine a collection of accounts describing the same town. The original material includes familiar landmarks, obscure local history and conflicting perspectives. If a model generates simplified versions, and later models learn primarily from those versions, the most common details may survive while unusual but valuable information fades.

This is more specific than a chatbot getting worse. Repetitive language, strange fixations or nonsensical answers can also arise from changes to instructions, reinforcement learning, sampling settings or software defects.

Gerard interprets some unusual chatbot behavior as a possible sign of collapse. That is a hypothesis, not a demonstrated diagnosis. Establishing the cause would require evidence about training data, model changes and controlled evaluations—not simply screenshots of peculiar answers.

Synthetic data is not automatically bad data

The important distinction is not simply human-written versus machine-generated. It is whether the material is accurate, diverse and meaningfully checked.

Synthetic examples can be useful where there is an independent way to verify them. Generated programming exercises can be checked with tests. Some mathematical answers can be evaluated with formal tools. Simulations can produce examples under specified conditions.

Open-ended prose is harder to validate. A fluent explanation may repeat an unsupported claim or omit an uncommon perspective without triggering an obvious error signal.

• Verification matters: A second model agreeing with the first is weaker evidence than an independent test or authoritative source.
• Coverage matters: A large dataset can still contain a narrow range of examples.
• Provenance matters: Knowing where material originated helps identify duplication and recursive reuse.
• Evaluation matters: Improvements should be measured against independent tasks, not merely examples resembling the training material.

Synthetic data is therefore neither a guaranteed escape from data scarcity nor a guaranteed route to collapse. Its value depends on how it is generated, selected and used.

The economic “doom loop” is a separate problem

Technical model collapse concerns what happens inside the training process. An economic feedback loop concerns what happens to the people and institutions producing new information.

The risk is straightforward: an AI service answers a question using knowledge derived from published work; users have less reason to visit the original source; the source loses revenue; and fewer resources remain for reporting, testing, documentation or research.

That outcome is not inevitable. Summaries can sometimes introduce readers to sources, and licensing can create revenue. The crucial question is whether these benefits compensate for the attention and income displaced.

Repackaging existing knowledge is not the same as funding its continued production. A system can become better at answering questions while the underlying information ecosystem becomes less capable of discovering new answers.

Copyright is not one yes-or-no question

Copyright litigation could change the economics of training, but sweeping conclusions are risky. In the United States, fair use depends on the facts and circumstances of a particular use.

Courts consider four statutory factors: the purpose and character of the use, the nature of the copyrighted work, the amount used, and the effect on the work’s potential market. Whether a use is transformative is part of that analysis, not a substitute for it.

Several questions also need to be kept separate:

• How were copies of the source material obtained?
• Was using that material for training lawful?
• Does the system reproduce protected expression in its outputs?
• What market effects arise from the particular use?

Scientific research is not automatically exempt from copyright, just as commercial use is not automatically disqualified from fair use. A ruling about one dataset or product does not necessarily settle the status of every other model.

If licensing becomes more important, it could compensate some creators while increasing barriers to entry. Large developers may be better positioned than smaller competitors to negotiate extensive rights agreements. And a payment resolving past infringement claims is not automatically a license for future use.

Growth figures cannot answer the sustainability question

Gerard’s broader argument is that pressure for rapid expansion makes these problems harder to address. That is a critique of incentives; assessing any particular company still requires reliable financial evidence.

Readers should distinguish annual revenue from an annualized run rate. A run rate extrapolates recent performance; it does not show what a company earned over a completed year. Likewise, debt, contractual spending commitments and contingent liabilities are not interchangeable categories.

A service can also generate a positive margin on serving requests while its developer remains unprofitable after training, research, staffing and other expenses. Any claim of profitability needs a clear explanation of which costs are included.

What a healthier AI data economy would require

A more durable approach would treat reliable information as an ongoing input worth maintaining, rather than a resource collected once and reused indefinitely.

That means clearer data provenance, independent quality testing, licensing terms that specify permitted uses, and compensation arrangements that support new work. Attribution helps readers find sources, but credit alone does not pay for an investigation or maintain technical documentation.

The central question is not whether every chatbot anomaly signals collapse. It is whether AI services can remain useful while sustaining access to fresh, trustworthy information. Better models and a healthier information ecosystem should be complementary goals. Treating them as separate problems risks weakening both.

---

This article was inspired by ‘This smells like model collapse’ | David Gerard from The Tech Report. Please visit the original video for the creator’s full presentation and context.