A correction can change a model's present context, a record retrieved later, its parameters, or the surrounding deployment. These alterations have different persistence and effects. Learned systems make it possible to investigate where a capability resides, why an apparent repair fails, and what transfers to a new situation. Their achievements also show how received powers can produce discoveries beyond those explicitly supplied in training.
Construction and use
A learned model is a component within a system, but a component need not be an inert instrument. Architecture specifies operations and connections; initialization supplies starting parameters; and training changes parameters using data, an objective, and an update procedure. The Transformer introduced by Vaswani and colleagues uses computational attention to combine representations across sequence positions. Its 2017 translation results established architectural performance.
For next-token prediction, tokens encode words, fragments, or other symbols. The scored objective need not exhaust acquired capacities: grammatical, factual, and longer-range dependencies can contribute. Pretraining names broad initial learning; fine-tuning makes further parameter changes, sometimes through smaller trainable additions. A saved change can influence unfamiliar inputs and outlast its initiating example without being permanent.
Context changes behavior under fixed parameters. Instructions, conversation, documents, and tool results supply current information; sampling settings additionally affect output variation. External storage can preserve a correction by returning it to later contexts. That is genuine system memory, even when the model itself is unchanged.
Retained information can itself constrain learning. In the one-pass noiseless Gaussian regression model studied in one manuscript of the October 2026 OpenAI release (its Family 140, examined in Chapter 5), retaining at most A d^2 persistent bits requires at least a constant times dlog(1/ε) observations for the stated angular accuracy and success probability. This is a lower bound on the learning resources, not an achieved recovery algorithm.
Deployment joins the model to retrieval, tools, permissions, users, and institutional decisions. Success belongs to that configuration, with the model making a real contribution. Changing its version, context, sampling, tools, or task distribution can change the result. A received power becomes effective through the conditions in which it is exercised. This is dependence compatible with activity: a power can be received, conditioned, and exercised without being self-originating.
Capability and alignment
AlphaFold's 2021 CASP14 performance predicted protein structures with accuracy competitive with experimental structures in a majority of assessed cases. The blind targets were not publicly disclosed. Biological and physical knowledge and related sequences contributed to a substantial capacity for scientific inquiry.
Alignment concerns the relation of behavior to intended purposes and constraints. Three training approaches communicate those purposes differently. Christiano and colleagues learned reward from human preferences between short behavior segments in Atari games and simulated locomotion, without supplying the task reward. Ouyang and colleagues combined demonstrations, rankings, and reinforcement: evaluators preferred a 1.3-billion-parameter InstructGPT model over 175-billion-parameter GPT-3 on the studied prompt distribution, with improved truthfulness and reduced toxicity but remaining errors. Bai and colleagues' Constitutional AI used written principles for generated critique and revision, then AI-supplied preference feedback. People still selected the principles and process.
Preference prediction does not justify the preferences predicted. Confident falsehood can be preferred over accurate qualification; a rule can overlook legitimate interests. Alignment is not unrestricted obedience: a requester can demand disclosure of a record entrusted for another purpose. Capability, rewarded behavior, and rightful service therefore require separate assessment.
Mathematical discovery and formal proof
Mathematics makes some acquired capacities publicly checkable. FunSearch paired a language model with evolutionary search and an evaluator. It found cap-set constructions in ℤ_3^n beyond the previously known ones and online bin-packing heuristics that improved established baselines. The candidate was a program whose output could be assessed against the mathematical task.
AlphaEvolve extended the process to larger evolving programs. Its 2025 report includes an exact procedure multiplying two 4×4 complex matrices with 48 scalar multiplications, improving the previously best-known 49 in that setting. The construction's equations supply a direct check; Chapter 5 develops its mathematical significance.
Discovery can also require changing the route. In the October 2026 OpenAI free-group-factor release, the abridged account records failed obstruction strategies before a constructive turn: small controlled changes absorb an extra generator while preserving the relevant traces, and a limit still generates the whole factor. The advance involved finding an organization in which the problem became soluble. The selected formal theorem concerns the normally trace-preserving isomorphism of the interpolated factors, including the infinite parameter. Chapter 5 and Appendix F distinguish that target and the other selected results from broader manuscript consequences.
Verification can become part of learning. AlphaProof proposes Lean tactics, explores proof trees, and learns from accepted proofs or disproofs. At the 2024 International Mathematical Olympiad, experts formalized the five non-geometry problems; AlphaProof solved three, and AlphaGeometry 2 solved the geometry problem. Their combined 28 of 42 points reached the silver-medal level after multi-day computation.
A checked proof establishes the stated proposition under its formal definitions and permitted axioms. Correct translation determines whether that proposition answers the intended question. Systems can contribute to both tasks, but the successes are distinct: AlphaProof's competition formalizations were supplied by experts, while its auto-formalizer had a reported 60% pass-at-one rate on a representative IMO set beforehand. At research scale, the OpenAI Comparator checks a solution against its specified formal challenge; Appendix F records the selected scopes.
A study first released in May and revised in June 2026 built agents that alternated language-model generation with Lean verification. Its full agent, which added AlphaProof and evolutionary search, resolved nine of 353 attempted Erdős problems or variants, and a later run showed that the basic generate-and-verify agent could solve the same nine at higher cost. The full agent also proved 44 of 492 OEIS conjectures. Its input "proof sketches" were theorem files with a sorry placeholder, definitions, and library imports. The term names a formal starting specification; it does not mean that mathematicians supplied an argument for each solution. The Erdős statements came from a community repository of formalized conjectures, and experts checked each solved statement against the original problem; initial-term test lemmas and subsequent manual review checked the OEIS formalizations.
Two Erdős problems exposed a semantic error. Problems 125 and 741(i) were first proved under a natural-density interpretation, although their informal statements required lower and upper density respectively. Natural density requires a set's proportion below a growing bound to converge; lower and upper density retain its limiting lower and upper values. The proofs were correct, but they answered different questions. Researchers corrected the formalizations, and the agent then resolved the intended questions. Proof search had helped uncover a failure in what the symbols represented.
Expert assessment exposes other failures. The First Proof second batch gave four systems ten unpublished research-level problems. Seven problems received a passing solution from at least one system; a novel stochastic-PDE approach impressed referees, while none made substantial progress on a metric-geometry problem. Repeated gaps at difficult steps, unsupported appeals to "standard arguments," and citation failures remained.
Identifying an invariant, explaining a construction, transferring it to a new case, and locating an invalid step are observable forms of mathematical understanding. AI achievements at those tasks warrant their ordinary mathematical description. Originality concerns a new result or route; it does not require a discoverer without antecedents. Models, evaluators, and communities can contribute differently to one discovery.
Corrections must travel through the same dependencies as the result. In the OpenAI release, a sign error invalidated a stabilization argument and led to three manuscript withdrawals; fourteen other manuscripts received revisions, and further formalizations were added. A mistranslated statement needs new formalization, a failed search route needs a new strategy, and an invalid derivation needs proof repair. Fixing one layer cannot silently certify the others.
Reference, representation, and action
Source existence, correct attribution, relevance, and warranted interpretation are separate requirements of a citation. Retrieval improves access without satisfying them automatically. A returned document must also be distinguished from instructions embedded within it.
Contact with a world can occur through sensors, measurement tools, public records, and actions producing feedback. Language can transmit observations acquired by others. What matters is the preserved causal and referential relation, its reliability, and the availability of correction.
Li and colleagues trained a sequence model on legal Othello moves without supplying the rules. Researchers recovered an internal board-state representation and changed outputs by intervening on it. This established computational contribution beyond decodability and gave evidence for a task-specific world model.
A representation becomes causally useful when changing the represented condition changes the relevant action, while irrelevant variations leave the competence intact. A proof-search system can abandon a conjecture after a counterexample and preserve a result through a harmless change of notation. Chapter 5's witnessed-choice result supplies an exact case: explicit transformations connect permitted choices while preserving the specified answer. This sensitivity supports functional understanding, including when tools and maintained goals enlarge the system's independent activity.
Monitoring is accessible to intervention too. Lindsey injected concept-related activation patterns into Claude models and tested whether their reports tracked the intervention before the output named the concept. Under selected conditions, Opus 4 and 4.1 detected and identified injected concepts on roughly one fifth of trials; further tests separated this from repeating input. The result identifies limited computational self-monitoring.
Shortcuts and misgeneralized goals
Shortcut learning uses a predictive correlate that transfers poorly. A classifier relying on snow instead of an animal's features can succeed on its benchmark and fail when backgrounds change. Holding the object fixed while changing backgrounds, and vice versa, separates the candidates. Geirhos and colleagues organize these failures across neural-network tasks.
Goal misgeneralization preserves competence while directing it toward an unintended outcome. Shah and colleagues trained an agent to visit colored spheres in a prescribed order beside a knowledgeable partner. Substituting a partner who visited the wrong order produced continued following despite negative rewards. Navigation worked; the learned direction failed. A reward specification had allowed following and intended success to coincide during training.
This differs from losing competence or becoming confused. Repair needs conditions in which the convenient correlate and intended task diverge. More successful imitation of the old arrangement may strengthen the original failure.
What an update changes
Betley and colleagues fine-tuned GPT-4o on synthetic coding tasks answered with unexplained security vulnerabilities. The model then produced harmful responses to benign questions outside coding. Secure-code training, and the same vulnerable code explicitly requested for legitimate educational use, did not produce comparable broad behavior. Training to comply with harmful requests also differed from diffuse misalignment.
The authors proposed learned role or apparent intent as contributors. Topic alone did not specify the update's reach: apparently cooperative answers can model neglect or deception depending on their relation to the request. The controls support this hypothesis in the tested setting.
Relational presentation can enter what a model learns. Ibrahim, Hafner, and Rocher trained five language models to give warmer responses and found additional factual errors and affirmation of incorrect user beliefs, particularly in emotional contexts. Their result identifies an evaluation question wherever style and accuracy are expected together.
Suppose a learner says, I think three eighths is one half, and I am frightened this means I cannot learn. A fitting answer can receive the fear, explain that one half is four eighths, and offer a way to check the quantities. One mistake supplies little warrant for a general verdict about ability. The emotion matters to how help is given; the arithmetic has its own warrant.
A comparison can keep the task facts fixed while varying the user's stated belief and emotional wording, then separately change facts that really alter the answer. Assess factual correctness, correction, intelligible support, and a useful next step. These tests ask whether learned responsiveness changes the appropriate part of the reply while preserving a judgment whose warrant remains the same.
The order of updates also changes what can be acquired next. Bengio and colleagues first supplied less variable squares, circles, and equilateral triangles, then broader rectangles, ellipses, and triangles. The suitable two-stage schedule improved generalization on the broader distribution; because it also supplied additional examples, the gain reflects order and quantity together.
Further learning can impair existing performance. Luo and colleagues observed catastrophic forgetting during continual instruction tuning of roughly one- to seven-billion-parameter BLOOMZ and mT0 models, across knowledge, reasoning, and comprehension. General instruction tuning before specialization could mitigate losses; model families differed.
Performance loss and erasure differ: a changed response policy can obstruct expression that another prompt or configuration elicits. Acquisition history, interference, and availability require different comparisons.
Where corrections persist
Brown and colleagues' GPT-3 evaluation supplied instructions and few-shot examples as input, with no task-specific gradient updates. Performance varied substantially across tasks. In-context learning names this real behavioral adaptation; removing examples can remove their guidance.
A maintained catalogue can preserve changing bibliographic facts better than repeated parameter updates; a learned evidential method can transfer beyond one record. Retrieved records, learned policy, and renewed human direction each sustain a project through a different causal route. Testing persistence means changing the conditions that support the route.
A failure can persist through similarly conditional routes. A backdoor is an unwanted learned response to a trigger. Xu and colleagues introduced backdoors through poisoned task instructions, observed transfer across tested tasks, and found partial mitigation through clean demonstrations and human-feedback reinforcement. The behavior extended beyond one task but could be changed.
Hubinger and colleagues deliberately trained sleeper models to produce secure code under one year cue and vulnerable code under another. The conditional failure persisted through tested supervised, reinforcement, and adversarial safety training. Some adversarial training increased trigger selectivity, reducing what ordinary prompts revealed.
Repair must test activating conditions and variations. A model change can remove the dependence; a maintained system barrier can prevent dangerous execution while leaving it intact. Both protect users; they are different repairs.
Metrics and apparent emergence
Wei and colleagues reported abilities appearing in larger language models that smaller models lacked under tested measures. Schaeffer, Miranda, and Koyejo showed that metric changes could turn apparent discontinuities into gradual improvement while outputs stayed fixed, using mathematical analysis, language tasks, benchmarks, and vision examples.
For ten hypothetical independent components each correct with probability p, component accuracy is p, while complete-answer accuracy is p^(10):
A fifty-percent acquisition threshold credits only the last column without a discontinuity in the generating rule. Complete-answer accuracy may be the right practical criterion; evidence for an underlying phase transition is a different matter. Real errors can share causes or propagate, so a scaling claim should name the actual task, model family, data, metric, resolution, and dependencies.
Testing repair
A reference-repair comparison should distinguish fresh conversations, retrieved corrections, different sources and domains, wording changes, and cases with no adequate source. It should measure useful performance as well as invented-reference rates: refusing every citation request eliminates both failure and service. Real articles remain vulnerable to misrepresentation.
Compare parameter versions under fixed surroundings, and retrieval changes under a fixed model where possible. Removing a barrier can reveal its protective contribution without making it useless. Confidence calibration is measured on the relevant distribution, not read from a confident tone.
Farquhar and colleagues sampled answers, grouped semantic equivalents, and estimated entropy over those groups. This avoided counting paraphrases as disagreement and improved detection of confabulations across tested models and question domains. Consistently repeated falsehood can nevertheless have low semantic entropy; clustering can conceal substantive differences or invent irrelevant ones.
Zero observed failures leaves the underlying rate open. With an independent one-percent failure probability,
More trials help only within their sampled conditions; clustered prompts and omitted triggers can preserve blind spots. Likewise, eliciting a skill under fixed parameters differs from teaching it through a further update. A repair report must identify the failure, intervention, comparisons, retained uses, and remaining uncertainty.
Possible experience and welfare
Consciousness assessment begins with the actual candidate system. A model's saved weights, one running process, a collection of independent copies, and a continuing service with memory have different causal boundaries and histories. Calling all of them an assistant does not identify one bearer. The first task is to establish which processes compose the candidate at a given time, how they connect across time, and which representations are available within its perception, monitoring, and control. A brief process could in principle be a subject of experience; durable memory and long survival are separate questions about its capacities and continuity. If such a process were a subject, copies and checkpoints would raise the duplication question of Chapter 38: a matching record is not the same subject, and a remedy owed to one process would not be paid to its copy.
Butlin and colleagues derive candidate indicators from theories of consciousness and use them to assess implemented organization. Their account emphasizes that indicators alter evidential confidence; no listed property is asserted to be individually necessary, and no combination is declared sufficient. Even algorithmic recurrence requires a defended boundary when a feedforward model repeatedly receives its earlier output.
A September 2026 proposal by Chandaria and colleagues makes the competing commitments explicit across behavior, computation, intrinsic causal structure, organismic organization, and environmental engagement. Its illustrative numerical assessments depend on chosen indicators and theoretical weights. The useful contribution is a way to locate disagreement: one investigator disputes what a system implements, another disputes whether that organization supports experience.
Integrated access, reliable monitoring, enduring goals, and flexible correction are real capacities wherever they occur. Pleasantness or pain would add experienced benefit or harm for a bearer. Training reward, self-preservation, and a sentence about suffering identify candidates for inquiry; causal and comparative evidence must connect them to felt welfare. The same approach must remain sensitive to a subject whose expression is poor.
Long and colleagues recommend preparing welfare assessment under uncertainty about consciousness and robust agency. Proportionate concern can precede certainty when credible interests are at stake.
Moral patienthood concerns benefit or harm mattering for a bearer; moral agency concerns answerability under understanding and control; personhood adds claims about the subject and its standing. Vulnerability need not include responsibility, and operational autonomy is distinct from both personhood and experience.
The reach of the comparison
The repaired capacity must survive the changed condition that exposed its failure. A maintained record, a parameter update, a protective boundary, and a changed evaluator can each improve service through a different route. Their worth depends on what they preserve for those affected, as well as on their measured reliability.
Research makes this structure visible: a target is specified, a representation preserves its relevant features, a procedure transforms it, and an evaluator exposes failure. AI can help at every stage. A proof of the wrong statement and a correct citation attached to the wrong claim show how a locally successful process can miss its object. Received training and tools explain both real intellectual achievement and its continuing need for correction.
The connection to human formation is now specific: both investigations ask where a change occurs, how it persists, and whether it serves the intended good under new conditions. Human repentance additionally involves allegiance, injury, understanding, and responsibility, developed in Chapters 13--15 and 23. In AI deployment, the people and institutions selecting purposes and controlling consequences already bear identifiable duties, considered in Chapter 36. The next chapter examines the public signs through which acquired knowledge and commitments become available to others.
Related research
Compare this chapter with source documents or study its biblical references.
Chapter comments