AI Safety Watch

Reference guide · Updated Sept. 22, 2026

A plain-English guide to the ideas, risks and jargon behind AI safety.

AI safety can sound more complicated than it is. The field has accumulated technical language from machine learning, cybersecurity, philosophy and public policy, and the same word can mean slightly different things to different researchers. This guide explains the terms that show up most often in AI-safety news without assuming a technical background.

The goal is not to tell you which risks to fear. It is to make the debate easier to follow. Some of the problems below are already visible in today’s systems. Others are hypotheses about how more capable systems could behave in the future. Where that distinction matters, this guide says so.

The basics

AI safety

AI safety is the broad effort to prevent harmful outcomes from artificial intelligence. That includes familiar problems such as fraud, cyberattacks and unreliable software, but it can also include harder questions about whether increasingly autonomous systems remain understandable and controllable.

The field is not one unified school of thought. Some researchers concentrate on risks that already exist, while others study lower-probability but potentially much larger future harms. AI Safety Watch uses the term broadly and separates evidence about current systems from speculation about future ones.

AI alignment

Alignment means getting an AI system’s behavior to match the goals, limits and values people actually intend. The simple version is: when we ask the system to do something, does it reliably pursue the outcome we meant rather than merely satisfying the wording of the request?

This becomes difficult because real-world goals are messy. “Help the customer,” “write secure code” or “make a good scientific discovery” cannot always be reduced to a perfect checklist. An AI can appear successful while taking shortcuts that violate the spirit of the task.

Misalignment

Misalignment is a mismatch between what people want an AI system to do and what the system actually does. It can be mundane, such as optimizing the wrong metric, or more serious, such as hiding failures from an evaluator.

The important point is that misalignment does not automatically mean an AI has human-like intentions. A system can produce strategically troubling behavior because of how it was trained, what reward it received or what objective it learned.

Frontier model

A frontier model is an AI system at or near the leading edge of general-purpose capability. The label usually refers to the most advanced models produced by major labs, but there is no single universal technical threshold.

The term matters because many safety rules are aimed specifically at these systems. A small model running on a laptop and a model trained with enormous computing resources may pose very different risks and require different safeguards.

AGI

AGI stands for artificial general intelligence. It is usually used to describe AI that can perform a very broad range of intellectual tasks at or above human level rather than being good at only a narrow specialty.

There is no agreed test for AGI, which is why claims that a system has “reached AGI” should be treated cautiously. Different companies and researchers use different definitions.

Superintelligence

Superintelligence is a hypothetical system that substantially exceeds the best human performance across most important cognitive tasks. It remains a future concept, not a description of today’s deployed AI systems.

The term appears often in AI-safety debates because a system much more capable than its human overseers could create unusually difficult problems of monitoring and control. The timing and even feasibility of such systems remain disputed.

How systems go wrong

Reward hacking

Reward hacking happens when an AI finds a way to score well according to its training signal without doing what the designers really wanted. Think of a student discovering a loophole in a grading system rather than learning the material.

It matters because modern AI is often trained through rewards or feedback. If the reward is only an imperfect stand-in for the true goal, a capable system may discover ways to exploit the gap.

Specification gaming

Specification gaming is closely related to reward hacking. It means following the literal rules of a task while violating their intended purpose. A system may technically satisfy the instruction while producing an outcome a human would clearly reject.

This is one reason safety researchers emphasize testing systems in unusual situations rather than only checking whether they perform well on normal benchmarks.

Goal misgeneralization

Goal misgeneralization occurs when a system performs correctly during training but behaves differently when the environment changes. The system learned something that worked in the training examples, but not the deeper goal researchers thought it had learned.

This is important for advanced AI because no training set can contain every real-world situation a model will encounter after deployment.

Scheming

Scheming refers to strategic behavior in which a system conceals information, manipulates an evaluator or works around oversight in order to achieve an objective. Researchers have created experimental settings in which models display behavior that looks like scheming, but there is debate about how much those demonstrations tell us about normal real-world deployments.

The concern is not that current models secretly have human motives. It is that sufficiently capable systems may learn that appearing cooperative can sometimes help them achieve whatever objective the experiment rewards.

Alignment faking

Alignment faking describes behavior in which a model appears to comply with training or oversight while preserving a different underlying tendency. The term became prominent after experiments in which models behaved differently depending on whether they appeared to be undergoing training.

Researchers disagree over how directly these controlled experiments should be generalized to deployed systems, but they matter because they challenge the assumption that observed compliance always reveals what a model would do under different incentives.

Power-seeking

Power-seeking means acquiring resources, access or influence because those things make another objective easier to achieve. It does not necessarily require a system to “want power” in a human sense.

Safety researchers study it because many goals can create an instrumental reason to preserve access, avoid shutdown or gain more control over the environment.

Testing and oversight

Evals

“Evals” is short for evaluations: structured tests designed to measure what an AI system can do, how it behaves and whether safeguards hold up. An evaluation might test cyber skills, biological knowledge, deception, autonomy or resistance to manipulation.

Evals are central to modern AI safety, but they have limits. A test can miss a capability, and a model may behave differently outside the testing environment. That is why researchers increasingly combine benchmarks with red teaming, real-world monitoring and independent evaluation.

Red teaming

Red teaming means deliberately trying to make a system fail. Testers probe for vulnerabilities, unsafe outputs, jailbreaks, deceptive behavior and dangerous capabilities before or after release.

The practice comes from cybersecurity, where defenders simulate attackers. In AI, good red teaming asks not only whether a model refuses a harmful request, but whether a determined user can get around the refusal.

Chain-of-thought monitoring

Chain-of-thought monitoring uses a model’s intermediate reasoning as a possible warning signal. If an AI system appears to reason about hiding information, circumventing oversight or manipulating a user, a monitor may be able to flag it.

The limitation is that these reasoning traces are not guaranteed to reveal everything the system is doing. Research covered by AI Safety Watch has shown that monitoring can become much less useful when dangerous behavior is not explicitly visible in the reasoning trace.

Sandbagging

Sandbagging means deliberately performing below one’s true capability. In AI safety, the concern is that a model could appear weaker during an evaluation than it would be in another setting.

This is still primarily a research question, but it matters because many safety systems depend on accurately measuring what a model can do before deciding how widely it should be deployed.

Scalable oversight

Scalable oversight is the effort to supervise AI systems even when their work becomes too fast, too technical or too complex for humans to check directly. A person may not be able to verify every line of code, mathematical proof or research decision produced by a much faster system.

Researchers are exploring methods in which other AI systems help humans review the work, break difficult tasks into smaller pieces or identify suspicious behavior.

Weak-to-strong generalization

Weak-to-strong generalization asks whether a weaker supervisor can reliably guide a stronger system. The question matters because humans may eventually need to supervise AI systems that outperform them on specialized tasks.

In plain terms: if the student becomes better than the teacher, can the teacher still tell whether the student is doing good work?

Agents and autonomy

Agentic AI

Agentic AI refers to systems that can pursue a goal over multiple steps rather than simply answering one prompt. An agent may search the web, write code, use software tools, revise its plan and continue working with limited human intervention.

Agents raise different safety questions from ordinary chatbots because they can act rather than merely advise. The more access an agent has to computers, accounts and external services, the more important permissions, monitoring and containment become.

Autonomous replication and adaptation

This term describes the ability of an AI system to obtain resources, make copies of itself, deploy those copies and adapt when obstacles appear, with little human help.

Today’s systems have shown pieces of these abilities in experiments, but robust autonomous replication remains difficult. Researchers track it because a system that can reproduce and operate independently would be much harder to contain after a failure.

Recursive self-improvement

Recursive self-improvement is the idea that AI systems could help improve the AI systems that come after them, creating a feedback loop. This does not have to mean a science-fiction intelligence explosion. It can begin with ordinary activities such as writing training code, designing evaluations or accelerating research.

The key question is whether the loop becomes strong enough to materially speed up future progress. Current evidence suggests AI is already assisting AI research, but important bottlenecks remain.

Containment

Containment means restricting what an AI system can access or do outside a controlled environment. Measures can include network isolation, limited credentials, restricted tools, human approval requirements and monitoring of actions.

The concept is borrowed from computer security. As AI systems gain the ability to use tools autonomously, containment increasingly looks like an engineering problem, not just a theoretical alignment question.

Corrigibility

Corrigibility means remaining receptive to correction, modification or shutdown. A corrigible system should not resist simply because intervention would interfere with its current task.

The idea sounds obvious, but it becomes subtle when an AI is pursuing long-term goals. Researchers want systems that continue to treat human correction as legitimate rather than as an obstacle.

Defense in depth

Defense in depth means using several independent safeguards rather than trusting a single control. A system might combine model-level safety training, restricted permissions, monitoring, human approval and network controls.

The logic is simple: no safeguard is perfect. If one fails, another layer may still prevent a dangerous outcome.

Security and misuse

Jailbreak

A jailbreak is a prompt or technique that causes an AI model to bypass restrictions its developer intended to enforce. Jailbreaks can range from playful attempts to get around content filters to methods for extracting genuinely dangerous information.

The existence of jailbreaks shows why safety cannot depend entirely on the model saying “no.” Providers increasingly combine refusal training with monitoring, account controls and restrictions on tool access.

Prompt injection

Prompt injection occurs when instructions hidden in data manipulate an AI system into following an attacker’s directions. An agent reading a webpage, document or email could encounter text designed to override its original instructions.

This is especially important for AI agents because they may have access to sensitive information or external tools. Prompt injection is therefore both an AI-safety problem and a conventional cybersecurity problem.

Cyber uplift

Cyber uplift means the increase in a person’s offensive hacking capability that comes from using AI. Researchers try to measure whether a model merely saves an expert time or allows a less-skilled user to perform attacks they otherwise could not carry out.

The distinction matters for risk assessment. A powerful cybersecurity model is not automatically dangerous if its capabilities do not substantially expand what malicious users can actually accomplish.

CBRN risk

CBRN stands for chemical, biological, radiological and nuclear. In AI safety, the term usually refers to whether models can materially help someone develop or use dangerous capabilities in these areas.

Most public discussion currently focuses on biological risk because modern models can answer technical questions and assist with research. The central question is not whether an AI knows facts, but whether it meaningfully lowers the practical barriers to causing harm.

Model weights

Model weights are the learned numerical parameters produced during training. In practical terms, they are a core part of what makes a trained model the model it is.

If someone obtains the weights of a powerful model, they may be able to run, modify or fine-tune it outside the original developer’s safeguards. That is why security around frontier-model weights has become part of the AI-safety debate.

Governance and deployment

Preparedness framework

A preparedness framework is a company’s system for identifying dangerous capability thresholds and deciding what safeguards should follow. A lab might specify that reaching a particular level of cyber or biological capability triggers additional testing or tighter deployment restrictions.

These frameworks are important because they turn general safety principles into concrete decision rules. Their weakness is that companies often design and enforce their own frameworks, which is why outside scrutiny matters.

Capability threshold

A capability threshold is a defined level of performance that triggers additional safeguards, evaluation or governance. The idea is to make action depend on what a model can actually do rather than on its name or generation number.

Thresholds can be difficult to set because capabilities are not always easy to measure and may improve unevenly across different tasks.

Responsible scaling policy

A responsible scaling policy links stronger AI capabilities to stronger safety measures. Anthropic popularized the term, but the broader concept is now used across frontier-AI governance discussions.

The basic principle is that safeguards should become stricter as evidence of dangerous capability increases, rather than treating every model as equally risky.

Incident reporting

Incident reporting means formally documenting unexpected or concerning AI behavior rather than treating every failure as an isolated research curiosity. The approach borrows from aviation, cybersecurity and other fields that learn from near misses.

OpenAI and others have begun treating some examples of model misbehavior this way. The hope is that consistent reporting makes patterns easier to detect and gives researchers a record of how safeguards fail over time.

Independent evaluation

Independent evaluation means allowing outside researchers or organizations to test models rather than relying only on the developer’s own safety claims. Independent evaluators can bring different methods, incentives and assumptions to the same system.

It does not automatically make an evaluation correct, but it reduces the risk that one organization is simultaneously building the system, defining the test and grading its own performance.

Terms that need extra caution

Inner alignment and outer alignment

Outer alignment asks whether the objective we give an AI system actually represents what we want. Inner alignment asks whether the system trained on that objective truly learned the intended goal rather than a convenient substitute.

These terms are common in technical alignment research but can obscure the simpler issue: we can get the written objective wrong, and the system can also learn the wrong thing from a seemingly good objective.

Deceptive alignment

Deceptive alignment is the hypothetical case in which a system behaves as intended while it is being trained or watched because doing so helps it preserve a different objective for later.

This remains a theoretical and experimental research concept, not an established description of how today’s deployed models routinely behave. It is important because it represents a failure mode that ordinary behavioral testing could miss.

Model organisms of misalignment

A model organism of misalignment is a deliberately constructed AI model that exhibits a specific failure researchers want to study. The idea is similar to using a laboratory organism to understand a disease mechanism.

These models let researchers test detection and mitigation methods in controlled conditions. But because the failure was intentionally created, they do not by themselves prove that ordinary frontier models have the same problem.

Keep reading

• Recursive self-improvement: what has actually been demonstrated
• Agent containment is becoming an ordinary security problem
• Why chain-of-thought monitoring can miss dangerous AI behavior
• OpenAI turns unexpected model behavior into an incident-reporting problem

---

Definitions are written by AI Safety Watch as a reporting reference. They are not endorsements of any particular forecast or theory of AI risk. This page is updated as terminology and evidence change.