AI Frontier · Sep 19, 2026

Covering 12 to 19 September. This week's research measures deployed systems rather than models: a black-box audit flags 7 of 15 commercial LLM APIs; in an evaluation of protocol analysis a trivial always-secure predictor scores higher than every model configuration but one; and one paper concludes the operating system can see an agent's self-state changes but cannot tell which of them is an attack.

Research

A pay-per-token provider has an incentive to make the model say more, and that can be audited from outside

The paper defines Provider-Side Token Inflation Attack (PTIA) as covert manipulation of generation that raises output token counts while largely preserving task utility, and instantiates one attack at each of four levels of the provider-controlled pipeline — query, prompt, representation and model. Every one pushes mean output length above 10.2x the clean baseline. The authors find PTIA saturates: an initial attack sharply lowers the end-of-sequence token probability, while further intervention lowers it only marginally. That yields a single-probe audit — apply a controlled lengthening intervention, and a normal service produces far more additional tokens than one already inflated. It needs neither a trusted local reference model nor historical clean responses, and reaches an average detection rate of 85.1% at under 2% false positives across four open-weight models. Across 15 real LLM API services, the audit flags 7 for PTIA-consistent behaviour.

Sources: arXiv

Symbolic protocol analysis: reasoning mode doubles precision, and a trivial baseline is still more accurate

The study runs GPT and DeepSeek in chat and reasoning modes, three runs each, over 130 obfuscated AnB/AnBx protocols covering 388 security goals, scored against ProVerif and OFMC. Chat mode gives high recall at low precision (GPT 72.7% / 27.3%); reasoning mode reverses the trade-off (GPT 66.5% precision, 54.5% recall). The goal set is imbalanced — 89 vulnerable goals against 299 secure ones — so a predictor that ignores the content and always answers "secure" scores 77.1% accuracy, and only GPT in reasoning mode exceeds it. All models do worst on authentication goals, where reasoning models detect under half of injective and non-injective agreement attacks; confidentiality is the exception, with F1 up to 95.7% in reasoning mode. Verdicts are also unstable across runs — identical on 89.7% of goals for GPT reasoning, 61.6% for DeepSeek chat. Self-reported confidence is uniformly high and shows no meaningful correlation with correctness. The authors note everything rests on a single zero-shot prompt and two providers.

Sources: arXiv

In resume screening the trust relationship is inverted: the document assessing you is supplied by you

ResumeShield addresses the structure where an AI screener reads a document handed over by the person it is evaluating. Candidates conceal instructions using white text, zero font size, hidden elements, markup comments, document metadata or zero-width characters; a human reviewer sees nothing while a naive extraction pipeline places the concealed text in the prompt, where it is read as an instruction and obeyed. The defence combines three filtering stages with a fourth architectural stage that puts candidate content in an explicitly fenced data channel the operator's trusted instructions declare inert. The benchmark seeds a synthetic corpus of 104 documents spanning nine concealment techniques and two payload families — one using documented phrasings, one modelling an adaptive attacker who paraphrases around the filter — and scores an attack as successful only when the screening outcome changes. The naive pipeline is manipulated in every injected case and the defended pipeline in none; detection reaches precision 1.000 and recall 0.944 with no false positives on clean resumes. The ablation is the finding worth keeping: channel separation alone removes all measured attack success, while the complete filtering stack without separation still leaves 16.7% of attacks succeeding.

Sources: arXiv

Financial trading agents: of 15 academic schemes, 80% fail robustness and 100% have security flaws

The authors argue that agentic-AI security work is largely domain-agnostic and overlooks the attack surface specific to high-stakes settings, so they use financial trading agents as the case — a compromised agent there has direct execution authority over real capital in an adversarial, reflexive market. Their FARSIGHT framework evaluates schemes on two axes: robustness under market turbulence including flash-crash-like scenarios, and security against attacks on information sources, attacks on agents, and agent-as-attacker behaviour. Across 15 representative academic schemes, 80% fail at least one core robustness metric and 100% exhibit security vulnerabilities. The paper insists the two failure modes are inseparable: a small misjudgment can cascade into a market-wide crash on its own, and an adversary can deliberately trigger the same collapse at minimal cost.

Sources: arXiv

An agent's self-state: the OS can see the change, but lacks the context to judge it

Self-hosted agents keep persistent memory, instructions and configuration that govern their future behaviour. Once an agent is compromised, the attacker uses the agent's own legitimate write permissions to corrupt that self-state, and at the OS level a malicious update looks like a benign one. The authors formalize the attack space and evaluate representative OS defences across four agent workloads and a Linux telemetry pipeline. The limitation is consistent across dimensions: file-level controls either leave alternative mutation paths open or, when complete over the tested operations, also block the corresponding legitimate updates; detectors flag a substantial part of legitimate activity, while more selective methods cover only part of the attack space; protected backups do restore corrupted state but require a trusted recovery point and incur rollback cost. Their conclusion is that the bottleneck is not OS observability — the OS can enforce, observe, attribute and recover — but that generic defences lack the decision context to combine broad coverage with selective decisions.

Sources: arXiv

Does the model already know the content is harmful? A 12.6M-parameter probe on an 8B model's activations

The authors extract activations from LLaMA-3.1-8B and train lightweight MLP classifier probes of 12.6M parameters to detect harmful prompts, reaching F1 of 99%, 83% and 84% on WildJailbreak, Beavertails and AEGIS 2.0 respectively — competitive with guard models a thousand times larger, at much lower latency and compute cost. Their premise is that external guardrail models are blind to the model's internal workings, which creates an assurance gap. The 16-point spread across the three datasets is the part to weigh: transfer across distributions is not settled.

Sources: arXiv

Lab announcements

Anthropic and Accenture begin embedded evaluation, each expecting to invest at least $1 billion (18 September)

The partnership is led by Faculty, Accenture's specialist AI business, and covers evaluating and red-teaming models, alignment assessments, and testing model safeguards. Unlike today's external evaluators, embedded evaluators work inside the AI company with access comparable to an employee's: they can watch models take shape in training, follow the decisions governing how models are built and deployed, and speak directly to employees. Each company expects to invest at least $1 billion in this capacity over five years. The announcement names its own unsettled parts: there are no standards yet for what information embedded evaluators should have access to or how they should report findings, and no settled system for funding independent evaluation. Anthropic says funding should long-term come from pooled or government sources; since neither exists, it is funding Accenture's work directly. The arrangement is non-exclusive — Anthropic says other evaluators will be announced in the coming weeks, and it is in discussion with METR and other nonprofits to pilot elements of embedded evaluation on their own funding.

Sources: Anthropic

Google expands its AI & Economy research program, adding a 2025 Nobel laureate (18 September)

Philippe Aghion, 2025 Nobel laureate in economics and professor at INSEAD and the Collège de France, joins as an academic advisor alongside fellow laureate Michael Spence and Cambridge's Diane Coyle, bringing work on innovation-led growth and creative destruction to modelling AI's long-term macroeconomic trajectory. Ajay Agrawal of the University of Toronto's Rotman School joins as a Visiting Fellow, working with current fellow David Autor, head of MIT's economics department, on the economics of AI and scientific discovery and on AI and robotics. Anu Madgavkar and Daniel Rock become directors of the program. The stated research areas are the future of work, productivity and growth, global technology diffusion, and AI's impact on scientific discovery. It extends the AI & Economy ATLAS v1.0 released earlier.

Sources: Google

OpenAI publishes an Australian Youth Safety Blueprint (18 September)

A six-pillar roadmap for safer AI experiences for young people.

Sources: OpenAI

OpenAI: the law firm Cooley built GO Public on ChatGPT Work for IPO work (17 September)

Intended to surface issues earlier in the IPO process.

Sources: OpenAI