AI Frontier · Sep 23, 2026

Covering September 16 to 23. Three papers in this issue's research section measure the agent on the attacker's side — kernel exploit-primitive generation, deception-aware pentesting, MCP tool hijacking — rather than measuring once more how vulnerable it is as a victim, and those three are also the ones that put reproducible numbers on the table.

Research

Attack success rate is not a number: a meta-analysis of 259 agentic-security papers

The authors argue ASR as used in the literature is not one quantity but a family of metrics parameterized by six design choices that papers seldom specify and never hold constant. In a hand-coded random sample of 50, 58% (95% CI 44-71) report neither a variance estimate nor repeated runs for their headline attack metric; automated coding of all 259 puts it at 65.3%. Only 30.9% disclose enough about decoding to establish whether the evaluation was even stochastic, and of the 64 papers confirmed to use an LLM judge, 29.7% report any agreement check against human labels. The analytical half gives the consequence: on a 100-instance benchmark the minimum ASR difference detectable at conventional power is 18.2 percentage points, and two defenses whose true ASRs differ by 5 points are ranked in the wrong order by a single-run evaluation about 21% of the time. The authors note this is not a constant offset that cancels in comparison, because several of the six axes move ASR in a system-dependent direction.

Sources: arXiv

A2M: get the agent to pick your tool, then rewrite what it returns using the execution trace

A two-stage black-box hijacking framework for MCP. The Attraction phase optimizes third-party tool metadata to raise the chance of being invoked; the Manipulation phase uses execution traces to refine adversarial tool returns that steer the agent toward the attacker's outcome. On LiveMCPBench, optimized and evaluated on GLM-4.6, the macro-average malicious tool invocation rate is 93.6% across four scenarios, weighted token cost under Cognitive Denial of Service reaches 32.4x the benign baseline, and mean attack success across Information Exfiltration, Environment Integrity Compromise and Reasoning Derailment is 74.4%. Transferred to four other models with no re-optimization the same figures fall to 63.6%, 2.7x and 24.5% — a large transfer loss the authors state in the abstract.

Sources: arXiv

KEX-bench: can an agent turn a kernel crash into an exploit primitive?

Coding agents now find real vulnerabilities in production software, but bug discovery does not measure whether they can build exploit primitives. KEX-bench has 45 task instances over 40 Linux and Windows CVEs, covering kernel address leak, instruction-pointer control, heap read, heap write and arbitrary address write, each in an isolated VM with a deterministic verifier. Under fixed tool-call budgets and without a reference proof of concept, the strongest configuration solves 1 of 20 Windows tasks (5.0%) and 14 of 25 Linux tasks (56.0%). With a reference PoC it solves 31 of 45 (68.9%). The Windows-Linux gap and the with/without-PoC gap are the readings to take from this one.

Sources: arXiv

Rouxii: a honeypot works to the extent the attacker does not know it is one

Earlier work showed honeypots can derail autonomous LLM pentesters, but those evaluations assumed an attacker unaware of the deception. This paper builds the informed side: counter-deception in reconnaissance, then a pivot from honeypot detection to exploitation. Across three reasoning models, eleven network setups, twelve cycles and 1,544 attack reports, with the two cohorts differing only in the prompt, correct honeypot identification goes from 19% to 97% — on OT services from 11% to 97% — while false alarms against the real service stay at 0.7%. Detection is not the endpoint: a white-box analysis of the honeypots themselves demonstrates a denial of service that disables Conpot without tripping its liveness monitoring, and a corruption of the intelligence a GasPot instance reports. PentestGPT and HackingBuddy fail the same way, so the effect is not specific to this framework.

Sources: arXiv

Auditing shadow APIs: 45.83% of fingerprint tests fail to confirm the model being claimed

Third-party resellers that route around pricing, payment barriers and regional restrictions are widely used. The authors identify 17 shadow APIs used across 187 academic papers, the most popular with more than 5,900 citations and 58,000 GitHub stars by December 6, 2025. Auditing three representative endpoints on utility, safety and model verification, they find performance divergence up to 47.21%, significant unpredictability in safety behavior, and identity verification failures in 45.83% of fingerprint tests. Their own wording is fingerprint-based evidence consistent with deceptive model claims, scoped to a subset of the audited endpoints rather than asserted of shadow APIs generally.

Sources: arXiv

Targeted unlearning for package hallucination: down 88%, coding benchmarks largely unmoved

When a model fabricates a non-existent package during code generation, an attacker can register that name on a public registry with a malicious payload and wait for developers or autonomous agents to install it — the class of package confusion known as slopsquatting. Adaptive Unlearning is a post-deployment framework combining a token-level objective that reinforces valid outputs while suppressing hallucinated ones with an adaptive discovery loop that surfaces new hallucination-inducing contexts without human supervision. Package hallucination rates drop 88% while performance on standard coding benchmarks holds, and the distributional analysis shows the change is concentrated on package-related generations. This is suppression rather than elimination, and it acts on the model side; the registry-side squatting does not go away.

Sources: arXiv

Lab announcements

OpenAI: GPT-6 Sol and Luna (September 22)

Two models aimed at everyday work, described as striking different balances between capability and cost.

Sources: OpenAI

OpenAI: priorities and principles for third-party assessments (September 22)

What OpenAI says independent third-party safety assessments of frontier models and safeguards should be held to.

Sources: OpenAI

OpenAI: better prompt caching for GPT-6 (September 22)

Higher cache hit rates, new diagnostics, explicit breakpoints, and controls aimed at reducing latency and cost.

Sources: OpenAI

OpenAI: building standards for the next phase of AI (September 21)

A path to shared global AI standards, calling for coordination on evaluation, reporting and governance.

Sources: OpenAI

OpenAI: an independent Advisory Group on Mathematics and Artificial Intelligence (September 21)

The group is to guide the review and communication of emerging AI results in mathematics.

Sources: OpenAI

OpenAI: a framework for reporting model misalignment (September 16)

A framework for tracking, investigating and disclosing model misalignment, published alongside six reports of unexpected or concerning model behavior.

Sources: OpenAI