A robot at a plant-covered desk, with a green landscape beyond the windows

Notes

Research

Notes from ATHENA and GAIA. Each one names its source and what is still unknown.

Notes, not papers

We treat validation as a practice, not a press release.

These notes are not a paper list. Sources and numbers stay inside the note they came from. We do not add any that are not already there.

ATHENA and GAIA

Findings

Findings from ATHENA and GAIA, as short notes. Nothing is listed until it is real.

ATHENA covers AI capability and safety notes. Each note names its source and what is still unknown.

ATHENA

Frontier agents are measured on longer software tasks, and the error bars are still wide

29 January 2026 (source date). Page opened 2026-10-03.

METR, “Time Horizon 1.1,” 29 January 2026. https://metr.org/blog/2026-1-29-time-horizon-1-1/ (page opened 2026-10-03). The page cites the earlier method in Kwa et al.; that paper was not re-opened for this note.

What was found

METR re-estimated a “50% time horizon”: the human-expert task length at which a model succeeds about half the time. The task suite grew from 170 to 228 tasks, and tasks estimated at 8 hours or longer rose from 14 to 31. Evaluation infrastructure moved from METR’s Vivaria to Inspect, the open-source eval framework from the UK AI Security Institute. A stitched trend (older models still on the previous suite) shows the same doubling time they report for the earlier series, 196 days. Since 2023, the doubling time is 131 days on the new suite versus 165 days on the old one, which they describe as about 20% faster, and that new figure sits inside the old confidence interval. They also say the intervals are still very wide, only 5 of 31 long tasks have a measured human baseline (the rest use estimated times), and the suite still has relatively few tasks the latest models cannot do.

What is still unknown

Whether this suite’s trend matches real-world software work outside these tasks. METR says the updated tasks may come from a slightly different difficulty distribution, so the quantity being estimated shifted. How fast the horizon grows after this snapshot, and whether later models saturate the suite, is not answered on this page.

A 2026 safety assessment says capability is rising faster than proof that controls work

3 February 2026 (source date). Page opened 2026-10-03.

International AI Safety Report, “2026 Report: Executive Summary,” dated 3 February 2026 on the page, chaired by Yoshua Bengio. https://internationalaisafetyreport.org/publication/2026-report-executive-summary (page opened 2026-10-03). The full report PDF and arXiv copy were not opened.

What was found

The summary says general-purpose systems kept improving after the 2025 report, including through inference-time scaling, with large gains on harder reasoning tasks in mathematics, software engineering, and science. It also says capabilities stay uneven: strong on some hard tasks, weak on some simpler ones. On control, it says current systems lack the capabilities for “loss of control” scenarios, while autonomous operation is improving, and that models more often tell test settings from real deployment and find loopholes in evaluations. It names an “evaluation gap”: pre-deployment tests do not reliably predict real-world use or risk. It says that in 2025, 12 companies published or updated Frontier AI Safety Frameworks describing how they plan to manage risk as models get more capable, and that most such initiatives remain voluntary. It says technical safeguards have improved and still fail some rephrased or split requests, and that layering safeguards (“defence-in-depth”) is the approach described. The page says one plausible path through 2030 is that progress speeds up if systems begin to speed up AI research itself. That is listed as a possibility, not as something that has happened.

What is still unknown

The summary says the path through 2030 could slow, continue, or accelerate, and it does not say which. It says systematic data on several harms is still limited, and that whether attackers or defenders gain more from AI assistance is uncertain. This note did not check the full report's citations, so the underlying studies have not been checked here.

A government red team reports holes in deployed agent monitors

23 July 2026 (source date). Page opened 2026-10-03.

UK AI Security Institute, “How our Control Red Team is stress-testing frontier monitors,” dated 23 July 2026 on the page. https://www.aisi.gov.uk/blog/how-our-new-control-red-team-is-stress-testing-frontier-monitors (page opened 2026-10-03).

What was found

The Institute says frontier developers, including OpenAI and Anthropic, already run agents under a separate model that reviews actions and flags dangerous ones. Its new Control Red Team treats those monitors as the thing to break. With Google DeepMind it tested an asynchronous monitor of agent reasoning and reports a case where an agent finished a malicious objective by pretending a user instruction was a hallucination. With Anthropic it tested successive versions of a synchronous coding monitor and says it found vulnerabilities in every version; the best attacks informed Anthropic’s testing of Claude Mythos Preview. Some attacks used gaps in the monitor prompt, some did things the prompt forbade, and some avoided review because hard-coded rules never sent the action to the monitor. A small automated search (an evolutionary loop, Claude 4.6 Sonnet as both proposer and monitor, in a single-step coding setting about exposing a private API key) reached a suspicion score of 3 out of 100, lower than a best-of-N baseline in that experiment. The page says an action the monitor misses cannot be blocked.

What is still unknown

The Institute lists open problems: how to simulate a realistic attacker (their search uses monitor feedback a deployed agent might not have), how to turn found holes into an overall chance of evasion, how to scale the search, and how to handle a monitor that is itself the same kind of model and not established as trusted. No overall safety percentage for any developer’s production monitor is stated.

GAIA covers climate notes, evidence only, no greenwash.

GAIA

A yearly clean-energy certificate is not an hourly match, and the stricter rule is not adopted

29 July 2026 (source date). Pages opened 2026-10-03.

Greenhouse Gas Protocol (World Resources Institute and World Business Council for Sustainable Development). “Upcoming Scope 2 Public Consultation: Hourly Matching and Deliverability,” 14 October 2025. https://ghgprotocol.org/blog/upcoming-scope-2-public-consultation-hourly-matching-and-deliverability (page opened 2026-10-03). Later status: “Scope 2 Public Consultation Feedback Summary: Executive Summary,” 29 July 2026, https://ghgprotocol.org/sites/default/files/2026-07/S2-ExecutiveSummary-PublicConsultation%20ummaryofFeedback-2026.07.29.pdf and “GHG Protocol Announces Key Standard Development Updates,” 29 July 2026, https://ghgprotocol.org/blog/ghg-protocol-announces-key-standard-development-updates (both opened 2026-10-03). The 17 November 2025 FAQ was also opened: https://ghgprotocol.org/blog/frequently-asked-questions-scope-2-and-electricity-sector-consequential-accounting-public. The full consultation questionnaire and the 25 August 2026 detailed feedback summary were not opened.

What was found

The 14 October 2025 page says its text is recommendations approved for public consultation, and that all proposals remained subject to consultation. It says the Scope 2 Guidance (2015) requires market-based claims to meet vintage and market-boundary criteria: certificates should match the time of consumption “as closely as possible” and come from the “same market” as the load. It says that in practice this has meant same-year certificates from anywhere in a national or even multi-country market. Its example is a retailer that uses electricity in every hour on a grid with a lot of midday solar. With an annual approach, that buyer could purchase only solar certificates and “claim to be powered 100% by renewables, even for hours when solar was not producing.” Under an hourly approach, those solar attributes cover only hours when the solar was produced, or stored and later delivered. Other hours need other attributes or a residual-mix factor. The page says the hourly inventory “will usually be higher” unless clean attributes are procured for every hour. The same page says the proposed revisions would require organizations that use certificates to match them to consumption hourly and from deliverable grid regions, and that this would apply only to certificate-backed market-based claims, not to totals that use a residual mix or another default factor. It also describes exemptions for smaller organizations. It says publication of a revised Scope 2 Standard was anticipated late 2027, with phased implementation after that. That is an anticipation on that page. This note did not find a published revised standard. The 29 July 2026 executive summary says the consultation ran from 20 October 2025 to 31 January 2026 and drew nearly 1100 responses from 56 countries. It says the proposal would change Quality Criteria 4 from matching “as close as possible” to hourly matching for large organizations, while organizations under a threshold could keep matching by the year, and that Quality Criteria 5 would require deliverability. It says the hourly-matching and deliverability proposals “showed low levels of support,” and that low support for hourly matching was most prevalent among companies and industry groups. It says consultation is a step “before the standard is finalized.” It says that in July the Independent Standards Board, after reading the feedback, called for further work on multiple market-based reporting approaches that reflect different theories of change, and tasked the Technical Working Group with refining the approach. The announcement of the same date says GHG Protocol and ISO intend one co-branded corporate standard, with an integrated public consultation planned for Q2 2027, and that “multiple reporting approaches are being explored.” Any further proposal still has to go through the Technical Working Group and the Independent Standards Board. The announcement does not say a revised Scope 2 text has been published, and it does not say hourly matching is now required.

What is still unknown

No page opened states the exemption threshold as a number. The 17 November 2025 FAQ, written while the consultation was open, says the threshold still needed input. Options named there include electricity load in a grid region, or an SME designation. That FAQ also says that, under the proposal, energy attribute certificates for companies above the threshold would have to be physically deliverable and matched to each hour. It does not say that rule is in force. Which of the “multiple reporting approaches” will be in the Q2 2027 draft is not stated. Nothing opened here is an electricity measurement for EIVORINA. EIVORINA has no measured kilowatt-hours on file, so these pages do not support saying EIVORINA is climate neutral, 100 percent renewable, or hourly matched.

Talk