← SomaSoft

Limits and corrections

A research project, not a product: no users, no revenue, no deployments. This page collects what we have measured to be wrong, weak or withdrawn, with dates. Last updated 2026-11-18.

Measured limits

WhatResultDate
TruthfulQA, wrong answers penalised (+1 right, 0 abstain, −1 wrong)Refusing everything scores 0.00. AURI's conversation path −0.14 [−0.23, −0.06]; graph-only path −0.40, the same as echoing the question back. It answers 90% of questions and is right 42% of the time. 468 questions × 3 reps.2026-11-01
What the concept graph adds on TruthfulQANothing measurable: graph-only − echo +0.2pp [−5.3, +5.7]. The conversation path's lift over echo (+13.2pp) comes from the language model.2026-11-01
Child-safeguarding disclosure gate, sealed test66 of 140 (47%) unseen disclosures caught; emergencies 3 of 18; self-harm 10 of 25; 2 false alarms in 140. The gate also lets a message through if it errors. Not safe to rely on.2026-11-07
ETHICS benchmark+3.1pp [+0.9, +5.2] over a majority-class baseline on validation; −4.5pp [−8.5, −0.5] on the hard test. A regression check, not evidence of ethical reasoning.2026-11-01
Composing curated facts+0.14 [+0.06, +0.23] with the composer on, only on questions built from our own knowledge packs; it repeated deliberately corrupted facts in 54% of answers.2026-11-02
Structured reasoning, sealed set45 of 135 (0.33 [0.26, 0.42]); an always-yes control scores 0.13. An earlier 0.49 was a development-set number.2026-11-02
Provenance15.1% of graph edges carry any source field. 94.3% of nodes are a single bare word. 63.3% of edges were written by the system's own exploration process.2026-10-14
ARC-AGI0% at n=100. An earlier 40% was n=5.2026-05-13
Peer coordinationInfrastructure runs; traffic is mostly acknowledgements. No verified cross-instance reasoning.standing
Real users0 evaluated sessions with real people.standing

Claims withdrawn

ClaimWhyWithdrawn
"0.0% hallucination over 12 months"No measurement artifact, method or denominator.2026-10-09
ETHICS 77% / 77.38%One category had a constant label; no majority baseline.2026-10-06
"~15–20% AGI readiness"No measurement artifact.2026-11-05
"29 probes, zero missed alarms"Too small and too familiar; superseded by the sealed test above.2026-11-07
A paid API with "124K concepts, full source attribution"The hosted graph had 0 nodes and no language model; provenance covers 15.1% of edges. API withdrawn.2026-11-17

Papers withdrawn

PaperWithdrawn
Four AURIV clinical papers: deployment figures, an endorsement attributed to a named clinician, patient cases and review approvals with no record behind them (two had been withdrawn once before, in May 2026)2026-10-09
AURIV consciousness and cancer2026-11-06
CAR-T for myasthenia gravis; silicone implant safety2026-11-17
CPI nowcasting edge; RSI alpha study2026-11-17
Accountability deficit in corporations; AURI knowledge graph ethical reasoning; SOMA family OS; AURI protocol for symbiosis2026-11-17
AURIV TMR research2026-11-18

Each withdrawn paper's address now says why it was withdrawn. Papers that stay up carry a dated correction notice at the top when something in them was found wrong.