Proof · data, computation, accuracy

What we can defend with real data

The MESSAI platform’s evidence for prediction accuracy, computational depth, and a curated data corpus — sourced from on-disk training artifacts and live database counts. Numbers refresh per request; every tile cites its source file. Where the evidence is thin or missing, the honest-gaps section says so — nothing is silently hidden. For the method these numbers are evidence for, see the Science ▸ Methodology page.

Out-of-sample coverage

97.98%

95% CI · 940 held-out obs

Expected calibration error

1.96%

overall · calibration.json

Measured parameter values

200,135

live · ExtractedParameterData

Research papers

23,598

live · ResearchPaper

Canonical parameters

706

live · ParameterClassification

DAG edges

2,812

live · ParameterEdge

All numbers above are read at request time from apps/web/public/data/computed/research/ + Prisma. Live counts grow across sessions; snapshot fallbacks render when the DB is unreachable.

§1 Prediction accuracy

Out-of-sample 95% credible-interval coverage

Out-of-sample prediction accuracy

7 strata · 940 held-out observations · cohort holdout-2026-05-21 · generated 2026-05-21

generalizes

OOS coverage

97.98%

vs 95% target

In-sample coverage

95.85%

-2.13 pp gap (lower is better)

Expected calibration error

—

calibration.json

Holdout strata

940 observations

Per-stratum coverage (sorted by n)
Parameter	System	n	Inside	OOS coverage	IS − OOSpp
power_density_areal	MFC	234	229	97.86%	-0.43
cod_removal	MFC	220	214	97.27%	-3.18
current_density_areal	MFC	163	159	97.55%	-0.61
coulombic_efficiency	MFC	135	135	100.00%	-7.41
current_density_areal	MEC	69	69	100.00%	-1.45
coulombic_efficiency	MEC	64	64	100.00%	+0.00
cod_removal	MEC	55	51	92.73%	+0.00

Negative IS − OOS gap means the model generalizes better than its fit cohort would suggest — the holdout papers happened to be slightly easier than average. A large positive gap would indicate overfitting; none of the 7 strata show that.

§2 Computational moat

Models, physics, and discovered laws

GP-SCM — Gaussian Process Structural Causal Model

Multi-output GP with learned coregionalization kernel. Supports inversion (target → design parameters), Pareto front, and global sensitivity.

in production

Trained artifact

508 KB

gp-scm-fitted-named-db-2026-05-19-v3.pkl

Kernel

ICM

Intrinsic Coregionalization

Training data

195,846

from 9,511 papers

Trained at

2026-05-19

v3, latest of 6 snapshots

Physics constraint laws — 5 encoded

Pure-function residual checks (services/ml-engine/physics/constraints.py) anchored to Logan et al. and Newman & Thomas-Alyea. They run on every extracted operating point whose inputs are reported, and a violation is stored as a flag on the rows involved, so downstream queries can exclude them.

Law	Formula	Source
Power identity	P = V · I	Newman & Thomas-Alyea
Coulombic efficiency within 0–100%	0 ≤ CE ≤ 100%	Logan et al. 2006
Coulombic efficiency charge balance	CE = ∫I dt / Q_substrate	Logan et al. 2006
Cell voltage below open-circuit voltage	V_cell ≤ V_oc (I > 0)	Thevenin
Max power theorem	P_max = V_oc² / 4R_int	Thevenin

Symbolic regression — PySR (Cranmer 2023)

Genetic-programming search for closed-form mechanistic laws. 2 of 5 fits succeeded · 18.7s wall time

40% success

Discovered laws

powerDensity ~ f(currentDensity)MFC · n=51 · complexity=11
(x0 + (0.22200376 / ((x0 + 2.3158035) * -1.8066067))) - 0.6148749
P = V·I. At matched load V≈V_oc/2 ≈ const → P ∝ I
coulombic_efficiency ~ f(cod_removal)MFC · n=29 · complexity=13
((x0 + -0.19921306) / ((x0 + (x0 + 0.46581972)) / 0.0033559396)) + -1.308836
At high COD removal, more substrate goes to biomass (not e-) → CE drops

Skipped fits (3) — honest reporting

· internalResistance ~ f(electrolyte_conductivity) — insufficient_data (n=1)
· specific_h2_production ~ f(currentDensity) — insufficient_data (n=2)
· powerDensity ~ f(currentDensity, internalResistance) — insufficient_data (n=15)

Learned causal DAG — Bayesian network discovery

HillClimbSearch + BICScore on 173 papers across 9 features · v1-pgmpy-hillclimb-bic-2026-05-10

Novel edges learned (1)

· powerDensity → currentDensity

Handcrafted edges NOT confirmed by data (14)

· appliedPotential → currentDensity
· biofilmThickness → massTransportLimitation
· coulombic_efficiency → energy_efficiency
· coulombic_efficiency → h2_yield
· coulombic_efficiency → specific_h2_production
· electrodeMaterial → specificSurfaceArea
· exchangeCurrentDensity → chargeTransferResistance
· flowRate → biofilmThickness
...and 6 more

§3 Data moat

Curated corpus, taxonomies, and provenance

Data corpus and taxonomies

Live database counts + curated catalog snapshots. All counts read live from Supabase Postgres.

live

Proprietary corpus (live from DB)

Research papers

23,598

ResearchPaper

Extracted parameter values

200,135

ExtractedParameterData

Canonical parameters

706

ParameterClassification

DAG edges

2,812

ParameterEdge

Curated taxonomies (snapshot — open-source/mess-*)

Microbe species + priors

mess-microbes seed v1.3

Genus electroactivity priors

EET pathway × inoculum source

Electrode catalog entries

Pourbaix · conductivity · cost

Materials Project entries

160

DFT-backed, cross-referenced

What makes this a moat

· Every extracted value carries provenance: derivationMethod, uncertaintyPlus/Minus/Type, confidence, snippet, conditions (JSONB), canonical (JSONB).
· Parameter DAG is DB-as-source-of-truth with FK constraints on slug identifiers (post 2026-05-09); no JSON drift.
· Materials catalog bridges DFT (Materials Project API) + lab electrochemistry + Pourbaix system-class data — hand-curated, not bulk-imported.
· Two-tier extraction with provider fallback (Anthropic Gateway / Gemini / Groq) + schema-tolerance shim for non-Anthropic providers.

What is curation, not ownership

· ~35 MB of Zenodo / FigShare / Recherche Data Gouv metadata in mess-datasets-catalog is cross-linked & classified — but the underlying datasets are public.
· Public datasets count as a curation moat (consolidation + cross-classification effort) but not a data moat (we don’t own the underlying measurements).
· Research papers themselves are public; the moat is the extraction → DB pipeline with verifier sign-off, archive-never-delete, and per-row reproducibility scoring.

§4 Trust map

Per-parameter Bayesian prior states

Parameter trust map

102 unique parameter slugs · 268 stratum-level states (pooled + by system_type + by application_domain) · v2-trust-2026-05-16 · fitted 2026-05-16

live snapshot

Calibrated

4.5% of strata · converged + LOO reliable

Modeled

25.7% of strata · converged, weak LOO

Curated

178

66.4% of strata · literature prior, low-n

Flagged

3.4% of strata · anomalous posterior p

Breakdown by axis
Axis	Calibrated	Modeled	Curated	Flagged
Pooled (across systems)	8	20	71	3
By system_type	3	36	75	4
By application_domain	1	13	32	2

Each parameter has up to 3 stratification axes; a parameter that converges with reliable LOO at the pooled level might still be ‘curated’ at narrower system_type slices where n is small. This is honest stratification — the headline number above counts every stratum-level state, not just the best ones.

§5 Literature retrospective backtest

25+ peer-reviewed papers, encoded and re-predicted

Literature retrospective backtest

25 peer-reviewed papers encoded as LoganPaperPreset objects with DOI, figure reference, reported values, units, and caveats. Re-predicted at each paper’s exact experimental configuration; the percent-delta vs reported values is sorted into three bands by size (thresholds below).

93% green (MFC)

Encoded papers

apps/lab/src/lib/sweep/presets/*.ts

MFC green band

13/14

|Δ| ≤ 25% · 92.9%

Median |Δ|

6.7%

across MFC presets

Anchor tests passing

10/10

100% · Jest CI gate

MFC band distribution (MESSAI thresholds)

Green — |Δ| ≤ 25%13 / 14 · 92.9%

Close to the reported value

Amber — 25% < |Δ| ≤ 50%1 / 14 · 7.1%

Investigate; may indicate missing physics term

Red — |Δ| > 50%0 / 14 · 0.0%

Genuine model gap

Source files

· apps/lab/src/lib/sweep/presets/ (25 preset files)
· apps/lab/src/lib/sweep/__tests__/backtest-presets.test.ts
· apps/lab/src/lib/sweep/__tests__/butler-volmer-calibration.test.ts

Anchor tests assert 10 hard-coded numerical ranges from canonical peer-reviewed papers (Liu-Logan 2004 wastewater/PEM/acetate, Cheng 2006 carbon-cloth, Cheng-Logan 2007 brush anode, plus 5 more). All 10 pass the Jest regression gate pre-release. Snapshot dated 2026-05-21; auto-refresh would require a scripts/research/snapshot-backtest.ts helper (planned follow-up).

§6 Honest gaps

What we don’t yet have evidence for

We're honest about gaps — peer section, not buried

Below are 9 known gaps in MESSAI’s defensibility today. They sit beside the moat sections with the same visual weight because surfacing them is the trust signal — reviewers should see them on the first scroll, not hunt for them in a footnote. Each card cites its source memory/handoff doc so the gap can be independently verified.

Gap · cov-power-density1,285%

Power density CoV ~1,285%

Why & mitigation›

Power-density measurements span ~4 orders of magnitude across the literature; point estimates are not meaningful without conditioning on system class, scale, and electrode geometry.

Source

open-source/mess-parameters/data/SCIENTIFIC_INTEGRITY.md (Rule §3)

Mitigation

Predictive intervals are reported per-stratum (system_class × application_domain) via hierarchical Bayesian priors; point estimates only render in the lab UI when n ≥ 5 in the stratum.

Gap · canonical-coverage51.6%

Canonical-slug coverage 51.6% of extracted values

Why & mitigation›

104,116 of 201,954 extracted values carry a canonical parameter slug (measured 2026-08-11). The other 48.4% are raw names with no canonical parameter yet — aliases that need DAG nodes added.

Source

apps/site/src/data/methodology-stats.ts (canonicalMappingRatePct, measured 2026-08-11)

Mitigation

Alias mapping continues in open-source/mess-parameters/scripts/sync-from-database.js.

Gap · extraction-success

v2 extraction success rate: not currently measured

Why & mitigation›

The earlier ~75% figure was withdrawn and has no measured replacement. Known failure causes: schema tolerance for non-Anthropic providers, malformed PDFs and paywalls.

Source

apps/site/src/data/methodology-stats.ts (extractionSuccessPct: withdrawn)

Mitigation

Schema-tolerance shim landed 2026-05-09 (commit 3d215187b); BioC-PMC XML path is the next planned mitigation.

Gap · gemini-other-regression52%

system_class=OTHER 52% under Gemini fallback

Why & mitigation›

When the Gateway is throttled and Gemini-2.5-flash takes over, system_class classification regresses from ~14% OTHER (Haiku) to ~52% OTHER on the first 21 batch5 papers. Several papers with "microbial fuel cell" in the title land in OTHER.

Source

docs/extraction/provider-fallback-2026-05-09.md

Mitigation

Mitigations under evaluation: inline prompt enumeration, Tier-1 regex strawman, try gemini-2.5-pro. paper_class field IS reliable on Gemini.

Gap · multiclass-validation2 of 7

Bayesian holdout validation: MFC + MEC only

Why & mitigation›

The 97.98% OOS-coverage number covers 7 strata across MFC + MEC. MDC, MES, MNRC, MMRC, MBES analytical predictors landed 2026-05-15 but have not been validated against held-out paper cohorts.

Source

libs/shared/electrochemistry/src/predictors/{mdc,mes,mnrc,mmrc,mbes}.ts + holdout-coverage-2026-05-21.json

Mitigation

Multi-class holdout cohort assembly is the next planned validation workstream.

Gap · bioc-pmc-unconsumed1,400+

1,400+ BioC-PMC XML files unconsumed

Why & mitigation›

paperscraper accumulated 1,400+ BioC-PMC XML files alongside PDFs. Neither v1 nor v2 extractors consume XML today — these are ground-truth full-text that would feed v2 with higher accuracy than PDF + Nougat.

Source

CLAUDE.local.md → "Known gaps" → "BioC-PMC XML extraction path"

Mitigation

Either write an XML-aware extractor or feed XML to v2 with a different system prompt — sized but not scheduled.

Gap · bvm-mechanistic-data-gated19 / 30 cells

Mechanistic BVM kinetics: data-gated, not yet identifiable

Why & mitigation›

After the 2026-06-09 curve-sync fix recovered stranded 3-electrode data, the corpus now has 843 half-cell (vs-SHE) polarization points and 19 of 30 BVM-posterior-ready cells (up from 0 and 13). But that still falls short of the 30-cell target, and the genuine potential-sweep data is thin (≈3 CV curves), so the Butler-Volmer transfer coefficient α remains under-identified. We make NO mechanistic BVM prediction claims today.

Source

scripts/audits/bvm-identifiability-audit.ts + scripts/extraction/sync-curves-to-polarization-points.ts (2026-06-09)

Mitigation

We surface the validated within-design DIRECTION skill instead (power↔substrate 63%, power↔pH 62%, current↔external-resistance 71–78%); closing the remaining α gap needs new 3-electrode / cyclic-voltammetry papers + lab partnerships.

Gap · ml-predict-uq-heuristic±25% band

/api/ml/predict per-class UQ is a heuristic band

Why & mitigation›

Per-class routing IS live: /api/ml/predict dispatches MEC/MES/MDC/MNRC/MMRC/MBES to dedicated analytical predictors (runFullPrediction → predictBaseForSystemClass), with MFC on the data-tuned empirical path. The remaining gap is uncertainty: those analytical predictors emit a flat ±25% heuristic band (uncertaintyBasis="heuristic_fixed_fraction"), NOT physics-propagated or calibrated UQ, and MES/MNRC/MMRC/MBES still fall through to MFC-surrogate physics enrichment.

Source

libs/shared/ml/src/run-full-prediction.ts + libs/shared/ml/src/predictions/per-class-base.ts

Mitigation

Per-parameter posterior UQ (WS-D) and per-class physics enrichment for MES/MNRC/MMRC/MBES are tracked follow-ups; the interval label already declares when a band is heuristic vs calibrated.

Gap · symbolic-regression-coverage2/5

Symbolic regression: 2 of 5 fits succeeded

Why & mitigation›

PySR run on 2026-05-10 attempted 5 mechanistic fits; 2 succeeded (powerDensity↔currentDensity, coulombic_efficiency↔cod_removal); 3 were skipped for insufficient_data (internalResistance, specific_h2_production, P from current+R).

Source

apps/web/public/data/computed/research/symbolic-regression-laws.json

Mitigation

Re-run PySR after extraction backfills bring per-pair n above the data-sufficiency threshold (currently many pairs at n < 5).