Tool-Augmented Vision-Language Systems

ClaimLens

Pair-blinded visual probes and deterministic host decisions for auditable scientific-figure claim verification.

A VLM should not describe the same pixels differently because a claim swaps “higher” for “lower.” ClaimLens turns the pair into one symmetric measurement and makes that invariance machine-checkable.

Qwen3-VL-8B Counterfactual Probes OCR-Ranked Crops Strict JSON Paired Bootstrap Zero-Shot

Code & Results Frozen Release

ClaimLens pair-blinded scientific figure verification interface

The microscopy frame is a copyright-safe illustration. The masked-probe behavior, deterministic verdict mapping, and aggregate metrics come from the executed frozen v0.4.3 experiment.

The held-out result

ClaimLens and the direct baseline were statistically tied on overall accuracy. The interesting result is the changed failure profile—not a fabricated benchmark win.

300 held-out claims 100 complete triplets

59% vs 57% overall accuracy paired CI −6.3 to +10.0 pp

71% vs 46% NEUTRAL recall ClaimLens vs baseline

29% vs 17% exact triplets ClaimLens vs baseline

Primary endpoint: agent-minus-baseline accuracy was +2.0 percentage points with a paired base-claim cluster-bootstrap 95% interval from −6.3 to +10.0 points. Because the interval crosses zero, the experiment does not establish an overall accuracy improvement.

Separate visual measurement from judgment

The direct baseline sees one claim and writes a verdict. ClaimLens instead measures the changed concept once, validates the transaction, and lets deterministic code map that evidence to both counterfactual claims.

1 · Audit Remove complete clusters containing impossible cross-label duplicate inputs.

2 · Mask Diff paired claims without labels and replace changed values with symmetric options.

3 · Inspect Send the full figure and OCR-ranked crops to one bounded visual probe.

4 · Validate Enforce schema, retain every attempt, and cache the shared pair transaction.

5 · Decide Map the observed option to SUPPORT or CONTRADICT in deterministic host code.

Capability tradeoffs

Metric	ClaimLens	Direct baseline
Accuracy	59.0%	57.0%
Macro-F1	58.6%	58.4%
SUPPORT recall	48.0%	69.0%
CONTRADICT recall	58.0%	56.0%
NEUTRAL recall	71.0%	46.0%
S/C pair separation	65.0%	60.0%
Exact triplets	29.0%	17.0%
Schema validity	100.0%	99.3%

Grouped bar chart of ClaimLens and baseline held-out metrics

Accuracy was prespecified as the primary endpoint. Recall, pair separation, and exact triplets are secondary diagnostics.

The audit is part of the result

What the system verified

100/100 SUPPORT–CONTRADICT pairs used identical prompt bytes and source order.
300/300 agent outputs had complete traces and valid schemas.
Metrics and the generated report reproduced byte-for-byte from raw predictions.
The held-out set contained zero development clusters and zero exact label conflicts.
110 benchmark base claims touching impossible cross-label duplicates were excluded before sampling.

What the audit found

SUPPORT recall fell by 21 points because the agent over-predicted NEUTRAL.
All agent certainty values were HIGH, so confidence was not calibrated.
Only 26/300 decisions cited a crop; no crop-benefit claim is justified.
A lexical normalization ignored comparator scope in one paired transaction.
Removing that normalization changed one answer and left the statistical conclusion unchanged.

Engineering contributions

VLM orchestration

Label-free counterfactual differencing, masked option construction, OCR-ranked multiscale sources, strict structured outputs, one logged repair attempt, and persisted pair-level cache reuse.

Evaluation infrastructure

Cluster-preserving selection, split and plan fingerprints, conflict auditing, resumable inference, complete raw traces, deterministic rescoring, per-label failure analysis, and paired cluster-bootstrap uncertainty.

Scope and limitations

ClaimLens is explicitly pair-aware and is not presented as a general single-claim classifier.
MuSciClaims contradictions are controlled perturbations rather than natural scientific disputes.
Deterministic host logic prevents label inconsistency; it cannot prevent visual perception errors.
Exact-triplet accuracy is 29%, so substantial headroom remains.
The frozen primary result is inconclusive and should not be described as “beating” the baseline.

Inspect Code & Traces Download Reproducible Bundle Back to Projects