A 66-Cent Draft Caught 5 of 36 Fabrications

The full stack reached 33, at twelve times the cost and 108 times the tokens.

Table 3 of the paper. Instrumented runs, eight pre-registered topics.

On August 12, a team from Vast Intelligence Lab and the University of Technology Sydney posted Spark-to-Paper, a system that takes one line of research idea and returns a compiled paper: literature retrieved, experiments planned and executed, claims revised against the measured results, figures drawn as editable vectors, the whole thing assembled into a venue template.

The interesting part is not that it exists. Sakana's AI Scientist did this in 2024, and several successors have done it since. The interesting part is the packaging: Spark-to-Paper ships as thirteen skills inside Claude Code, with no orchestration server, no graph database, no scheduler. Every skill reads and writes files in a shared project directory. The pipeline decides which task comes next. The coding assistant decides how the task gets done.

That design choice is what makes this issue worth your time even if you never intend to generate a paper. The measurements in it are a rare, itemised accounting of what verification costs inside a long-running agent, separated from what generation costs. The answer is lopsided.

Red: a deterministic program that fails the build. Blue: a model reading its own output. Amber: a counter.

The paper runs an ablation whose reference row is a single-pass draft from the same model backbone: ask once, get a manuscript. That draft costs 0.66 dollars, burns 0.11M tokens, and finishes in 16 minutes. The full stack costs 8.1 dollars, burns 11.9M tokens, and takes 3.2 hours.

Put those side by side and the shape of the system becomes obvious. Producing text that looks like a paper is roughly one percent of the token budget. Everything else is the apparatus that decides whether the text is allowed to stand.

Rows 1 to 5 are Tables 3 and 4 of the paper. Share column and bottom row are my arithmetic on those means.

The bottom row is the one I keep returning to. The full stack consumes 108 times the tokens of the single-pass draft, but costs 12.3 times as much and takes 12 times as long. Cost and time scale together. Tokens scale nine times faster than either.

That gap has to come from somewhere. The blended rate falls from 6.00 dollars per million tokens to 0.68, which is the signature of a long-lived project directory being read over and over from cache rather than re-sent at full input rate. The paper does not break its costs down that way and does not explain the gap, so treat the mechanism as my reading and the ratio as arithmetic.

To test the stack the authors seeded 36 unsupported claims across ten failure families into the source material before drafting, at the same injection point, with the same detector, for all four configurations. That is a clean design: what changes between rows is the quality machinery, not the test.

Table 4 of the paper. Final column is mine: detection points gained divided by the dollars that row added.

Read the last column and the ordering inverts. Gates gain the most ground in absolute terms, 55 points, and they are also the most expensive single addition in the system at 5.30 dollars. Self-review returns 12 points for 60 cents, which is twice the efficiency of gates and nearly three times the efficiency of adversarial review.

There is a practical reading here for anyone assembling an agent on a budget. The layer where a model rereads what it wrote against the surrounding context is the highest-yield thing you can add per dollar. The expensive layers are the ones that require running programs over artifacts and the ones that require several independent reviewers to argue about the same document.

One footnote from doing the arithmetic. The paper cautions that the ablation deltas come from a three-topic paired subset and need not sum to the eight-topic mean in the main table. Summed anyway, they give 11.91M tokens and 8.16 dollars against a reported mean of 11.9M and 8.1. The agreement is closer than the paper's own caveat promises, which is worth noticing before anyone treats the two tables as independent evidence.

The design decision underneath the whole system is that planning happens before observation. At the planning stage the system writes down the datasets, baselines, metrics and ablations that the paper will need, and lays out the result tables with the cells empty. The experiment stage inherits that design and is forbidden from redefining it, including by substituting an easier proxy metric or a different data construction when the specified one turns out to be hard to run.

That is preregistration, implemented as a file. It exists because the alternative failure is so familiar: a system that runs whatever it can run, then writes the claim that the available output happens to support. Two integrity modes enforce the boundary. In proposal mode, an unobserved result must stay unspecified, and a number appearing in prose without backing data fails the build. In data-aware mode, every quantitative statement must trace to the supplied evidence.

Once results land, each major claim gets one of five labels, and the label picks the edit.

Appendix C of the paper. Null and inconclusive results are preserved rather than dropped.

The paper names a failure mode it calls the Self-Refutation Loop. The system forms a hypothesis, runs experiments, judges the evidence insufficient, revises the method, runs again, judges it insufficient again, and keeps going. Nothing is broken. Each individual step is correct behaviour. The system is stuck because it will not entertain the possibility that the idea does not work.

The response is blunt: cap the experiment, critique and revision cycles at seven. Past that, terminate the trajectory and write a failure report recording the original idea, what was tried, what was measured, and why the evidence fell short. The report is the output. Then start a different idea and rerun the pipeline from the top.

The case studies test exactly this. The authors planted two wrong expectations in the input proposals: that accuracy should be the headline metric for a clinical screening task, and that causal models would perform comparably in a forecasting task. In both, the measured results contradicted the instruction and the manuscript went with the measurements. A system that flatters the prompt cannot do research, and this is the smallest experiment that demonstrates the difference.

Citation validity here is defined as resolved references divided by total references, computed after generation against external bibliographic services. Across 384 references from eight generated papers, about two failed to resolve. That is a genuine result and it is a narrow one. It says the paper exists at the identifier given. It does not say the cited work supports the sentence it is attached to. Those are different failures, and the harder one is not what this number measures.

The comparison table needs the same care, and to the authors' credit the paper says so itself.

Cost in dollars, at each system's own reported pricing assumptions. Prior systems were not rerun and not repriced.

Two rows deserve a second look. The reference counts behind the audited systems are small: 45, 64 and 28 references respectively, so a single bad entry moves those percentages by two points or more. And Agent Laboratory produced a paper for 2.33 dollars in about 19 minutes, which undercuts everything else in the table on both cost and time. The trade it makes is visible in the editability column, at zero out of thirty figure elements.

The figure editability result is the strangest number in the paper, because the generated papers beat the human ones by a wide margin, 96.4 against 58. Human authors paste bitmaps. A system that draws every diagram in code does not. Worth noting that the measurement excludes figures intentionally rasterized by design, and covers roughly 1,900 figure elements.

Four more things the paper is careful about, and one it could not be. Review precision is 74 percent, meaning 15 of the 57 rated review issues were not verifiable problems even after three refutation checks. Claim-level evidence diagnosis is performed by the model, not machine-checked, which the paper states twice. The positioning figure's coordinates are labelled as the authors' own ordinal assessment. The showcase figure counts are labelled as self-reported by the maintainers.

The thing it could not be careful about is drift. The paper describes thirteen skills, and a figure pipeline that reconstructs generated images as editable HTML before export. The repository as of this writing advertises fourteen skills, and its skill table lists fourteen rows under a heading that still says thirteen. Its figure engine has been replaced: version 1.2.0 introduced a stage that renders candidates with PaperBanana, learns the render's design language, redraws the figure natively, then repairs it over at least four rounds against a geometry audit. The repository's own description also puts typical cost at about ten dollars against the paper's 8.1. The 96.4 percent was measured on a figure engine the project has since moved past.

Split judgment from execution inside every step. The model chooses what to argue and which evidence counts. Programs check structure, resolve identifiers, plot measured values, and compile. The rule for sorting them is whether the correctness criterion can be written down. If it can, it should not be a prompt.

Commit the output shape before the run. Empty cells that a program refuses to let you fill with generated text hold better than any instruction telling a model to avoid inventing numbers. Fourteen percent detection is what instructions alone bought here.

Make critiques quote the passage they attack. Every proposed issue must identify and quote the specific text, then survive three tests: is the problem real, was it addressed elsewhere, is it outside the stated scope. This is an anti-skim device, and it still lets one in four bad critiques through.

Give the loop a counter and a way to lose. Seven cycles, then a failure report and a new idea. The bound matters less than the fact that terminating without a result is a defined, recorded outcome rather than an error state.

The number to carry out of this paper is 14 against 92 on the same 36 probes with the same backbone. The distance between those two is not model quality. It is the 11.8M tokens of apparatus sitting between the draft and the artifact, and per the ablation, the layer that pays for itself fastest is the plain one where the model reads its own output again.

sources

Paper: Spark-to-Paper, arXiv 2608.11924, August 12 2026 · full text with appendices
Code: spark-to-paper-skills · figure engine PaperBanana
Compared systems: AI Scientist · AI Scientist-v2 · Agent Laboratory

researchaudio.io · arithmetic marked as mine was computed from the paper's published means