Writing

Press Play

Play a record twice and you hear the same thing twice. It is time an experiment did the same.

Play a record twice and you hear the same thing twice.

That is not luck. Somewhere there is a master, and every copy is pressed from it. The performance was live once and will never happen that way again. The record is fixed. It plays back the same today, next year, on any machine that can spin it. You can trust it, build on it, hand it to a stranger and know they hear what you heard.

Science mostly cannot do this. When a scientist runs an analysis, there is usually no master. The work happens, a number comes out, a figure goes into a paper, and the run that produced it is gone. Ask for it a year later and you may get a different answer, or none. The result survives. The thing that made it does not.

This was fine for a long time, and the reason matters. The tools we analyse data with were built well, each for its moment. A spreadsheet put statistics in the hands of people who will never write code. Stats packages did the same for whole fields. Scripting languages gave power users command of every step. None of them was built so a stranger could rebuild the result from nothing, because none had to be. A person did the work, and a person could check it. Trust lived in the human. The tool never had to carry it.

That arrangement is now breaking, for a simple reason. The analyst is no longer always human.

Give a capable AI model a set of tools and it will reach into the database, write the code, run it, read the output, adjust, and hand back a finished result. It is fast and it is good and it will do a great deal of the world's data analysis. But the trust that used to sit with the person at the keyboard has nowhere to go. The thing at the keyboard now ran a hundred steps no one saw and no one can replay.

We already know how fragile analysis is even without a machine driving it, because people have measured it. A study of more than a million shared notebooks found that only about a quarter would run at all, and only four percent returned the result they claimed.1 A closer look at over nine hundred published notebooks that should have run found nearly three-quarters could not be reproduced without guessing the order their cells were meant to run in.2 R fares no better: across more than nine thousand R files from deposited replication packages, three-quarters failed to run without error, and more than half still failed after automated repair.3 In biomedical papers specifically, of notebooks tied to published articles, about one in ten could be re-run and one in twenty reproduced the original result.4 Four studies, different languages, different sources, one finding.

The response was real engineering, freezing environments and pinning software so a computation carries its dependencies with it. That helped. It made the software trustworthy. It said nothing about the steps the analysis took, and those steps are now chosen by a machine.

The missing piece is the process itself: a record of what was done, bound to what came out. A recording studio has worked this way for years.

Think about how a finished track gets made. You start with the raw recordings, straight from the microphones, unshaped. Then you run them through a chain of processing: EQ to balance the tone, filters to cut what you don't want, effects, level adjustments, each applied in a particular order at particular settings. Change the order or the settings and you get a different result. The polished track that comes out the end, the master, is simply what that chain produces from those recordings. And here is the catch: the master alone doesn't show you the chain. Hand someone only the finished track and they cannot see what you did to get there. They have to take it on trust.

A data analysis is the same shape. The raw data are the recordings. The analysis is the chain: every filter, transformation, and step applied to turn raw numbers into a result, in a specific order, with specific settings. The result you report is the master. And a result reported on its own has the same problem as a track handed over on its own. It doesn't show the chain that produced it.

raw data the recordings Record session-record Write session-write Execute session-execute master session-witness
The chain, and the stamp that proves the master belongs to it.

This is exactly what the session suite is built to do. It comes from the Swamp, which models real work as typed, recorded events instead of one-off scripts, and it borrows that logic on purpose. The name is not an accident: a session, in a studio, is where the recording and everything done to it are captured together, and these tools treat an analysis the same way. session-write fixes the setup so the shape of the processing is locked before anyone touches it. session-record captures the raw inputs and every step applied to them. session-execute runs that chain start to finish and produces the result, stamped with a fingerprint, a short code unique to exactly that output, which session-witness keeps. The result no longer arrives on its own. It arrives with the full chain that made it, and a stamp proving the two belong together.

The fingerprint has one honest limit, and it is worth naming. It does not tell you the science is right. It tells you what happened, and whether it happened the same way twice. A sealed master can still be a bad mix. What you get is not proof that the analysis is correct, but proof of what the analysis was, which is what you need before any argument about correctness can begin.

A sealed master can still be a bad mix. What you get is proof of what the analysis was.

The session suite runs in both directions. Forward, at your own work: fix the setup, run the analysis, stamp the result. And backward, at work done somewhere else. When a model runs an analysis inside a proprietary tool, that work is real and completely stuck, locked in a format you do not own. session-ingest lifts the whole session out with its steps intact, brings it home, and runs it again to confirm the result still matches its stamp. It does not redo the science. It brings it back under your control.

There is one place this stops being convenient and becomes necessary. When an AI pipeline reads patient data and produces something a clinician acts on, "trust me" is not an answer, and regulators are starting to say so out loud. A session that captures every input and proves what the model did is what that setting will ask for.

Look at what the Swamp already holds. Almost all of it is infrastructure: servers, cloud accounts, certificates, secrets, clusters. There is more than infrastructure, too: notes kept in Obsidian, issues tracked and triaged, plans made. Thousands of installs, and it holds up. What none of it touches yet is science.

That is the opportunity. Everything the Swamp already does well, the typed records, the fixed setups, the stamped results, can be brought to how science is done, and the session suite is where that starts. For now it works where analysis is already software: computational biology, where the chain is code and the result is a figure or a number. That is the natural first home, and it is where the suite runs today.

The frontier is further out. The same logic that seals a computation can seal a protocol. Picture a full lab procedure run this way: the steps fixed in advance, every input captured as it happens, the whole run stamped and replayable as an account of exactly what was done. The biology will still vary, as biology does. What becomes fixed is the protocol and the record of the run, so a wet-lab experiment can carry the same sealed chain a computation does. That is the next thing to build.

A result that travels the way a record travels: a stamped final result, a fingerprint anyone can check, and the full chain that made it attached, whether the analyst was a postdoc or a model. Analysis cited by its signature instead of its screenshot. That is what the Swamp offers science. Not a cleverer way to work, but a way for good work to travel without losing what makes it trustworthy.

Play a record twice and you hear the same thing twice.
It is time an experiment did the same.

Press play yourself

The suite is a set of swamp extensions. Pull them once and you have both directions — pressing your own master, and lifting a locked one home. The commands below show the shape of each move; every step is one typed, recorded swamp method.

$ swamp extension pull @vcjdeboer/session-suite

A A-side · press a master

1 · Fix the setup. Fill a typed analysis template so the shape of the processing is frozen before any data moves — session-write checks the fill left the structure intact and every parameter satisfies its slot contract.

$ swamp model method run writer validate \
    # the filled template, structure frozen
    --arg qmd=analysis.qmd
✓ structure frozen · 6/6 slots satisfy their contracts

2 · Run the chain. session-execute runs the filled template's code headless in a pinned nix environment with the recorder armed, then verifies the output against the template's declared swamp.returns contract.

$ swamp model method run executor run \
    --arg qmd=analysis.qmd
✓ ran in locked env · recorder captured 41 steps
✓ output matches swamp.returns

3 · Seal it. session-witness chains the session's records into one sha256 digest and attests its authors. That digest is the fingerprint — the stamp that says this master came from this chain.

$ swamp model method run witness seal \
    --arg session=analysis
⬢ sealed · sha256:9f3c…a71e · 1 author attested

B B-side · lift a locked session

1 · Look inside. A model's work done in a proprietary tool sits in a format you don't own. session-ingest reads your own quiescent session read-only and emits a manifest of what's there — no mutation, never touching secrets.

$ swamp model method run ingest inspect \
    --arg session=my-analysis
manifest · 77 cells · 9 artifacts · 3 external sources

2 · Bring it home. Capture the pieces as portable, typed swamp resources — the ordered cells, the byte-for-byte corpus, the external-data inventory, the environment locks.

$ swamp model method run ingest capture_cells   --arg session=my-analysis
$ swamp model method run ingest capture_corpus  --arg session=my-analysis
$ swamp model method run ingest lock_env        --arg session=my-analysis
✓ 77 cells · corpus sha-verified · env pinned

3 · Seal what you lifted. Stamp the recovered session so it, too, carries a fingerprint anyone can check — the locked work is now back under your control, and travels like a record.

$ swamp model method run ingest seal \
    --arg session=my-analysis
⬢ sealed · sha256:2b8d…c4f0 · portable & replayable

The open links

References

  1. Pimentel, J.F., Murta, L., Braganholo, V., & Freire, J. (2019). A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks. MSR 2019, 507–517. doi:10.1109/MSR.2019.00077
  2. Wang, J., Kuo, T., Li, L., & Zeller, A. (2020). Assessing and Restoring Reproducibility of Jupyter Notebooks. ASE 2020, 138–149. doi:10.1145/3324884.3416585
  3. Trisovic, A., Lau, M.K., Pasquier, T., & Crosas, M. (2022). A large-scale study on research code quality and execution. Scientific Data, 9, 60. doi:10.1038/s41597-022-01143-6
  4. Samuel, S., & Mietchen, D. (2024). Computational reproducibility of Jupyter notebooks from biomedical publications. GigaScience, 13, giad113. doi:10.1093/gigascience/giad113
← Back to vcjdeboer.github.io