This page is out of date: the Lab was updated since it opened.
Autopilot
Replays a debate through the pipeline, scores the board and suggests better settings. Nothing changes until you apply it.
Sources
?
to
Run
?
?·min?
Start from?
Models
On OpenRouter, the paid API the product runs on. Yours to choose; the autopilot never touches these.
The fact-check judge is a re-check: the same positions checked again, twice, by this model. Empty makes it a consistency check rather than a second opinion.
Judging
How much each axis counts?
The goal
What a perfect board is. Quoted verbatim into the judge's and the proposer's prompts, so editing it re-aims the whole loop.
Agents now
Each agent lights while it runs; its last call's time and cost sit under it.
Lineage
Algorithms
What each run produced: the version it was saved as or proposed, or else its best generation. Download one to keep it or take it to another Lab; apply one from App Config.
Record
Every run so far?
Live log
Pipeline
Drag or scroll to pan. Pinch to zoom. Click a step for its settings.
App Config
Every setting the app runs on. Change a setting on staging; production gets it when you promote it.
No setting matches that.
Scoring
The golden set the scoring engine is held to. A model labels every item three times, you review each label, and the check scores today's engine and the new one against your review.
Review
Scoring check
Replay
Plays a saved debate through the relay in real time, so you can watch the board build.
Runs on your machine. A replay streams the fixture's audio lanes to a relay and plays them out loud, and this runner holds no audio. Start the Lab locally for this.
Fixture
?
Where it runs
?
Production is a real, live debate on your public board for the length of the fixture, using your plan's hours and real transcription and model spend. You are asked before it starts.
About 2,600 people watch. The fake accounts write comments. Both run while the debate is live. Staging and local only.
Set it up like a debate
?
Every field is optional. Empty means the fixture's own title and question, and the column defaults the wizard starts from.
Ingest & Label
Turns a YouTube debate into a test debate: the audio, who spoke when, and a name for each voice.
New fixture
?
A recording from this computer, audio or video, up to four hours. It is decoded here and sent once to Deepgram to tell the voices apart; nothing keeps it, and the lanes are written into your fixtures folder.
Videos on this machine
?
Each video you ingest leaves a working copy here: the mixed audio and Deepgram's segments. Labelling one cuts a fixture from it, which the Autopilot and Replay tabs use.
Verify Users
Credential applications and their proof. The AI review suggests; you decide.
No applications in this database yet.
All Debates
Every debate, including deleted ones until they are erased 30 days later.
Debates
0
Command
Health, spend and switches for the whole app.
Activity
Users
Every account, with its details, bans and reported comments.
Open reports
0
Accounts
0
Rewards
Looks viewers earn and wear: made here, given in rank cases and for right calls on markets.
Studio
Pick a kind, then a look. Every card is the reward as the app draws it.
Library
Cases
Finishing a rank earns its case once. Opening it rolls one reward by weight, and a reward the viewer already owns is rolled last.
Markets
A reward dropped on a market goes to everyone whose call on it settles right.
This runner cannot manage rewards.
Markets
The Kalshi markets the app offers. Payouts are set in App Config, under Predictions.
On the app
Add from Kalshi
Words, or say what you're after. When the words find nothing, Jev reads what you meant. Add one outcome, or the whole topic with every outcome still trading.
Suggested topics
Pictures
From Wikimedia Commons only: public domain and CC0 first, then CC BY and CC BY-SA. A picture is approved for a subject, and every market whose words name it shows it, old ones too.
The library
This runner cannot manage markets.
Sources and the hold-out
Sources are where a change is found. The hold-out is where it is believed.
A change is kept because it won on the sources, which is grading its own homework. It may have learnt that quarter hour rather than got better at debates.
So before a run proposes anything, what it kept is replayed against where it started on the hold-out, minutes the search never saw. It is proposed only if it wins there too.
Leave it on. With it off, each tested generation is one more chance for luck. A two-generation run of changes that do nothing proposes a version one time in twenty at most, and a five-generation run between one time in ten and one in five. With it on, under one time in twenty however long the run.
It must not overlap a source, and the form says so before Start if it does. Times are points in the recording. A fixture marked folder is in the fixtures folder you connected; one marked disk is on the laptop that runs the Lab.
Generations, shots and candidates
One generation tests one change per candidate, each to one setting. The settings as they stand, called the parent, and the changed ones are replayed side by side on every source. It takes about as long as the slice, because the replay is real time.
Shots are how many times each side is replayed. Every board of one side is read against every board of the other, in both orders. The judge is not told which is which.
With three shots on one recording, a change that wins at least 7 of its 9 pairings earns a second look: two more replays a side. It is kept only if it then holds 21 of all 25 pairings and no score clearly falls.
A change that does nothing passes one time in twenty at most. Anything less is reverted, and so is a change that only ties the parent.
Two replays of the same settings differ, and that gap is the yardstick. A change three gaps better is kept 19 times in 20. One two gaps better is kept 3 times in 4, and one a single gap better about 1 time in 4.
Candidates are rival changes in one generation. More rivals mean more chances for luck, so with two a change has to hold 23 of its 25.
The run starts from the pipeline algorithm staging runs right now. The cap covers the whole run. At the end, a run whose kept changes held up on the hold-out proposes a new version and asks; it never applies one.
The fact checker is measured only when its step is unlocked on the Pipeline page: two checks a shot, then none. Money for a number nobody can act on is money wasted.
The estimate is priced from the newest replays of the pipeline model you picked, a dozen at most, because an older pipeline cost half as much. It shows a middle and a likely range. The default cap is the top of that range for the default form.
Before a run, and before each generation, the Lab reads what OpenRouter has left. It will not start a step the balance cannot pay for, and says how much is needed.
When a run ends early the banner at the top says why and what to do. What it measured is kept. Continue starts a new run from its last kept generation with the same settings, and Start it again runs it from the beginning.
A deploy to main restarts the hosted Lab and interrupts a run on it. On a laptop the Lab keeps the machine awake while a run goes, but a closed lid still sleeps and ends it.
What the scores are for
The head-to-head judge decides. It reads two boards made from the same minutes and says which a viewer would rather have. The scores here never keep a change.
Each shot's positions score is 0.65 the judge's axes and the viewer eye under these weights. The other 0.35 is the code checks, which cost nothing. It is what a shot's bar shows and what the next change is aimed at.
The scores cannot keep a change, but they can veto one. A change that wins its pairings is still refused if this score, the fact-check score or the viewer score clearly falls.
A fact-check or viewer change leaves the board as it was. So its replays are paired on its own score, not on the boards.
A weight of 0 silences an axis; 2 doubles it. The default weights put attribution and position fidelity first, because a wrong mouth on a public surface is the failure the product cannot afford.
The goal document is the rubric every judge reads. It is versioned content: what ships is in the repo, and an edit here is kept in the database and overrides it until you go back.
Reading the record
Generations run left to right. One setting changes between a parent and its candidate.
Top rows are each recording's score out of 10, coloured by band: under 5, 5 to 7, 7 and above. Aggregate is the candidate's own score, Change its move from the parent replayed in the same generation.
Pairings is how many the change won against the parent's boards, and how many it needed. Kept became the next parent. Reverted did not, and that includes a change that only tied.
The column headed Hold-out is the confirmation: what the run kept against where it started, on minutes the search never saw.
Where a replay runs
Local streams to the relay on this machine, ws://localhost:8787/ws. Staging streams to the deployed staging relay, so the debate appears on the staging site exactly as a real one would. Same database as local.
Production streams to the live relay, from the deployed Lab only: a genuine debate on your public board, owned by the streamer in LAB_OWNER_HANDLE_PRODUCTION, that pays to transcribe and analyse the whole fixture and spends that account's plan hours.
Setting a replay up
The same framing and model choices the debate wizard asks for, minus the audio step and the speakers, which the fixture supplies. Leave a field empty and the fixture's manifest, or the column default, fills it.
What a replay does
Streams a fixture through the relay as a real debate: real sockets, transcription and analysis, at wall-clock speed. Nothing is graded and nothing is applied. Nothing reaches a paid provider until you press Go live in the app; after that it costs what a real debate of that length costs.
To end one, end or delete the debate in the app; the Lab notices and stops the replay.
What ingest does
Downloads the audio, asks Deepgram who spoke when, plays you a sample of each voice, and you put a name to each one. What comes out is a set of isolated per-speaker lanes, which is what the relay expects from real speakers on real microphones, plus a transcript the autopilot grades against. Saving writes it into relay/dev/fixtures; nothing goes to a server.
What is kept here
One working copy per video you ingest: the mixed audio and Deepgram's segments. They stay on this machine, and are never uploaded, committed or deployed. Labelling one cuts a fixture from it. They are large; deleting a video frees the disk and loses nothing but the ingest time.
