Can a model read an entire physical book from a shaky phone video of someone flipping through it?

Hypothesis

We had one source for a 119-page Norwegian book: a five-minute phone video of someone flipping through it, a hand covering 15–30 % of every single frame. The classic answer is an OCR pipeline. We built one — frame extraction, 4K upscaling, EasyOCR with Norwegian, position-aware stitching across frames, a dictionary of Norwegian OCR corrections. Seven iterations later it still produced roughly half garble on body text, and at one point confidently logged the same spread as three different pages.

The hypothesis worth testing: skip OCR entirely. A vision model reads pages the way a person does — so let it.

What we did

Extract frames at 1 fps, 1080p, nothing fancier. Then have the model read the frames directly and type out what it sees, verbatim. The hand problem turns out to solve itself: the hand moves between consecutive frames of the same spread, so a covered sentence in one frame is usually visible in the next. Whatever stayed hidden in every frame got marked as hidden — never guessed.

To make it fast, four agents each took a quarter of the book and wrote one markdown file per printed page. To make it trustworthy, four verifier agents then re-read every page against a different frame than the one used for transcription and diffed word by word. Figures — the tax-line diagrams, the compound-interest graph — were cropped straight out of the frames with ffmpeg and embedded as images. A ten-line shell script assembles the pages into one readable book file.

What happened

The whole book, transcribed and verified, in one afternoon. The verification pass found seven minor issues across 119 pages — three highlighter spans that started a few words off, two paragraphs wrongly italicised, two annotations — and everything else was word-perfect, including two tax tables digit for digit. Eight small hand-covered gaps are honestly marked instead of invented. The verifiers even caught four printing errors in the book itself, like a summary that says 403 000 kr where the numbers add to 413 000.

The OCR pipeline we spent most of the day building was deleted from the workflow. The lesson generalises: when a vision model can look at the source directly, a recognition pipeline in the middle is usually the weakest link.