Start with a screen recording
Näky reads an AV1 screen recording, whether it captures continuous motion or a timed series of screens.
Screen perception for reasoning models
Screen recordings in.
Structured state out.
Näky accepts an AV1 screen recording and produces compact, chronological text for a reasoning model: what text was observed, where it was, what persisted, and when the screen changed. The model can follow a task over time without receiving every frame.
From recording to useful context
A small, evolving account of the screen.
Give the model chronological text
Observed text and screen regions become a compact state that records what persists and what changes over time.
Keep pixels for visual detail
Appearance, icons, color, fine visual state, and anything OCR misses may remain available only in pixels.
Loading recording, one shared playhead
See the pixels and the state together
What the model receives
Loading the public recording gallery…
A text model receives the exact state stream on the right. The recording remains visible here only as a reference.
01
Current-state overlay
What the screen shows
Loading authentic output…
Video playback is unavailable, but the timeline and state stream remain inspectable. Download the selected source MP4.
@ timeS screen= + ~ > − set · add · change · move · remove! observed activity
Read the overlay literally
Observations, not inferred actions.
OCR-derived text and boxes are observations and can be wrong. Geometry and stable IDs show what Näky associated over time.
The live overlay is a clean view of retained observations, not event history. After reported activity, a region awaiting the next structural sample remains location-visible as a faint neutral outline and is counted with the recording; the exact state and raw streams remain unchanged.
Visual activity means pixels changed in a region during an interval. It is not a click, scroll, focus, selection, action, intent, or cause label.
The released comparison found higher quality from screenshots. Näky scored 82.61% versus 88.21% for screenshots across 3,468 tasks, while using 50.91% fewer logical three-trial input tokens and an estimated 46.50% lower cache-aware reader cost. Read the full matched comparison.
Demo source
Select a recording to see its source. Third-party names, marks, and interface content belong to their owners. Full attribution and transformation notes.