Teaching a fly brain to use a mouse

Cursor is the first weight set we ship, and the one that taught us most about the method. This post walks through how a fly brain ends up moving a mouse. It is a placeholder for the full write-up, which will follow with measurements.

The input side

A fly sees through about 800 ommatidia. A 4K screen is a lot of pixels for that. Cursor gives the brain two views: a coarse map of the whole screen and a foveal window around the cursor, both resampled onto the hexagonal photoreceptor grid and mapped onto the 11,390 visual afferents.

The cursor’s own position and button state enter as proprioception, on the ascending neurons that in the fly carry body state up from the nerve cord.

The target is the interesting part. In the first release it is a marker on the screen. With the Intent co-processor it is an embedding of an instruction, “click Save”, injected through the olfactory port: the 2,282 olfactory receptor neurons are the fly’s native high-dimensional input, and they make a good embedding port.

The output side

Motion leaves through the 1,303 descending neurons, the same population that drives legs and wings. A small decoder reads them at 100 Hz into dx, dy and three button states. Double-click, right-click and drag are not separate actions; they are temporal patterns the recurrent brain learns.

Training

Two stages. First, imitation from recordings of people using a mouse, with the screen and events captured at 100 Hz, so trajectories acquire the acceleration profile, the overshoot and the correction of a real hand. Second, reinforcement in a synthetic Fitts-law environment: random targets, reward for hitting them, penalty for time.

What we measure

Median time to target, click accuracy inside the target, and a human-likeness score: the correlation between Cursor’s velocity profiles and recorded ones on the same target geometry. Those three numbers are the ones on the Cursor page, and they are targets until the first release replaces them.

What it is not for

Cursor is licensed for interfaces you own or are authorised to automate. It is not a tool for evading bot detection, and we will not position it as one.