A fly brain learns to use a mouse

Cursor was the first task we gave the fly brain: move a mouse pointer to a target and click.

How the brain is connected

The screen goes into the photoreceptors of the compound eyes. Where the pointer is and which button is pressed goes into the ascending neurons, which in the fly report from the nerve cord about the body. The target position goes in as a pattern on the olfactory receptor neurons of the antennae. The machine reads the pointer movement and the click from the 1,299 descending neurons, which in the fly carry the brain’s commands to the nerve cord. The 15.1 million connections stay unchanged.

Measurement

Cursor is measured on 1,000 targets of a Fitts test, the standard test for pointing movements, and compared with people on the same task. The fly brain hits 64.2% of the targets at a median movement time of 0.49 s (people: 0.62 s).

Speed

The demo runs in the browser, at about 4 ms per step with WebGPU and about 13 ms with WebAssembly (SIMD).