A compound eye for your camera.
Gaze detects and tracks objects in video through the connectome's visual system, with a classical CNN co-processor for class labels. Built for motion, so it finds what moves first.
Frames are resampled onto the hexagonal photoreceptor grid, then flow through the optic lobes as they do in the fly. The intrinsic population encodes position and motion; a light CNN co-processor names the object. Together they give a box, a class and a velocity per object.
30 fps on a laptop. Moving objects first.
Gaze runs at 30 fps on Apple silicon and reaches 0.71 mAP on COCO-motion, the subset of COCO video where objects move. Static clutter is intentionally down-weighted.
target Figures marked "target" are design targets for the first release, not measurements.
Same brain, its own wiring.
Input adapters occupy afferent regions, output adapters read efferent ones. Co-processors feed in through the embedding port.
- FlyCore v0.1.0
- weights v0.1.0-beta
- frames (compound eye)
- boxes, classes, velocities
- CNN classifier
import flycore
brain = flycore.load("flylabs/gaze") # engine + weights + adapters
for frame in flycore.io.Camera():
objects = brain(frame)
# objects -> [Object(box, cls, velocity, score), ...]What is where, every frame.
Robot and drone vision
Detection that shares a brain with the controller, so tracking and steering are one loop.
Retail and traffic counting
Count what passes a line, on the device, with nothing uploaded.
- Beta: 80 COCO classes only; custom classes need a co-processor fine-tune.
- Weak on static scenes by design.