Command understandingBeta

Intent

From "click Save" to a target.

Command understanding

The co-processor that tells the brain where.

Intent turns a short instruction plus a screen into a target embedding the control weights can act on. It is the bridge between language and motion, and it ships as a co-processor for Cursor and Keys.

A small vision-language model reads the instruction and the screen and emits a target embedding. The embedding is injected into the afferent population through the target adapter, and the brain takes it from there.

Finds the button 94% of the time.

On ScreenSpot, Intent grounds the named element in 94% of desktop screenshots with a 380 MB co-processor, and hands the target to Cursor in one step.

target Figures marked "target" are design targets for the first release, not measurements.

94%targetGroundingScreenSpot desktop
380 MBtargetCo-processoron disk, 4-bit
How it is built

Same brain, its own wiring.

Input adapters occupy afferent regions, output adapters read efferent ones. Co-processors feed in through the embedding port.

The 3D view needs WebGPU or WebGL2.
Engine
  • FlyCore v0.1.0
  • weights v0.1.0-beta
Input adapters
  • instruction
  • screen
Output adapters
  • target embedding
Co-processors
  • vision-language model
Python
import flycore

brain = flycore.load("flylabs/intent")      # engine + weights + adapters
cursor = flycore.load("flylabs/cursor", coprocessors=[brain])
cursor.do("click Save")
# target grounded by Intent, motion by Cursor
Use cases

From "click Save" to a target.

Natural-language UI automation

"Export as PDF" becomes a sequence of Cursor and Keys actions.

Limits
  • Beta: English instructions.
  • Requires Cursor or Keys to act; Intent alone only produces targets.