Visual Intelligence & Computer Use
Proteus is an AI engineering lab building the next generation of computer use agents, combining fast visual understanding, efficient context compression, and RL to let machines learn to operate multi-interface workflows on any device, autonomously.
Our Thesis
01 — Fast perception
Fine-tuned compact models for instant visual grounding and action prediction, paired with large VLMs that handle high-level planning and reasoning at different latencies simultaneously (think: fast model acts and collects data to generate rollouts, while slow model determines correct actions).
02 — Efficient encoding
We use fast OCR, segmentation, and video models to build optimized visual representations that eliminate the redundancy of feeding raw screenshots into frontier models -- leading to faster inference, longer horizons, and lower cost.
03 — Continuous operation
Agents should learn diverse, long-horizon workflows from video and be able to navigate real interfaces on desktop, mobile, and the web. Annotated DOMs or HTML should be only used when learning new workflows, not at inference time.
Demo
We built lightweight action interfaces for desktop, web, and mobile, with a shared brain to handle reasoning and reflection. Here's a preview.
Real-time Android agent · Prompt: "Find me an Airbnb in San Francisco for 10 people this weekend. Money is no object, pick the best place." · Real-time, clicks where needed, handles popups, recovers from failure, and asks the user for clarification when intent is ambiguous.
Rapid computer use on Android and iPhone. The agent sees the screen, reasons about what to do, and takes action, navigating apps, filling forms, and completing multi-step tasks with no predefined scripts.
Powered by compact vision models for fast grounding and frontier LLMs for planning, with custom optimizations for efficient visual context that keep latency low and accuracy high.
Native computer use on Windows desktops. Operates Win32 and UWP apps, manages files, runs multi-step workflows across the OS.
Full desktop automation on Ubuntu Linux. Navigates GNOME, terminal, and GUI apps with the same visual understanding pipeline.
Seamless computer use on macOS. Controls native apps, Finder, and system interfaces through real-time screen understanding.
Platform
01
Custom trained video model for rapid action with continuous screen understanding. Streams desktop and mobile frames through learned compression, maintaining temporal context without encoding every raw screenshot.
02
5× faster than state-of-the-art OCR and segmentation models with SOTA performance, purpose-built for UI understanding. Extracts text, bounding boxes, and semantic layout from screens.
03
Planning harness built on foundation models and fine-tuned VLMs with extensive computer use context baked into LoRA adapters. Decomposes complex goals into executable action sequences across any interface.
04
We built labeled screen recording software at screencap.sh. Captures action-labeled desktop and mobile recordings for training data collection at scale.
05
Offline RL training on our trajectory recordings paired with online RL training using OS-level forking for GRPO rollouts. Fork the OS state, explore multiple action trajectories in parallel, and learn from the outcomes in simulation.
06
Optimized perceptual hashing with DinoHash, a state-of-the-art hash we invented that outperforms Meta's and Apple's best algorithms, combined with aggressive VLM caching and inference optimizations to minimize time to first token and redundant computation across frames.
Team
Our team's published research prior to Proteus.
We also contributed to work on decoding visual imagery via fNIRS, adversarial examples, and reinforcement learning.
Our team comes from
We partner with teams building computer use agents, visual AI infrastructure, and autonomous systems that need to see and act in the real world.