Desktop Control

Usejarvis controls native desktop applications through a Go sidecar process that connects to the daemon over a JWT-authenticated WebSocket. The sidecar runs natively on each platform — Windows, macOS, and Linux — using platform-specific APIs for window management, UI automation, screenshots, and input simulation.

Architecture

Usejarvis Daemon (Bun, any machine)
  ↕ WebSocket (JWT auth)
Sidecar (Go, target machine)
  → Platform APIs (Win32 + UI Automation / AppleScript + JXA / X11 + AT-SPI2)
  → Chrome DevTools Protocol (over an inherited pipe)
  → Terminal, Filesystem, Clipboard, OCR
  → Native UI: webview panels, pebble overlays, tray, notifications

The sidecar is a standalone Go binary that enrolls with the daemon using a JWT token. Once connected, it receives RPC commands over WebSocket and executes them using native platform APIs. Multiple sidecars can connect to the same daemon, giving Usejarvis control over several machines simultaneously.

Installation

Install the sidecar on each machine you want Usejarvis to control:

bun install -g @usejarvis/sidecar

Or download the prebuilt binary for your platform from the releases page. Each release ships archives for macOS (arm64, x64), Linux (x64, arm64), and Windows (x64), e.g. jarvis-v0.9.0-darwin-arm64.tar.gz. Extract and run the jarvis binary inside.

The installed binary is called jarvis. Update it later with bun update -g @usejarvis/sidecar (or grab the newer release archive) and check the version with jarvis --version.

Enrollment

Since 0.9, devices are enrolled from the CLI on the machine running the brain, not from the dashboard.

1. Install the sidecar

See Installation above.

2. Enroll the device on the brain

On the machine running the daemon:

jarvis enroll "work-laptop"

If the daemon runs in Docker:

docker exec jarvis jarvis enroll "work-laptop"

This mints a long-lived enrollment token (ES256 JWT) and prints it. Enrolling an existing name re-mints its token; add --rotate to also invalidate all previously issued tokens for that device.

3. Run the sidecar and paste the token

On the target machine, start the sidecar:

jarvis

The first-run window asks for the token (self-hosters: click Paste your enrollment token). You can also pass it directly:

jarvis --token <your-token>

The token is saved to ~/.jarvis/sidecar.yaml, so on subsequent runs you just need jarvis.

Once connected, the sidecar appears as online in Settings → Sidecar, where you can configure its capabilities and see its version status.

After enrollment, the sidecar reconnects automatically with exponential backoff if the connection drops.

Managing devices

jarvis sidecars list [--json]   # list enrolled devices
jarvis revoke <sid>             # revoke a device

Device management works even when the daemon is not running, so you can manage devices over SSH. Revoking severs live sessions within about 30 seconds and blocks new connections immediately.

Capabilities

The sidecar advertises its capabilities during the preflight check. Each capability is verified at startup — only capabilities that pass the platform check are registered.

Capability	Description	Windows	macOS	Linux
terminal	Run shell commands	cmd.exe	sh	sh
filesystem	Read/write files, list directories	Yes	Yes	Yes
clipboard	Get/set clipboard content	PowerShell	pbcopy/pbpaste	xclip
screenshot	Capture screen to PNG	PowerShell	screencapture	scrot / import / gnome-screenshot
desktop	Window management & UI automation	UI Automation (COM)	AppleScript + JXA	xdotool/wmctrl + AT-SPI2
browser	Chromium-family control via CDP pipe	Yes	Yes	Yes
system_info	Hostname, platform, CPU info	Yes	Yes	Yes
ocr	Read text from screen regions	Windows.Media.Ocr	Vision (ocr-helper)	tesseract
awareness	Periodic screen/window capture	Yes	Yes	Yes
windows	Window observation events	Yes	Yes	Yes
pebble, sub_pebble	The ambient pebble overlays	Yes	Yes	Yes
file_watch	Watch folders for changes	Yes	Yes	Yes
processes	Process observation	Yes	Yes	Yes
notifications	OS notifications	Yes	Yes	Yes

Desktop Tools

When the desktop capability is available, the agent has access to these tools (agent-facing names shown; each maps to a sidecar RPC):

desktop_list_windows

List all visible top-level windows.

{ "windows": [
  { "hwnd": 123, "title": "Visual Studio Code", "pid": 4211, "process_name": "Code.exe",
    "class_name": "...", "left": 0, "top": 0, "right": 1280, "bottom": 900, "is_foreground": true }
] }

desktop_snapshot

Get the UI Automation element tree for a window — reveals buttons, text fields, menus, and other controls with numbered element IDs.

Input: pid (number, omit for the active window), depth (integer, optional)
Returns: nested element tree with id, name, control_type, automation_id, rect

desktop_click

Act on a UI element by its element_id from the last snapshot.

Input: element_id (number), action, value (optional)
Actions: click | double_click | right_click | invoke | toggle | select |
         set_value | get_value | get_text | expand | collapse |
         scroll_into_view | focus

desktop_type

Type text, optionally into a specific element.

Input: text (string), element_id (number, optional)

desktop_press_keys

Send a key combination. Keys are comma-separated.

Input: keys (string)

Key format: "ctrl,s", "alt,f4", "ctrl,shift,esc". Modifiers: ctrl, alt, shift, win/super.

desktop_launch_app

Launch an application by name or path.

Input: executable (string), args (string or string[], optional)

desktop_focus_window

Bring a window to the foreground.

Input: pid (number)

desktop_find_element

Search for a UI element and return its properties.

Input: pid, plus at least one of: name, control_type, automation_id, class_name
Returns: { id, name, automation_id, class_name, control_type, enabled, focusable, rect, patterns }

desktop_screenshot

Capture the screen to PNG.

Browser Tools

When the browser capability is available, the sidecar drives any Chromium-family browser (Chrome, Edge, Brave, Vivaldi, Opera, Chromium) over a CDP pipe. The browser launches lazily on the first browser tool call, headed by default (headless: true to hide):

Tool	Description
browser_navigate	Navigate to a URL; returns page text plus numbered elements
browser_snapshot	Re-scan the current page (DOM scan of interactive elements)
browser_click	Click an element by its id from the last snapshot
browser_type	Type into an input field
browser_hover	Hover an element
browser_press_key	Send a key to the page
browser_screenshot	Capture the page as PNG
browser_scroll	Scroll the page or an element
browser_evaluate	Execute JavaScript in the page context
browser_close	Close the sidecar’s browser session

Terminal & Filesystem Tools

Tool	Description
run_command	Execute a shell command with configurable timeout and blocked-command list
read_file	Read a file (respects blocked paths and max file size)
write_file	Write content to a file (respects blocked paths)
list_directory	List directory entries with types and sizes
get_clipboard	Read clipboard content
set_clipboard	Write to clipboard
capture_screen	Take a full-screen screenshot
get_system_info	Get hostname, platform, architecture, CPU count

The sidecar also exposes OCR (read text from screen regions), awareness capture, file/process/notification observers, and the ambient UI families (panels, pebble, tray, notifications) as internal RPC methods.

Platform Details

Windows

Desktop automation uses native UI Automation COM (via go-ole on a dedicated STA thread). This gives access to the accessibility tree of any Windows application: element trees, clicking via UIA patterns, value get/set, expand/collapse. PowerShell is used only for window enumeration, SendKeys, and screen capture. OCR uses Windows.Media.Ocr.

Windows 11 has the required WebView2 runtime preinstalled; Windows 10 may need it installed for the sidecar’s panel UI.

macOS

Desktop automation uses AppleScript and JXA:

• osascript for window listing, app launching, and UI scripting
• JXA for the accessibility element tree
• clicks via cliclick when installed, otherwise Quartz events through python3
• screencapture for screenshots, pbcopy/pbpaste for clipboard
• OCR via a bundled Swift Vision helper (ocr-helper)

Grant Accessibility and Screen Recording permissions in System Settings → Privacy & Security.

Linux

Desktop automation uses X11 tools plus AT-SPI2:

• xdotool for input simulation (required for the desktop capability, along with a DISPLAY)
• wmctrl for window listing and focusing
• AT-SPI2 for the UI element tree: install python3-gi and gir1.2-atspi-2.0
• xclip for clipboard
• scrot, import (ImageMagick), or gnome-screenshot for screenshots
• tesseract for OCR
• WebKitGTK 4.1 for the sidecar’s panel UI

Multi-Machine Setup

Connect multiple sidecars to a single Usejarvis daemon for cross-machine orchestration:

Enroll each machine under its own name on the brain, then connect each sidecar with its token:

# On the brain
jarvis enroll "workstation"
jarvis enroll "build-server"

# On machine A (e.g., your workstation)
bun install -g @usejarvis/sidecar
jarvis --token <workstation-token>

# On machine B (e.g., a build server)
bun install -g @usejarvis/sidecar
jarvis --token <build-server-token>

The agent can then reference machines by hostname when dispatching tools. For example, it can run a build on your server while monitoring the result in your browser locally.

Sidecar Configuration

The sidecar stores its config at ~/.jarvis/sidecar.yaml (written with owner-only permissions). The brain URL normally comes from the enrollment token itself; brain is an override:

brain: "ws://localhost:3142/sidecar/connect"   # usually derived from the token
token: "eyJ..."                    # JWT from enrollment
capabilities:                      # defaults: all 15
  - terminal
  - filesystem
  - clipboard
  - screenshot
  - system_info
  - awareness
  - desktop
  - browser
  - ocr
  - windows
  - pebble
  - sub_pebble
  - file_watch
  - processes
  - notifications
terminal:
  blocked_commands: []             # opt-in blocklist, empty by default
  default_shell: ""                # auto-detected
  timeout_ms: 30000
filesystem:
  blocked_paths: []                # opt-in blocklist, empty by default
  max_file_size_kb: 100
browser:
  executable_path: ""              # auto-detected
  profile_dir: ""                  # defaults to a per-browser temp profile
awareness:
  screen_interval_ms: 7000
  window_interval_ms: 2000
  min_change_threshold: 0.02
  stuck_threshold_ms: 120000
  ocr_enabled: true
  capture_dir: "~/.jarvis/captures"
preferences:
  start_at_startup: false
  ethereal_pebble: false
telemetry:
  enabled: true                    # or JARVIS_SIDECAR_TELEMETRY=0

Remote brains are dialed over wss by default; plain ws is used only for localhost and private IPs. Older config files are auto-merged with current defaults on load.

Building from Source

Requirements: Go 1.25 or later. The sidecar is a cgo program (webview, GTK/Cocoa overlays), so plain cross-compilation does not work; build on the target platform.

git clone https://github.com/vierisid/jarvis
cd jarvis/sidecar
make build        # produces ./jarvis for your current platform

Platform notes:

• Linux needs libwebkit2gtk-4.1-dev libgtk-3-dev build-essential pkg-config first
• Windows builds link with -H windowsgui (no console window)
• macOS also needs make build-ocr-helper (requires swiftc), or OCR runs disabled; make app-macos produces the .app bundle

Troubleshooting

Sidecar won’t connect

1. Verify the daemon is running: jarvis status (or docker ps for Docker)
2. Re-enroll from the brain: jarvis enroll "<device-name>" and paste the new token into the sidecar
3. Check firewall rules — port 3142 must be reachable from the sidecar machine
4. If the daemon is not on localhost, make sure daemon.brain_domain is set with an explicit scheme (http:// or https://), otherwise the sidecar may attempt a wss:// handshake against a plain-HTTP daemon
5. Run jarvis doctor for a connectivity check

Desktop tools not working

1. Verify the desktop capability passed preflight: check sidecar startup logs
2. On Linux, ensure xdotool and wmctrl are installed: sudo apt install xdotool wmctrl
3. On macOS, grant Accessibility permissions in System Settings > Privacy & Security
4. On Windows, ensure the sidecar is running with appropriate permissions

Screenshots are blank

• On Linux, ensure ImageMagick is installed: sudo apt install imagemagick
• On Windows, ensure the sidecar is not running in a headless/service context without desktop access