Screentack for AI agents

list_windows

List all visible application windows with their app name, title, position, and dimensions.

No parameters.

capture_window

Capture a screenshot of a specific application window. Returns the image as base64 PNG. Use list_windows first to find the app name.

• app_name (string)required — Name of the application (case-insensitive substring match, e.g. 'chrome', 'terminal')

capture_region

Capture a screenshot of a specific screen region in pixel coordinates. Pass ocr:true to also return text recognized in the region.

• height (integer)required — Height in pixels
• width (integer)required — Width in pixels
• x (integer)required — Left edge X coordinate in pixels
• y (integer)required — Top edge Y coordinate in pixels
• ocr (boolean) — If true, also OCR the region and include recognized text in the result (default false)

activate_window

Bring an application's windows to the front. Call this before capture_window if other windows might be blocking the target. Returns the number of windows raised.

• app_name (string)required — Name of the application to bring to front (case-insensitive substring match)

capture_window_region

Capture a specific region within a window at readable resolution. Coordinates are relative to the window (not the screen). The cropped area is returned at up to 1024px — great for reading text, UI details, or small elements. Use this instead of capture_region when you know which window you want and need to zoom into a part of it. Pass ocr:true to also return recognized text.

• app_name (string)required — Name of the application
• height (integer)required — Height in pixels
• width (integer)required — Width in pixels
• x (integer)required — Left edge X in pixels, relative to window top-left
• y (integer)required — Top edge Y in pixels, relative to window top-left
• ocr (boolean) — If true, also OCR the region and include recognized text in the result (default false)

capture_annotated

Capture a full window screenshot with highlighted regions. The regions are rendered as colored overlay markers on the screenshot, so the LLM sees both the full context and the specific areas of interest. This is the primary tool for visual debugging — it solves the 'zoomed-in loses context, zoomed-out dilutes signal' problem.

• app_name (string)required — Name of the application
• regions (array)required — Regions to highlight on the screenshot

capture_diff

Capture before/after screenshots of a window. First call stores 'before' state. Second call with same app returns both 'before' and 'after' images with timestamps. Use this to show what changed after an action.

• app_name (string)required — Name of the application

start_recording

Start recording visual changes of an application window. Captures at ~1fps and detects which screen regions changed using grid-based diff. Saves start frame, end frame, and crops of changed regions. Returns the session ID.

• app_name (string)required — Name of the application to record
• interval_ms (integer) — Capture interval in milliseconds (default: 1000)

stop_recording

Stop the current recording session. Saves end frame and writes SESSION.md with the full timeline of visual changes. Returns the session ID and path.

No parameters.

poll_recording

Poll for new visual change events during an active recording. Returns text-only event list with frame filenames (no images). Use this to check what changed, then use read_frame on specific frames if needed.

No parameters.

read_frame

Return a raw recording frame as an image. Use to visually inspect recording frames. Note: puts the image into your context window.

• frame (string)required — Frame filename (e.g. 'change-001.png', 'frame-000-start.png')
• session_id (string) — Session ID (default: latest)

get_annotations

Read what the HUMAN marked — the latest session data, either annotations from the interactive overlay (Cmd+Ctrl+P) or human-initiated recordings (Cmd+Ctrl+R). Returns SESSION.md content with timeline, key area readings, and region images. Use when the user says they've annotated, recorded, or captured something. This is the human→agent half of the feedback loop; its inverse is annotate_understanding (you draw for the human). Human recordings include Source: human in the metadata. Pass ocr:true to also OCR each region and include recognized text.

• ocr (boolean) — If true, OCR each region image and include recognized text (default false)

annotate_understanding

Draw YOUR understanding/intent onto the user's LIVE screen as labeled shapes (boxes/arrows/ellipses/highlights/text) and reference images, so the human can confirm or correct it before you act. Each stroke declares the frame it was authored against — a window (from list_windows/capture_window), a display, or a screen region (from capture_region) — plus pixel coords within the image you reasoned over. Strokes appear on a present/draw overlay as EDITABLE shapes the human can move, relabel, or delete. Use it to show 'this is the primary CTA I'll restyle', 'I read this value as $69.23', or 'make it look like this <image>'. This is the agent→human half of the feedback loop; its inverse is get_annotations (read what the human marked). Returns a JSON result block with sessionId/name/path. Free — no license required.

• strokes (array)required — The shapes to draw onto the live screen.
• title (string) — Short name for this annotation set (shown to the human in History as the session Name).

get_annotation_feedback

Read the human's review of a drawing you presented with annotate_understanding. Poll this with the sessionId from that call. Returns status ('pending' until the human submits, then 'submitted'), the human's `comments` (their feedback marks — the actual signal: what they typed and where it points, with displayIndex + normalized x/y), and your own `agentStrokes` with their state (removed:true means the human cleared that mark — treat as context, NOT as a rejection verdict). Interpret meaning from the comments and the conversation, not from which marks were removed. A composited image is also returned, but it covers only the PRIMARY display — comments on a secondary monitor appear in `comments` (with their displayIndex) but not in the image; capture_region that display if you need to see it. Free — no license required.

• sessionId (string)required — The sessionId returned by annotate_understanding.