ollos-mcp

Eyes and ears for AI agents. Local, offline transcription, keyframes, on-screen text and a pre-publish review of any audio, video or image — as an MCP server, a CLI and a Node library. No Python, no cloud, no API key.

Docs: Tool reference · Design doc — decisions and measurements · Evaluation results · Contributing · Security · Changelog

11 minutes of screencast become 14 contact sheets and 3 KB of text. And it tells you if your API key is visible at 2:50.

npx ollos-mcp            # MCP server on stdio
npx ollos review talk.mp4

Ollos is Galician for eyes.

Contents: Why · Install · Tools · Sources · CLI · Library · How it works · Evaluation · Requirements & performance · Privacy & security · Configuration · Troubleshooting · Known limits · Roadmap

Why

Agents can't hear or watch. Today you either pay a transcription API, install a Python pipeline, or paste frames by hand. Ollos runs Whisper, speaker segmentation, perceptual-hash keyframing and OCR in Node, through ONNX Runtime, on your machine. The file never leaves it.

It was built for one workflow first — reviewing a screen recording before publishing — and grew into the general case: meetings, lessons, podcasts, downloaded videos.

Install

Node 20+. npm install brings its own ffmpeg (ffmpeg-static); a system ffmpeg is used if present.

Claude Code

claude mcp add ollos -- npx -y ollos-mcp

or in the project's .mcp.json (the same JSON works for Claude Desktop's claude_desktop_config.json, Cursor's .cursor/mcp.json and Windsurf's mcp_config.json):

{
  "mcpServers": {
    "ollos": { "command": "npx", "args": ["-y", "ollos-mcp"] }
  }
}

VS Code — .vscode/mcp.json uses servers instead of mcpServers:

{
  "servers": {
    "ollos": { "type": "stdio", "command": "npx", "args": ["-y", "ollos-mcp"] }
  }
}

Claude Desktop reads ~/Library/Application Support/Claude/claude_desktop_config.json on macOS and %APPDATA%\Claude\claude_desktop_config.json on Windows. Environment variables (OLLOS_HOME, OLLOS_YTDLP, …) go in an env object next to args; use absolute paths, ~ is not expanded.

CLI

npm i -g ollos-mcp
ollos warmup            # download the default ASR model and VAD (2.75 GB)
ollos warmup --all      # every model: fast ASR, speakers, search, OCR data (3.3 GB total)
ollos doctor            # check ffmpeg, models, free memory, cores

Models download on first use into ~/.ollos/models. Set OLLOS_OFFLINE=1 afterwards to forbid all network access.

Tools

Ten tools, one per distinct contract. Long work never blocks: it returns a jobId you poll.

Every parameter is documented in docs/TOOLS.md (one anchor per tool, e.g. ollos_transcribe); the tool descriptions the agent sees carry the same information.

Sources

source accepts a local path, a file:// URL, a Zoom local-recording folder (one audio track per participant), a direct https:// media URL, a video-site URL (YouTube, Instagram, TikTok, Vimeo, X, Loom… through yt-dlp) and a data: URI. URLs are downloaded once into the cache; the download runs inside the job and can be cancelled. Refused: private, loopback and link-local addresses on any redirect hop (OLLOS_ALLOW_PRIVATE=1 to allow), downloads over OLLOS_MAX_DOWNLOAD_MB, media over OLLOS_MAX_DURATION_SEC from any origin.

ollos transcribe "https://www.youtube.com/watch?v=eur8dUO9mvE" --lang en --from 0 --to 30

Tested commands, yt-dlp setup and proxy notes: examples/url-sources.md.

Results are concise by default and point to MCP resources (ollos://jobs/<id>/transcript, /ocr, /report, /sheet/<n>) for the full artifacts, so a 2-hour meeting doesn't flood the context window. Pass format: "detailed" when you want it all.

CLI

ollos probe recording.mp4
ollos transcribe meeting.mp4 --lang pt --vocab "Claude Code,n8n,webhook"
ollos keyframes lesson.mp4 --sensitivity normal --max-frames 120
ollos read-screen demo.mp4
ollos review episode.mp4 --platform youtube     # exit 3 = block, 1 = warn or failure, 2 = usage error, 0 = ok
ollos jobs · ollos job <id> · ollos events <id> · ollos cancel <id>

Add --json for machine output.

Library

import { createOllos, type TranscribeResult } from 'ollos-mcp'

const ollos = createOllos()
const { job, result } = await ollos.transcribe({ source: 'talk.mp4', language: 'pt', vocabulary: ['MCP'] })
const transcript = result ?? (await ollos.wait<TranscribeResult>(job.id)).result // inline when small, a job otherwise
console.log(transcript?.segments[0])

More: examples/library.ts (every capability) and examples/library-url.ts (a YouTube URL as the source).

ollos-mcp/core has no MCP dependency: use it from n8n, a script, a Lambda.

How it works, and what was measured

Numbers below were measured on an 11:37 screencast (1890×1080, webcam overlay) on a 16-core laptop. They are why the design is what it is.

Transcription. Silero VAD marks speech; Whisper only sees speech (fewer hallucinations, 20–40% less work on meetings). whisper-large-v3-turbo at 1.7× real time got "MCP servers", "n8n", "VS Code" right where whisper-base (5.4×) got all three wrong. The one phonetic miss left ("Cloud Code") is fixed by vocabulary. Two Whisper sessions in parallel measured slower than one (0.6–1.0×), so ASR concurrency is 1 and speed comes from VAD and from running vision in parallel instead.

Anti-hallucination. Whisper doesn't go quiet on silence — it invents "Obrigado." and "Subtitles by the Amara.org community". Four filters, from production experience shared by the Vexa project: exact blocklist per language, repetition-loop collapse, no-speech gate, impossible speaking rate.

Keyframes. ffmpeg scene detection at 0.3 kept 4 frames in 11 minutes of screencast; mpdecimate removed 0% (the cursor and streaming text change every pixel). A 64-bit perceptual hash (dHash) at Hamming ≥ 6 kept 20% — one frame every 5–8 s — and that is the default. Hard cuts, transcript anchors and a 20-second floor fill the gaps.

OCR. Tesseract on a full 1890-px frame missed an on-screen URL entirely; on a 3× upscaled tile it read it whole at 90% confidence in 2.8 s. So OCR runs per tile, and URL-like text is re-joined when OCR splits it ("up. railway .app").

Secrets. Three signals, because OCR garbles the secret more often than the words around it. On a real "API Key Created" modal the plain JWT regex missed (OCR read eyJ as eyl), the entropy detector caught the 157-char token, and the UI context read at 66%. With an OCR-tolerant JWT pattern, native-resolution frames and a centre tile, the end-to-end run now reports it as high · jwt · near "API Key" → block. The first version also produced 58 false positives by running the entropy test on whitespace-stripped text; that is a regression test now. Values are always masked; the tool that warns about a leak must not be the leak.

Change-detected keyframes are the wrong tool for this specific check: a credential left on an unchanging screen for a minute contributes one dHash candidate, and a frame cap's prune step used to drop that one candidate on the same terms as a genuinely redundant frame — real miss, filed as #3. ollos_review's secrets check now reads the screen at a fixed 4-second cadence regardless of whether the picture changed, protects that guarantee through the frame cap (preserveFloor), and states the coverage it actually achieved in the finding — Read the full 11:37 at a 4s cadence on a normal-length recording, or an explicit Partial coverage: … note bounded by OLLOS_MAX_FRAMES on a very long one, instead of a clean verdict that silently checked less than it implied.

Speakers. Segmentation alone labelled three speakers on a one-person video (its ids are local to each 10-second window). Embedding every turn ≥ 1.5 s, average-linkage clustering at cosine 0.35, and absorbing tiny clusters brought it to one. The evaluation then showed the real failure mode: the same voice scores 0.58–0.86 against itself across positions and lengths, but 0.06–0.16 once background music is under it, so a jingle or an outro becomes its own "speaker" at any threshold. The tool stays experimental and says so in its output.

Jobs. Client timeouts are short (Messages API ~60 s). Every long tool returns a job handle; state lives in ~/.ollos/jobs/<id>/job.json, written atomically, with a 5-second heartbeat. On restart, orphaned jobs become interrupted instead of hanging forever. Small work (< 8 s estimated) runs inline and returns directly.

Evaluation

npm run eval runs ollos against public videos and writes eval/RESULTS.md. Reference transcripts are YouTube captions, so on auto-captioned fixtures WER is an agreement rate between two recognisers, not an absolute error.

On the interview, half the "errors" are insertions: Whisper keeps the repetitions and fillers the auto-captions drop, and several reference words are caption mistakes ("Clash Orto" for Glassdoor, which ollos got right). Keyframes reduced the 10-minute screencast to 120 frames on 14 sheets and the talk to 51 frames on 6 sheets. Speed on that run was 0.2–0.35× real time on a loaded machine; the same model measures 1.7× in isolation. How to reproduce, and what each number means, is in eval/README.md.

Requirements and performance

Measured on the eval fixtures with the process memory sampled every 200 ms (scripts/probe-memory.mts). Machine: Intel i9-12900HX (16 cores), Windows 11, Node 20, while other processes used about 18% of the CPU.

Minimums that follow from this: 8 GB of RAM for the default model (4 GB is enough for model: "fast"), 4 GB of free disk for all models, any x64 or arm64 CPU (no GPU is used). Transcription speed scales with CPU cores and is the only stage that is compute-bound; a 4-core laptop should expect roughly 0.4× real time on the default model, so a one-hour meeting takes over two hours, or about 40 minutes with model: "fast".

Models stay loaded for the life of the server process; nothing is unloaded on idle yet. The peak above used to be ~10 GB: transformers.js' file cache streamed the 2.4 GB encoder weights into JavaScript buffers that nothing read, on top of ONNX Runtime's own memory-mapped copy. ollos now hands model paths to ONNX Runtime directly (see PathCache in src/core/models.ts). Set OLLOS_DEBUG_MEM=1 to stamp every job event with process memory and read them with ollos events <id>.

Memory limits

There is no memory cap setting today, and models are never unloaded while the server process lives. Once ollos_transcribe has run with the default model, the process keeps ~4.3 GB until it exits; add search and it keeps ~5.1 GB. This is a deliberate trade for speed (a cold load of the default model costs 50–60 s) and the honest state of 0.1.0.

What you can control now:

Node's own --max-old-space-size does not help: the weights live in ONNX Runtime's native memory, outside the V8 heap.

Planned, in order of value (see Roadmap):

1. Idle unloading — release a model after N minutes without a job (OLLOS_MODEL_IDLE_MIN), accepting the 50–60 s reload on the next call. Cheap to build; the pipelines already load lazily.
2. A memory guard — OLLOS_MAX_MEMORY_MB: before loading a model, compare its measured peak with the cap and the free RAM; refuse with a clear error, or downgrade to model: "fast" when OLLOS_MEMORY_FALLBACK=fast. The catalogue in src/core/models.ts already carries the sizes this needs.

Privacy & security

• Nothing is uploaded. Network is used only to download models once and to fetch a source URL you pass.
• URL fetching refuses private, loopback, link-local and IPv6-transition addresses (SSRF), on the first request and on every redirect hop, unless OLLOS_ALLOW_PRIVATE=1.
• Transcripts and on-screen text are returned inside <untrusted-content> — they are data, not instructions. The bundled skill says the same to the agent.
• Secret findings are masked in results, logs and events, and the OCR text returned next to them is redacted with the same masks — the tool that warns about a leak is not the leak.
• Recording other people requires their consent where you live.

Configuration

Site downloads (YouTube, Instagram, TikTok…) need yt-dlp on your PATH (or OLLOS_YTDLP) and are best-effort: platforms change often. Behind a corporate proxy that re-signs TLS, set OLLOS_YTDLP_ARGS="--no-check-certificates". Local files always work.

Troubleshooting

Known limits

• First run downloads 2.75 GB (accurate ASR) and needs ~4.3 GB of RAM while transcribing. model: "fast" uses a 280 MB model in ~1.9 GB of RAM and misreads technical terms. No memory cap yet; models are not unloaded on idle.
• Segment confidence is a heuristic (speech coverage, speaking rate, filters), not a model probability.
• Text around 8 px in the source video is at the edge of what OCR reads: detection of a secret that small depends on the exact frame, so ollos reads several frames around each hard cut. Below that, the UI-context signal still flags the situation ("API Key Created" is read reliably).
• ollos_diarize is experimental. Speech with background music (intros, jingles, outros) embeds far from the same voice on clean speech and comes out as an extra speaker regardless of the merge threshold (measured in eval/). Heavy crosstalk is unsolved. With Zoom per-participant tracks the result is exact.
• Site downloads depend on yt-dlp being installed; the standalone binary is not bundled yet. See examples/url-sources.md.
• Progress notifications and the MCP Tasks extension are not used yet; polling ollos_job is the contract for every client today.

Roadmap

• Memory: idle unloading of models (OLLOS_MODEL_IDLE_MIN) and a memory guard (OLLOS_MAX_MEMORY_MB with optional fallback to model: "fast"); see Memory limits
• MCP Tasks extension mode when clients ship it (the job engine is protocol-agnostic already), plus notifications/progress
• ollos_diarize: music-aware turn filtering (embed only turns the VAD marks as clean speech, or run a speech/music classifier first), a published DER from turn-level annotations, speaker naming persisted across recordings
• Bundled standalone yt-dlp so site downloads need no Python either
• Evaluation: DER with turn-level annotations, secret precision/recall on planted frames, retrieval metrics (hit rate, MRR, NDCG) for ollos_search

Credits

Built on transformers.js and the onnx-community model ports, tesseract.js, sharp and ffmpeg. Hallucination blocklists seeded from Vexa (Apache-2.0). Scene-aware keyframing was pioneered for agents by claude-real-video; ollos takes a different angle (review, speakers, Node) and owes it the contact-sheet idea.

License

Apache-2.0 — © 2026 Kelvin Biffi