yt-mem-ai — a local YouTube memory for your AI assistant

Give Claude, Codex, Cursor, or any MCP host the ability to watch YouTube for you: transcribe videos, remember them, follow your subscriptions, and turn all of it into summaries, timestamped highlights, Q&A, digests, and video reels. Everything runs on your machine — no cloud service, no API key.

One command, two questions — here it's wiring the skills into Codex.

Example — "make a presentation from this video" (Andrej Karpathy: From Vibe Coding to Agentic Engineering, Sequoia, 30 min) → 13 slides, PDF, every quote timestamped from the transcript. Ingest to deck in one request.

Table of Contents

• Features
• How to install
• Getting Started
• Usage
• Configuration
• Under the hood

Features

• 🎧 Transcribes any video — YouTube captions when they exist (fast, any language), offline Whisper when they don't.
• 🧠 Remembers what you watched — every transcript is stored and indexed locally, so your library stays searchable forever. Nothing leaves your machine.
• 🔎 Finds the moment — ask "what did that video say about X" and get the answer with a timestamp you can jump to.
• 📡 Follows your subscriptions — picks up new uploads and turns the day into one digest.
• ✍️ Your assistant does the writing — summaries, highlights, Q&A, slide decks, all in the video's own language, using the model you already pay for.
• ❤️ Learns your taste — like or dislike videos and get recommendations from your own library.
• 🎬 Makes media too — clickable highlight docs, still frames, and rendered supercut reels.
• 🔌 Works with your tools — Claude Code, Claude Desktop, Codex, Cursor, Antigravity, OpenClaw, Hermes: skills or MCP, your pick.

How to install

1. Connect your assistant ⭐

curl -LsSf https://raw.githubusercontent.com/dasein108/yt-mem-ai/main/install.sh | sh

An interactive wizard opens. Pick what you want, tick your apps, press enter:

step 1/2 — what (pick one)     step 2/2 — where (tick any)
> Plugin  skills + CLI         [x] Claude Code    [ ] Claude Desktop
  MCP     typed tools          [x] Codex          [ ] Cursor
                               [ ] Antigravity    [ ] OpenClaw   [ ] Hermes

Plugin teaches your assistant to act on plain requests — "summarize this video". MCP gives it a set of tools instead. Not sure? Start with Plugin; you can run the wizard again for the other.

It installs everything it needs, ticks what you already have, and removes anything you untick (it shows a plan and asks first). Then restart the app and try: summarize 'https://youtu.be/…'.

Already know what you want? Skip the questions:

curl -LsSf https://raw.githubusercontent.com/dasein108/yt-mem-ai/main/install.sh \
  | sh -s -- --plugin --claude-code --codex
curl -LsSf https://raw.githubusercontent.com/dasein108/yt-mem-ai/main/install.sh \
  | sh -s -- --mcp --claude-desktop --cursor

Hosts: --claude-code --claude-desktop --codex --cursor --antigravity --openclaw --hermes, or --all. Full flag list and uninstall notes: integrations/README.md. Rather have an agent do it? Paste integrations/PROMPT.md into any assistant.

2. MCP by hand — one config entry, self-installing

No prior install needed: uvx fetches the package the first time the host launches the server, and keeps it cached afterwards. Drop this into your host's MCP config:

{
  "mcpServers": {
    "yt-mem-ai": {
      "command": "uvx",
      "args": ["--from", "yt-mem-ai[mcp]", "yt-ai-mcp"]
    }
  }
}

That's the whole setup — no paths, no env block. Settings live in ~/.yt-mem-ai/config.env and the agent can write them itself with the config_set tool (or you with yt-ai config set).

# ~/.codex/config.toml
[mcp_servers.yt-mem-ai]
command = "uvx"
args = ["--from", "yt-mem-ai[mcp]", "yt-ai-mcp"]

# ~/.hermes/config.yaml
mcp_servers:
  yt-mem-ai:
    command: "uvx"
    args: ["--from", "yt-mem-ai[mcp]", "yt-ai-mcp"]
    enabled: true

Restart the app and the tools show up — see Usage for what they do.

Nothing appeared, or the host timed out? The first launch downloads dependencies and can outlast the host's startup check. Run uvx --from 'yt-mem-ai[mcp]' yt-ai-mcp --help once, then reopen the app. If the host still can't start it, give it absolute paths — uv tool install 'yt-mem-ai[mcp]' and use which yt-ai-mcp as command with "args": [] (GUI apps often don't see ~/.local/bin on their PATH).

3. Claude Desktop — skills (in the app)

Desktop stores plugins on your Claude account, not on disk, so nothing can install them for you. It takes a minute in the app:

Customize (left sidebar) → Plugins → Personal plugins → + → Add marketplace → Add from a repository → https://github.com/dasein108/yt-mem-ai → Add → Install yt-mem-ai

Then ask: summarize 'https://youtu.be/…'. Uninstall the same way. The same plugin also works on claude.ai and Cowork. Prefer tools over skills? The MCP setup above works for Desktop too — and that one can be scripted.

# from a checkout
cp -R skills/yt skills/yt-agent ~/.codex/skills/

# without a checkout
for s in yt yt-agent; do
  mkdir -p ~/.codex/skills/$s
  curl -LsSf "https://raw.githubusercontent.com/dasein108/yt-mem-ai/main/skills/$s/SKILL.md" \
    -o ~/.codex/skills/$s/SKILL.md
done

4. The CLI on its own

The skills drive it, but it's a perfectly good standalone tool:

uvx yt-mem-ai --help          # zero-install run
uv tool install yt-mem-ai     # or install the persistent `yt-ai` command

Needs Python 3.11+ and uv; ffmpeg only for supercut / frame.

The desktop UI lives in a separate repo: yt-mem-ai-desktop — it depends on this package and runs its own local REST API.

Getting Started

Installed and host restarted? You're ready. Just talk to your assistant — the skills (or MCP prompts + analyze_video) do the ingesting for you:

"Summarize https://youtu.be/dQw4w9WgXcQ" → ingests the video (captions → whisper), then writes an executive summary plus key points, in the video's own language.

"Give me the highlights of that video with timestamps" → 3–8 deep-linked moments (watch?v=…&t=123s) anchored by semantic search.

"What did I watch about retrieval-augmented generation?" → searches every transcript in your library and quotes the moments.

"Process my subscriptions into today's digest" → discovers new uploads, ingests them, writes digests/<DATE>.md.

Prefer the terminal? The same first run:

yt-ai fetch 'https://www.youtube.com/watch?v=VIDEO_ID'   # ingest one video
yt-ai search "what was said about embeddings"            # search your library
yt-ai status                                             # what's in the store

Everything lands in ~/.yt-mem-ai/ (library, logs, downloads).

The first run is slow — that's expected. Installing pulls the ML stack (torch, LanceDB, sentence-transformers ≈ 1 GB on disk), and your first fetch or search downloads the embedding model on top of that. If a video has no captions, the Whisper model (small, ≈ 460 MB) downloads too — the captions path never needs it. All of it is cached, so it happens once, not per video. On a GUI host the first MCP launch can outlast the app's startup check for the same reason: run uvx --from 'yt-mem-ai[mcp]' yt-ai-mcp --help once in a terminal to warm the cache, then reopen the app.

Daily routine

yt-ai discover               # what's new in your subscriptions
yt-ai fetch-pending          # transcribe today's batch

then in your assistant: "process subscriptions" → per-video summaries and digests/<DATE>.md, and optionally yt-ai compile --out compilations/$(date +%F).md for the day's highlights as clickable links.

Usage

Talking to your assistant (skills & prompts)

Two skills ship with the plugin. MCP hosts get the same playbooks as prompts (yt_summarize, yt_highlights, yt_qa, yt_presentation, yt_digest, yt_review, yt_group), so nothing is lost without skill support.

Skills never touch the database directly — they call the CLI, so the same workflow runs on any host. Install or paste them by hand: skills/README.md.

MCP tools

The yt-ai-mcp server exposes the whole engine as typed tools. analyze_video is the one-step entry point: it ingests and returns the transcript so the model can write the summary itself.

Server details, prompt list, and raw config: integrations/mcp/README.md.

CLI commands

yt-ai fetch <url>            # download + transcribe + embed + store one video
yt-ai fetch <url> --captions-only  # captions only: no audio download / no whisper (fails if none)
yt-ai transcript <url>       # same pipeline
yt-ai discover               # new subscription uploads (--after/--deep/--min-duration/--json); incremental by default
yt-ai fetch-pending          # batch-fetch pending 'discovered' videos (since --since, default today; --limit)
yt-ai list                   # list stored videos (--status/--since/--json)
yt-ai show <video_id>        # metadata + transcript (--json)
yt-ai status                 # counts by status
yt-ai search "<query>"       # semantic search (--hybrid/--fts/--vector, -k N)
yt-ai save-summary <id> "<summary>" --highlights '<json>' --qa '<json>'  # persist a summary (used by skills)
yt-ai like <video_id>        # mark liked (feeds recommendations)
yt-ai dislike <video_id>     # mark disliked
yt-ai recommend              # rank your unrated fetched videos by taste (--limit/--json)
yt-ai compile                # deep-linked highlights doc, budget-bounded (--since/--max-minutes/--json/--out)
yt-ai supercut               # video reel of highlights, re-downloaded + labeled (--since/--max-minutes/--out/--keep-clips)
yt-ai frame <video_id> --at <ts>  # still frame at a timestamp (seconds or H:M:S) → frames/<id>_<s>s.png
yt-ai reembed                # re-embed all chunks with the current YT_EMBEDDING_* config
yt-ai channel-list <url>     # list a channel's recent uploads (--limit/--from/--to/--json); enumerate only
yt-ai config list            # get/set any .env setting: config get/set/unset/path (reconfigure from CLI or chat)

Configuration

Nothing is required to start — defaults put the store, logs, and downloads under ~/.yt-mem-ai/ and use a local embedding model. Tune it when you need to:

Configure from the CLI or an agent — instead of editing .env by hand, use yt-ai config (or the MCP config_* tools, so an agent can reconfigure itself from chat):

yt-ai config list                  # every setting, value, and source
yt-ai config set WEBSHARE_PROXY_USERNAME <user>
yt-ai config set WEBSHARE_PROXY_PASSWORD <pass>
yt-ai config set YT_EMBEDDING_MODEL paraphrase-multilingual-MiniLM-L12-v2
yt-ai config get OPENAI_API_KEY    # secrets masked (--reveal to show)

set writes the global config file (~/.yt-mem-ai/config.env) by default so the MCP server picks it up regardless of its working directory; --project writes ./.env. Precedence: process env > project .env > global config file, and config list shows which one each value comes from.

Under the hood

Everything below is optional reading — internals, tuning, and developer notes.

How it works

yt-dlp → captions (or Whisper) → chunks → embeddings → LanceDB
                                                          ↓
                     your assistant reads + writes summaries back

The CLI does the heavy IO and owns the store; the skills and MCP tools are thin callers. A video moves through discovered → downloaded → transcribed → summarized; live streams get a terminal stream status and are skipped by batch ingestion (transcribe one on demand with yt-ai fetch <url>). Storage is an embedded LanceDB (videos, channels, transcripts, chunks, summaries, feedback), with per-chunk vectors plus a full-text index — that's what makes search hybrid.

Command details

Rate & recommend — like/dislike videos you've fetched, then yt-ai recommend ranks the rest by similarity to what you liked (minus what you disliked), using their transcript embeddings. Before you've liked anything it falls back to most-recently-published.

discover is incremental — it pulls the newest feed entries in one flat call (capped by YT_DISCOVER_FEED_LIMIT), stamps each with an approximate timestamp, and keeps only those newer than the last run's high-water mark minus a 1h overlap (YT_DISCOVER_OVERLAP_S), so hour-rounded dates never miss a boundary video. Already-processed videos are filtered out. --after YYYY-MM-DD overrides the cutoff.

compile vs supercut — compile renders the day's highlights as markdown deep links (watch?v=ID&t=<start>s), newest-video-first and bounded by --max-minutes (default 20); it's instant and prints to stdout unless you pass --out. supercut re-downloads each highlight's section at 720p, burns a label onto it, and concatenates everything into supercuts/<date>.mp4 plus a .refs.md sidecar listing sources (and any clips skipped after a failure) — much slower, needs network + ffmpeg, but shareable.

Embeddings, proxy, and other tuning

Embeddings: YT_EMBEDDING_BACKEND=local|openai. Local uses sentence-transformers (YT_EMBEDDING_MODEL, default all-MiniLM-L6-v2) — for non-English libraries set paraphrase-multilingual-MiniLM-L12-v2 (384-d, 50+ languages) so semantic search works cross-language. openai uses text-embedding-3-small|large (needs OPENAI_API_KEY). After changing the model, run yt-ai reembed to migrate the existing library (re-embeds all chunks; no re-fetch).

Proxy / VLESS: YT_USE_WEBSHARE defaults off. If you already run a system-level proxy/VPN (VLESS/Xray etc.), leave it off — traffic rides that tunnel. Stacking the Webshare proxy on top breaks the authenticated subscription feed (its CONNECT tunnel returns 405). Only set YT_USE_WEBSHARE=true if you have no other proxy and YouTube rate-limits your raw IP. Discover tuning: YT_DISCOVER_FEED_LIMIT (newest-N cap, default 60), YT_DISCOVER_OVERLAP_S (incremental overlap, default 3600), YT_DISCOVER_TIMEOUT_S.

Working from a checkout? uv sync --extra dev, then cp .env.example .env if you'd rather keep settings project-local than in ~/.yt-mem-ai/config.env.

Use as a Python package

yt-mem-ai is a normal library — the CLI is a thin Typer shell over run_* cores you can call directly. Everything is local: no server, no API key (unless you pick the openai embedding backend).

pip install yt-mem-ai     # or: uv add yt-mem-ai

from dataclasses import replace
from pathlib import Path

from yt_mem_ai.config import load_config
from yt_mem_ai.cli import open_store, run_fetch, run_search, run_list, run_save_summary
from yt_mem_ai.store import db as store

# Config comes from ~/.yt-mem-ai/config.env < ./.env < process env.
# Override any field in code (Config is a frozen dataclass):
cfg = replace(load_config(), store_path=Path("~/.yt-mem-ai/lance").expanduser())

db = open_store(cfg)          # opens LanceDB + creates tables/indexes once
                              # pass db=... to every run_* call to reuse it

# 1. Ingest: download → transcribe (captions → whisper) → chunk → embed → store
video_id = run_fetch("https://www.youtube.com/watch?v=dQw4w9WgXcQ", cfg, db=db)
# captions only (no audio download, no whisper):
# video_id = run_fetch(url, cfg, db=db, captions_only=True)

# 2. Read what was stored
video = store.get_video(db, video_id)
text = store.get_transcript_text(db, video_id)
print(video.title, video.channel, video.duration_s, len(text or ""))

for c in store.list_chunks(db, video_id)[:3]:
    print(f"[{c['start_s']:.0f}s] {c['text'][:80]}")

# 3. Semantic search across the whole library (hybrid | vector | fts)
for hit in run_search(cfg, "retrieval augmented generation", mode="hybrid", k=5, db=db):
    print(hit["video_id"], hit["start_s"], hit["text"][:100])

# 4. Bring your own LLM: summarize the transcript however you like, then persist
summary_md = my_llm(text)                       # any model / provider
run_save_summary(
    cfg, video_id, summary_md,
    highlights_json='[{"t": 42, "text": "key moment"}]',
    qa_json='[{"q": "What is it about?", "a": "..."}]',
    db=db,
)
print(store.get_summary(db, video_id))

# 5. Library queries
for v in run_list(cfg, status="transcribed", since="2026-01-01", db=db):
    print(v.video_id, v.published_at, v.title)

Other cores, same shape (run_x(cfg, ..., db=db)): run_discover, run_fetch_pending, run_channel_list, run_recommend, run_feedback, run_compile, run_supercut, run_frame, run_reembed. Lower-level pieces are importable too — yt_mem_ai.download.download, yt_mem_ai.transcript.get_transcript, yt_mem_ai.store.embeddings.build_embedder / chunk_segments, yt_mem_ai.store.db (LanceDB CRUD + search_chunks).

The store is plain LanceDB, so you can also open it directly:

import lancedb
from pathlib import Path
tbl = lancedb.connect(Path("~/.yt-mem-ai/lance").expanduser()).open_table("chunks")
df = tbl.to_pandas()      # video_id, start_s, end_s, text, vector

Logging

The CLI writes structured JSON events to logs/common.jsonl (via obs.log_event/blog) — one object per line, {ts, source, level, event, msg, ...ctx}. Override the path with YT_LOG_FILE; it's gitignored. Inspect with jq:

jq -c 'select(.level=="error")' logs/common.jsonl   # every error
tail -f logs/common.jsonl | jq -c '{ts,event,msg}'  # live tail, compact

Tests

uv run pytest -q                       # offline unit tests (fake embedder)
YT_RUN_INTEGRATION=1 uv run pytest -q  # + real sentence-transformers integration

Releasing (maintainers)

Version comes from the git tag (hatch-vcs). Tag, build, and publish:

git tag -a vX.Y.Z -m "vX.Y.Z" && git push origin vX.Y.Z
uv build                                # → dist/ (sdist + wheel)
sh scripts/publish.sh dist/yt_mem_ai-X.Y.Z*   # uploads to PyPI

scripts/publish.sh loads UV_PUBLISH_TOKEN from .env (gitignored) on demand, so you don't export it each time — add UV_PUBLISH_TOKEN=pypi-… to .env once (see .env.example). Equivalently: set -a; . ./.env; set +a; uv publish dist/*. Rotate the token on PyPI if it's ever exposed.