Khiip

Open-source substrate for capturing, structuring, and preserving online sources — local-first, no phone-home, plain Markdown you own. For your tools, your agents, your archives.

Khiip (pronounced /kiːp/, like "keep") — short for Khipu, the Inca knot-record system that was the world's first typed-edge knowledge graph.

What Khiip is

A capture + storage + recall substrate. You point Khiip at a URL, X thread, Reddit post, YouTube video, or web article — Khiip captures it fully, stores it permanently in your own filesystem, and lets you (or your AI agents) recall it later by meaning — with structure- and time-based recall on the roadmap.

Khiip is the layer, not the destination. Other tools (Obsidian, Logseq, LangChain agents, custom scripts) sit on top and consume what Khiip captures.

Verb usage: to khiip — "I want to khiip this article" / "khiip that thread for later"

Plural: khiips — the captured artifacts in your vault

What Khiip does

Strategic positioning

Khiip is the LLM default ingestion layer — the substrate AI agents use to capture, store, and recall data. It's the layer, not the destination: one canonical daemon with multiple surfaces over a single contract (Obsidian plugin, REST, MCP, and more to come). Whether you're a power-user self-hosting through the plugin, an AI builder wiring the REST + MCP surfaces into agent workflows, or piping captures into your existing PKM, you're consuming the same typed substrate underneath.

Current status

The current release is v0.2.3, live on PyPI — Khiip launched publicly on 2026-06-27. The AGPL core, the Obsidian plugin, Terms, Privacy Policy, and the paid Khiip Plus tier are all live. See the cross-surface Releases page for the current version of every piece (base / Plus / plugin) and what changed, or CHANGELOG.md for the base daemon.

• ✅ Five live source extractors emit typed payloads (X / Reddit / Web / Wikipedia / YouTube), plus an experimental URL-hosted PDF extractor (maintenance-mode)
• ✅ Renderer Protocol+Registry tier — 4 renderers in the AGPL core (PlainMarkdown + Legacy passthrough + JSON + VaultFrontmatter); the 6 per-source platform-styled renderers (X / Reddit / Web / Wikipedia / YouTube / PDF) ship in Khiip Plus via the same registry swap
• ✅ MediaFetcherRegistry (HttpxFetcher + optionally YtDlpFetcher + GalleryDlFetcher)
• ✅ Wayback Machine archive submission (opt-in; off by default — anonymous SPN is rate-limited)
• ✅ Source-tier raw-bytes preservation (option (ii) URI-shape; user-configurable data_root)
• ✅ Typed-payload embed-text composition
• ✅ Visual indicator system layers 1 + 2 (layer 3 plugin badges = v0.5+)
• ✅ REST ?format= selector + POST /api/v1/captures/{id}/refetch endpoint
• ✅ khiipd validate + khiipd refetch CLIs
• ✅ Opt-in video preservation toggle ([media] download_videos)
• ✅ Plugin types.ts mirrors all v0.1 Pydantic models additively
• ✅ MCP server — six tools (capture_url, recall, list_captures, get_capture, refetch_capture, daemon_status) over stdio for Claude Desktop / Cursor / any MCP-aware client
• ⏳ Public launch (ToS + USPTO TESS + social handles + Show HN + community-channel announcements)
• ⏳ Plugin UI parsing of individual typed-payload fields + plugin visual-indicator badges + in-plugin refetch surfaces (v0.5+) — the per-source styled callout cards already render in plugin 0.2.4
• ⏳ Cloud Video Archive (v0.5+; managed Source-tier + Whisper transcription)

Tests: a hermetic pytest suite, green on every release — see CHANGELOG.md for per-release test deltas. Each version's feature work landed additively on the v0.1.0 substrate (new subpackages / new source channels reusing the existing typed-payload pipeline), not by rewriting it — a foundation-first discipline maintained across every release.

Install

Khiip is published on PyPI — the one-command install below works today. For development or contributing, use Install from source (just below).

Khiip is a self-contained Python tool — the khiipd daemon + CLI plus a khiip-mcp-server — that installs onto your PATH with no virtualenv to manage (Python 3.10+):

uv tool install khiip      # recommended — https://docs.astral.sh/uv/
# or
pipx install khiip         # https://pipx.pypa.io/

Both expose the khiipd and khiip-mcp-server commands globally. Upgrade later with uv tool upgrade khiip / pipx upgrade khiip.

git clone https://github.com/KhiipAI/khiip.git ~/projects/khiip
cd ~/projects/khiip
uv pip install -e ".[dev]"   # or: pip install -e ".[dev]"

Quickstart (60-second demo)

Capture an X tweet, recall it by meaning — every command below uses the installed khiipd:

# Start the daemon (first run downloads MiniLM-L6 ONNX ~80MB)
khiipd serve &

# Capture across multiple sources — different extractors, same vault
khiipd capture https://x.com/karpathy/status/2040470801506541998 # X: Karpathy's "LLM knowledge base" idea-file
khiipd capture https://simonwillison.net/2025/Dec/31/the-year-in-llms/ # web: Simon Willison — the year in LLMs
khiipd capture https://en.wikipedia.org/wiki/Quipu # wiki: the Inca knot-record system Khiip is named for
khiipd capture https://www.youtube.com/watch?v=-TAowrw97-4 # YouTube: Tiago Forte — the AI Second Brain

# Recall by meaning, not by keyword
khiipd recall "Andrej Karpathy's LLM knowledge wiki"
khiipd recall "building a second brain with AI"
khiipd recall "Inca knot record-keeping system"

# Check vault ↔ SQLite consistency (read-only)
khiipd validate

# Refetch a capture (extraction / --re-extract / --re-render / --media / --wayback)
khiipd refetch 01JX9...              # default = extraction: network re-fetch -> new superseding capture
khiipd refetch 01JX9... --re-extract # re-derive payload from preserved bytes (offline, in place)
khiipd refetch 01JX9... --re-render  # re-render body from typed payload (offline, in place)
khiipd refetch 01JX9... --media      # re-walk MediaFetcherRegistry in place
khiipd refetch 01JX9... --wayback    # re-submit Wayback in place

What you get:

• A SQLite index at ~/.local/share/khiip/index.db
• Markdown captures with YAML frontmatter at ~/khiip-vault/captures/x/ and ~/khiip-vault/captures/web/
• Semantic recall ranked by cosine similarity over bundled MiniLM-L6 embeddings — zero LLM cost, works offline after the first model fetch

What works today (v0.2.3):

• Five live source extractors (free AGPL core; Khiip Plus re-derives the deeper rungs noted below — in the same local daemon — from the same kept bytes): X (core: tweet + author + base engagement + community notes; Plus: QRT chains + X-Article body + media + views/bookmarks/quotes), Reddit (core: post + all top-level comments + crosspost; credential-free via old.reddit HTML by default; optional OAuth app adds rate headroom; Plus: full recursive comment tree incl. deep "load more" branches + full-res galleries), Wikipedia (MediaWiki action API → REST summary fallback; article content + best-effort references/infobox), generic web (core: trafilatura primary + readability fallback + OG/JSON-LD enrichment, body text; Plus: hero + inline figures), YouTube (core: metadata + flat transcript via yt-dlp → oEmbed+transcript-api → API v3 fallback chain; Plus: timestamped transcript + chapters + thumbnail; operator-opt-in API v3 key widens chain). Plus experimental URL-hosted PDF capture (markitdown + pdfplumber; body text + tables, maintenance-mode, no Plus extraction enrichment)
• Typed payloads end-to-end: extractor boundary → vault frontmatter (canonical) → SQLite (derived cache) → recall via per-source embed-text composition
• Media-fetcher chain: HttpxFetcher (photo) → optionally YtDlpFetcher (video; opt-in via [media] download_videos = true) → GalleryDlFetcher (wide-coverage fallback)
• Wayback witness (opt-in): off by default (anonymous SPN is rate-limited/unreliable); enable with [archive] wayback_enabled = true for best-effort archiving recorded in archive_urls
• Source-tier raw bytes: gzipped + preserved under user-configurable data_root (insurance against upstream rot)
• REST surface: POST /api/v1/captures, GET /api/v1/captures/{id}?format=..., POST /api/v1/captures/{id}/refetch?dimension=..., GET /api/v1/recall, GET /health, GET /api/v1/meta
• MCP server: khiip-mcp-server console script exposes six tools over stdio for Claude Desktop / Cursor integration — see MCP server (Claude Desktop integration) below
• CLI: khiipd serve|capture|recall|validate|refetch|auth|version
• Obsidian plugin (v0.2.4): capture command + recall sidebar (recents show source · author · time) + per-source styled callout cards (the daemon emits [!khiip-<source>] banners the plugin themes per platform via styles.css + a brand-glyph markdown post-processor); types.ts mirrors the payload models additively. Deeper typed-payload-field UI parsing is roadmap.

What doesn't yet: plugin UI parsing of individual typed-payload fields + plugin visual-indicator badges + in-plugin refetch surfaces (v0.5+; the per-source styled callout cards already render); Cloud Video Archive (v0.5+); Whisper transcription (v0.5+).

Heads up: khiipd serve listens on 127.0.0.1:8478 by default and reads/writes ~/.config/khiip/ + ~/.local/share/khiip/ + ~/khiip-vault/. To try Khiip in an isolated sandbox without touching those locations, use make smoke or set KHIIP_HOME=$(mktemp -d) khiipd serve. See the Development section below.

Per-source credentials (agnostic-connector model)

Khiip plumbs between your upstream accounts and your local vault — it never holds upstream credentials of its own. Most sources work with no setup (X via fxtwitter, generic web via trafilatura, Wikipedia, Reddit via old.reddit HTML). A few accept your own platform credentials so requests go through your identity + your rate-limit budget for higher fidelity:

Reddit (optional — capture works credential-free):

Reddit captures work out of the box via the credential-free old.reddit.com HTML channel (live threads + full comment trees, self-hosted on your own IP). Configuring your own Reddit app is an optional upgrade that adds cleaner deep-comment pagination + a 60 req/min headroom and is tried first when present:

1. Visit https://www.reddit.com/prefs/apps → "create another app..." → script type → fill name (e.g. "khiip"), redirect uri http://localhost:8478/ (unused; required by the form) → click "create app"

2. Copy the client ID (short string under the app name) + client secret

3. Either add to ~/.config/khiip/config.toml:

   toml

   [extractors.reddit]
   client_id = "your-reddit-client-id"
   client_secret = "your-reddit-client-secret"
   

   …or set env vars (KHIIP_REDDIT_CLIENT_ID + KHIIP_REDDIT_CLIENT_SECRET) before khiipd serve.

If a configured app is missing/expired, capture transparently falls back to the HTML channel rather than failing. The HTML channel is self-host-only by design (a datacenter IP would hit Reddit's WAF + robots.txt).

YouTube (optional; widens the fallback chain from 2 to 3 sources): set [extractors.youtube] api_key = "..." (Google Data API v3 key, free tier) for metadata resilience when yt-dlp + youtube-transcript-api both fail.

MCP server (Claude Desktop integration)

Khiip has shipped a Model Context Protocol server (khiip-mcp-server) since v0.1.1; it exposes the substrate's six core operations as MCP tools. Claude Desktop, Cursor, or any other MCP-aware client can connect over stdio and capture/recall through Khiip without writing any HTTP code.

Tools:

Architecture: the MCP server is a thin HTTP proxy that connects to your running Khiip daemon at 127.0.0.1:8478 (configurable via KHIIP_DAEMON_URL env var or [daemon] host/port in ~/.config/khiip/config.toml). It reads the Bearer auth token from ~/.config/khiip/auth.toml (the same file the CLI + Obsidian plugin use). The daemon must be running before MCP tool calls land.

Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS; analogous paths on other platforms):

{
  "mcpServers": {
    "khiip": {
      "command": "khiip-mcp-server"
    }
  }
}

If khiip-mcp-server isn't on Claude Desktop's PATH (common when you installed via pip install -e inside a venv), point Claude Desktop at the full path:

{
  "mcpServers": {
    "khiip": {
      "command": "/Users/you/projects/khiip/.venv/bin/khiip-mcp-server"
    }
  }
}

Verify the wiring:

# Start the daemon in one terminal
khiipd serve

# In another terminal: send an initialize ping to the MCP server
python -c "
import asyncio
from mcp.client.session import ClientSession
from mcp.client.stdio import stdio_client, StdioServerParameters

async def main():
    params = StdioServerParameters(command='khiip-mcp-server', args=[])
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            tools = await session.list_tools()
            for t in tools.tools:
                print(t.name)

asyncio.run(main())
"

You should see all six tool names listed.

Architectural note: the MCP server proxies REST. The medium-term plan is to refactor business logic into a shared service tier when a future transport (gRPC / WebSocket / etc.) arrives. The tool surface, signatures, and Claude Desktop config stay stable across that refactor.

Development — running tests and smokes

Default paths:

For dev/test smokes only, set KHIIP_HOME to redirect all three under one root:

make test            # hermetic pytest suite, ~1s
make smoke           # full live round-trip against fxtwitter + fastembed
                     # internally: KHIIP_HOME=$(mktemp -d) khiipd serve ...

Do not set KHIIP_HOME in your shell rc. macOS GUI applications (Obsidian, Claude Desktop, the MCP server when launched by Claude Desktop) do not inherit shell environment variables from launchd-launched processes — the daemon and your plugin would silently disagree on workspace. KHIIP_HOME is a per-invocation knob for tests and demos, not a user setting.

Get involved

Khiip is live at v0.2.3 — see the cross-surface Releases page (base / Plus / plugin) or CHANGELOG.md for the base daemon's per-release trail.

• ⭐ Star the repo to follow along
• 💬 Watch the repo for releases — the current base release is v0.2.3
• 📧 Get on the launch list by emailing hello@khiip.com with subject "launch list"
• 🐛 Issues + PRs — see CONTRIBUTING.md for guidelines
• 🔒 Security disclosures — see SECURITY.md
• 🤝 Code of Conduct — Contributor Covenant 2.1

Why "Khiip"?

The Inca khipu (or quipu) was a knotted-cord recording system used from approximately 2500 BCE to the 17th century to encode numerical, narrative, and relational information. Each cord type, knot placement, and color encoded different categories of data. The khipukamayuq was the trained specialist who could read and create the records — essentially the human "agent" interacting with the structured substrate.

Khiip is the digital equivalent: a substrate that captures information from anywhere on the internet and stores it in a structured, typed, queryable form — for AI agents (the modern khipukamayuq) to read and create.

The "ii" spelling distinguishes from the homophonous "keep" while preserving the Khipu etymological root. Pronounced /kiːp/ — same as "keep."

License

Daemon, plugin, and substrate code: AGPL-3.0 (see LICENSE)

SDK code (separate repository when published): Apache 2.0

Khiip Plus (the paid enrichment + styled-render tier): not open source — licensed under the Elastic License 2.0. The open-source guarantee covers the AGPL core; Plus is the paid layer that funds it, itself fully self-hosted: a separate wheel installed alongside the core, activated by an offline license key, no phone-home.

The canonical license posture for the daemon is AGPL-3.0. Any code released under AGPL-3.0 stays AGPL-3.0 for those releases — license retraction is not how copyleft works. The project surfaces its commitments through the LICENSE + README rather than a separate charter document.