io.github.sunholo-data/parse

Official31 tools

by sunholo-data · HTML

AILANG Parse

Deterministic DOCX/PPTX/XLSX/PDF parser: track changes, comments, headers, footers, merged cells.

Parses DOCX, PPTX, XLSX, and PDF files with support for tracking changes, comments, and formatting details.

Topics

documentdocxmarkdownparserparsing-librarypptxxlsx

Captured live from the server via tools/list.

agentCard

A2A Agent Card for agent-to-agent discovery. Returns a hand-crafted Agent Card with AILANG Parse-specific skills, supported formats, and capability descriptions. @raw: return raw JSON (no {result:..., elapsed_ms:N} envelope) for A2A compliance. Switch to @nowrap once available (lighter than @raw). If using --a2a flag, the built-in route takes precedence (collision guard skips this).

Parameters (1)

_stringrequired

apiSamples

Sample files inventory — delegates to the upstream package. Package @route annotations don't auto-register with serve-api, so we provide a local wrapper here.

Parameters (1)

_stringrequired

apiTools

Tool definitions for Claude, OpenAI, MCP, and A2A agent frameworks. Delegates to the upstream package.

Parameters (1)

_stringrequired

capabilities

capabilities(() -> string ! {Env}) [pure]

Parameters (1)

_stringrequired

deviceAuthApprove

Approve a device authorization request. Called by the browser approval page after user signs in and confirms. Verifies Firebase ID token from Authorization header to ensure the caller is a real authenticated user. Extracts uid from the verified token. Dev mode: ALLOW_SELF_APPROVE=true skips Firebase verification (for testing).

Parameters (1)

reqobjectrequired

deviceAuthInspect

Look up a pending device code by user_code and return its provenance. No authentication required — possession of the user_code is the proof. The dashboard /approve.html page calls this BEFORE showing the approve UI so the user can see where the request came from (IP, User-Agent, Referer, age) and decide whether to trust it. Returns 404 if the user_code does not match any pending device_codes doc. Already-approved or expired codes return as INPUT_NOT_FOUND too — there is nothing to inspect after the fact.

Parameters (1)

userCodestringrequired

deviceAuthPoll

Poll for device authorization approval. Returns AUTHORIZATION_PENDING if user hasn't approved yet. Returns DEVICE_CODE_EXPIRED if the code has timed out. Returns approved status with API key on success.

Parameters (1)

deviceCodestringrequired

deviceAuthRequest

Request a device authorization code. Returns device_code, user_code, and verification URL. The agent should display the verification_url to the user.

Parameters (1)

reqobjectrequired

editDocument

Parse a document, apply JSON edit deltas, and return the modified blocks as JSON (same format as POST /api/v1/parse with outputFormat=blocks). filepath: uploaded file path (multipart upload). deltas: JSON array of edit operations — see edit_apply.ail for format. Empty array or "" → round-trip (parse + return unchanged blocks). apiKey: dp_ API key. Response: modified blocks JSON. Use the AILANG SDK or CLI to generate a file from the returned blocks (e.g. ailang run ... --convert output.docx). Only deterministic office formats are supported (docx, pptx, xlsx, odt, odp, ods). AI-required formats (pdf, image, audio, video) are rejected.

Parameters (3)

apiKeystringrequired
deltasstringrequired
filepathstringrequired

estimate

estimate((string, string) -> string ! {Clock, FS}) [pure]

Parameters (2)

filepathstringrequired
outputFormatstringrequired

formats

List all supported document formats for parsing and generation. Returns: parse formats (13), generate formats (9), output formats (blocks/markdown/html/a2ui), and which formats require AI (PDF, images).

Parameters (1)

_stringrequired

getKeyUsage

Get usage stats for a user's API key. Accepts Firebase JWT or apiKey. Verifies that the requested keyId belongs to the authenticated user.

Parameters (1)

reqobjectrequired

getUploadUrl

Request a pre-authenticated GCS upload URL for direct file upload. Business tier only. The returned URL allows the client to PUT file content directly to GCS, bypassing the 32MB Cloud Run request limit. After upload, pass the gcs_ref to POST /api/v1/parse.

Parameters (3)

apiKeystringrequired
filenamestringrequired
mimeTypestringrequired

health

Health check for the AILANG Parse API. Returns service status, version, AILANG commit hash, supported format counts, and billing catalog status. `billing_catalog_loaded` is FALSE when the BILLING_PLAN_CATALOG env var is unset or parses to an empty list — in that mode every authenticated request silently falls back to the safety-net "fallback" plan (limit=1) and is rejected as over-quota. release.sh asserts `billing_catalog_loaded == true` after every promotion to catch this regressing.

Parameters (1)

_stringrequired

listApiKeys

List API keys for a user. Accepts either: - Authorization: Bearer <firebase_id_token> (dashboard path) - {apiKey: "..."} or {args: ["dp_..."]} in body (CLI/SDK path) The resolved userId filters the Firestore query server-side.

Parameters (1)

reqobjectrequired

mcpAccount

View account info, pricing, entitlements, or list keys. Actions: "status" (default) → tier, quota, usage from /me/entitlements "pricing" → public pricing tiers (no auth required) "keys" → list user's API keys with per-key usage "usage" → alias for "keys" (per-key usage is shown there)

Parameters (2)

actionstringrequired
apiKeystringrequired

mcpAuth

Start device authorization to get an API key. Returns device_code, user_code, and verification URL. The agent should display the verification URL to the user, who signs in and approves the code. Then call mcpAuthPoll with the device_code. MCP wrappers don't have HTTP request headers, so provenance is empty here.

Parameters (1)

labelstringrequired

mcpAuthPoll

Poll for device authorization completion. Returns "pending" (keep polling every 5s), "approved" (with api_key and tier), or "expired" (start over with mcpAuth).

Parameters (1)

deviceCodestringrequired

mcpConvert

Convert is not supported on the hosted server (no persistent local filesystem to write the output file to). Use the local stdio SDK (@ailang/parse) for local conversions, where the user has filesystem access.

Parameters (4)

apiKeystringrequired
inputstringrequired
outputFormatstringrequired
outputPathstringrequired

mcpEstimate

Estimate cost and latency for parsing a document. Accepts a file path or sample_id. No auth required.

Parameters (2)

filepathstringrequired
outputFormatstringrequired

mcpFormats

List supported formats, samples, and service capabilities. Pure JSON, no auth required. Delegates to package implementation. Single source of truth lives in pkg/sunholo/ailang_parse/services/mcp/tools.

Parameters (1)

_stringrequired

mcpParse

Parse a document. Accepts a file path or sample_id (e.g. "sample_docx_formatting"). The hosted server requires a valid dp_ API key — get one via mcpAuth. Output formats: blocks (default), markdown, html, a2ui. requestId is reserved for future replay support.

Parameters (4)

apiKeystringrequired
filepathstringrequired
outputFormatstringrequired
requestIdstringrequired

myEntitlements

Get the authenticated user's billing entitlements, usage, and plan details. Returns: plan name, monthly request limit, requests used, remaining requests, upgrade/manage URLs. Requires a valid AILANG Parse API key (dp_ prefix).

Parameters (1)

apiKeystringrequired

parseFileSecure

Parse a document. Requires a valid API key. Validates the key, checks entitlement quotas, logs for replay. filepath: file path OR sample_id (e.g. "sample_docx_formatting" → resolved via /api/v1/samples). outputFormat: "blocks", "markdown", "html", or "a2ui". editable: "true" to emit editable A2UI component variants with block_index props; only meaningful when outputFormat="a2ui". gcsRef: optional gs:// URI for Business tier large file uploads (>32MB). When provided, the file is downloaded from GCS via our service account. Business tier only — Free/Pro users get TIER_UPGRADE_REQUIRED error. sourceUrl: optional https:// URL (e.g., a signed GCS URL or any public file). When provided, the file is fetched over HTTPS by docparse and parsed. Available on all tiers; tier dictates the max fetched-file size. Cannot be combined with gcsRef or filepath — sourceUrl wins. pdfBackend: optional PDF extraction backend override. "" — use server default (DOCPARSE_PDF_BACKEND env var, default "pdftotext"). "pdftotext" — deterministic text extraction via poppler. Fast, no AI cost. "docling" — IBM Docling layout analysis. No AI cost. "liteparse" — run-llama LiteParse. No AI cost. "ai" — Gemini multimodal via Vertex AI. Required for scanned/image-only PDFs. Explicit non-"ai" selection returns an error on failure (no silent AI fallback). @nowrap: raw JSON (no envelope), _headers extracted as HTTP response headers.

Parameters (7)

apiKeystringrequired
editablestringrequired
filepathstringrequired
gcsRefstringrequired
outputFormatstringrequired
pdfBackendstringrequired
sourceUrlstringrequired

partitionGeneral

Unstructured API-compatible endpoint (drop-in replacement for Unstructured.io). Returns elements in Unstructured JSON format (Title, NarrativeText, Table, ListItem, etc.). Accepts file upload (multipart/form-data) or JSON body with filepath/sample_id. API key: via unstructured-api-key header (Unstructured convention) or apiKey form field. strategy parameter: "auto" (default), "hi_res", "fast", "ocr_only". Uses _headers for header access while keeping @route multipart support.

Parameters (4)

_headersobjectrequired
apiKeystringrequired
filepathstringrequired
strategystringrequired

pricing

pricing(() -> string) [pure]

Parameters (1)

_stringrequired

requestHistory

List recent parse requests for a user. Returns up to 50 entries. Accepts Firebase ID token (dashboard) OR dp_ API key (programmatic). Dashboard sends Authorization: Bearer <firebase_token> with {args: [uid]}. API clients send {args: [apiKey]}. Uses Firestore structured query to filter by user_id server-side and order by timestamp descending. Only reads matching docs (not full scan).

Parameters (1)

reqobjectrequired

requestReplay

Retrieve a stored request/response pair for replay. Requires authentication: Firebase JWT or dp_ API key. The request must belong to the authenticated user (user_id match). Accepts optional outputFormat (blocks/markdown/html/a2ui) to re-render the stored blocks server-side using the ailang_parse pipeline.

Parameters (1)

reqobjectrequired

revokeApiKey

Revoke an API key by keyId. Authenticates via either Firebase JWT or apiKey.

Parameters (1)

reqobjectrequired

rotateApiKey

Rotate an API key: generate new key, revoke old one, preserve tier + usage.

Parameters (1)

reqobjectrequired

submit_feedback

Anonymous bug report / feature request / docs gap, queued for human review. Default routing: `public-feedback` inbox (general AILANG). Pass `package="vendor/name"` (e.g. "sunholo/auth") to route to that package's `pkg:vendor/name` inbox where its autonomous agent watches. Categories: bug, feature, docs, limitation. Body limit 10KB, snippet limit 4KB. Optional contact field for follow-up; opaque to the server. Set `auto_dispatch=true` to authorize the package agent to act on your submission immediately (default false — files for human triage; pkg-feedback agent template lands in a separate sprint).

Parameters (8)

ailang_versionstringrequired
The reporter's CLI version (free-form, used for triage)
auto_dispatchboolean
Authorize the receiving package's autonomous agent to act on this submission. Default false (files for human triage). Tagged on the Pub/Sub notification as category=auto:<original> for coordinator filtering.
bodystringrequired
Full description (≤10 KB)
categorystringrequired
bug | feature | docs | limitation
contactstring
Optional follow-up address (free-form, opaque to the server)
packagestring
Optional vendor/name (e.g. "sunholo/auth") to route to that package's pkg:vendor/name inbox. Empty = general AILANG feedback.
snippetstring
Optional code/error snippet (≤4 KB)
titlestringrequired
Short title for the report

README.md6 KBView on GitHub

AILANG Parse

Universal document parsing in AILANG. Extracts structured content from DOCX, PPTX, XLSX, PDF, and image files into JSON and markdown.

Office formats (DOCX, PPTX, XLSX) use deterministic XML parsing — no AI, no cloud, instant results. PDFs default to the deterministic pdftotext backend (poppler) — also no AI, no cloud — with docling and liteparse as local alternatives and pluggable AI (Gemini, Claude, local Ollama) for scanned/image-only pages via --pdf-backend ai. Images delegate to whatever AI model you plug in. AILANG Parse is AI-agnostic: swap --pdf-backend/--ai to change the backend, zero code changes.

Install

Requires AILANG CLI.

bash

# Clone and symlink
git clone https://github.com/sunholo-data/ailang-parse.git
ln -s "$(pwd)/ailang-parse/bin/docparse" /usr/local/bin/docparse

SDKs

Use AILANG Parse from your language of choice:

bash

pip install ailang-parse          # Python
npm install @ailang/parse         # JavaScript/TypeScript
go get github.com/sunholo-data/ailang-parse-go  # Go

Quick Start

bash

# Office documents (deterministic, no AI needed)
docparse report.docx
docparse slides.pptx
docparse spreadsheet.xlsx

# PDF (deterministic pdftotext by default — no AI); images (AI auto-enabled)
docparse document.pdf
docparse photo.png

# Options
docparse report.docx describe        # AI image descriptions
docparse report.docx summarize       # AI document summary
docparse contract.pdf                # PDF: deterministic pdftotext (default)
docparse scan.pdf --pdf-backend ai --ai gemini-2.5-flash  # Scanned PDF needs AI

# Format conversion
docparse report.docx --convert output.html
docparse data.csv --convert report.docx
docparse notes.md --convert slides.pptx

# AI document generation
ailang run --entry main --caps IO,FS,Env,AI --ai gemini-2.5-flash \
  docparse/main.ail --generate report.docx --prompt "Q1 sales report with tables"

Output

Every run produces:

docparse/data/output.json — Structured JSON with typed blocks
docparse/data/output.md — LLM-ready markdown

What AILANG Parse Extracts

Feature	DOCX	PPTX	XLSX	Best Competitor
Tables with merged cells	Yes	Yes	Yes	Raw OOXML only
Track changes (redlining)	Yes	—	—	Pandoc (3/3)
Comments (interleaved)	Yes	—	—	Raw OOXML (2/2)
Headers/footers	Yes	—	—	Kreuzberg (2/3)
Text boxes / VML shapes	Yes	Yes	—	Raw OOXML (1/2)
Equations (§22.1)	Yes	—	—	None
Field codes (§17.16)	Yes	—	—	Kreuzberg, OOXML
Speaker notes	—	Yes	—	None
Multi-sheet extraction	—	—	Yes	Kreuzberg

OfficeDocBench (69 files, 11 formats, 7 metrics): AILANG Parse 93.9% composite with 100% coverage vs nearest competitor 68.0% coverage-adjusted. 8 parsers compared including Raw OOXML, Pandoc, Kreuzberg, MarkItDown, Unstructured, Docling. Scores include aspirational ECMA-376 spec targets that intentionally lower our score.

Supported Formats

Parsing (16 formats): DOCX, PPTX, XLSX, ODT, ODP, ODS, HTML, Markdown, CSV, EPUB, EML, MBOX, TEX, RTF, PDF, images (JPG/PNG)

Generation (9 formats): DOCX, PPTX, XLSX, ODT, ODP, ODS, HTML, Markdown, QMD (Quarto)

Architecture

code

docparse/
├── types/document.ail           # Block ADT (9 variants)
├── services/
│   ├── format_router.ail        # Format detection (36 inline tests)
│   ├── zip_extract.ail          # ZIP layer (9 inline tests)
│   ├── docx_parser.ail          # DOCX XML → Blocks (6 inline tests)
│   ├── pptx_parser.ail          # PPTX slides → Blocks
│   ├── xlsx_parser.ail          # XLSX worksheets → Blocks
│   ├── direct_ai_parser.ail     # PDF/image → Blocks (AI)
│   ├── layout_ai.ail            # AI self-healing (optional)
│   ├── output_formatter.ail     # JSON + markdown output
│   └── docparse_browser.ail     # WASM browser adapter
└── main.ail                     # CLI entry point

28+ contracts, 50+ inline tests.

AI Configuration

AILANG Parse uses AILANG's AI effect — any model AILANG supports works:

bash

docparse scan.pdf --ai gemini-2.5-flash          # Google (default; fast)
docparse scan.pdf --ai gemini-3-flash-preview    # Google (slower; thinking model)
docparse scan.pdf --ai granite-docling           # Local Ollama (free)
docparse scan.pdf --ai claude-haiku-4-5          # Anthropic

AI usage is bounded by capability budgets (AI @limit=30), so costs are predictable.

Dev Commands

bash

docparse --check       # Type-check all modules
docparse --test        # Run inline tests
docparse --prove       # Static Z3 contract verification

Benchmarks

bash

uv run benchmarks/run_benchmarks.py --suite office     # Structural (no API, instant)
uv run benchmarks/run_benchmarks.py --suite pdf         # PDF extraction (needs AI)
uv run benchmarks/run_benchmarks.py --competitors       # Compare to Docling etc.

See benchmarks/ for details.

License

Apache 2.0

Install

Configuration

Environment variables

DOCPARSE_API_KEYsecret

AILANG Parse API key (dp_...). Optional — the bridge auto-loads keys saved at ~/.config/ailang-parse/credentials.json. Get one from https://www.sunholo.com/ailang-parse/

AILANG_PARSE_MCP_URL

Override the hosted MCP endpoint. Defaults to https://docparse.ailang.sunholo.com/mcp/

Arguments

subcommanddefault mcp

claude_desktop_config.json

{
  "mcpServers": {
    "parse": {
      "command": "npx",
      "args": [
        "-y",
        "@ailang/parse@0.8.0",
        "mcp"
      ],
      "env": {
        "DOCPARSE_API_KEY": "<YOUR_DOCPARSE_API_KEY>",
        "AILANG_PARSE_MCP_URL": "<YOUR_AILANG_PARSE_MCP_URL>"
      }
    }
  }
}