io.github.einiba/canyougrab

Official

by einiba · Python

CanYouGrab.it

Confidence-scored domain availability checking for AI agents via CanYouGrab.it

Check domain availability with confidence scores for AI agents using CanYouGrab.it.

Topics

ai-agentchatgptclaudednsdomain-availabilitydomain-checkermcpmcp-servermodel-context-protocolwhois

README.md25 KBView on GitHub

CanYouGrab API

Domain availability lookup API with subscription billing, built on FastAPI + DNS (Unbound resolver).

Live services:

API: https://api.canyougrab.it
Developer portal: https://portal.canyougrab.it
Auth: https://auth.canyougrab.it (Auth0 custom domain)

Architecture Overview

code

┌──────────────────────────────────────────────────────────────────┐
│                        Developer Portal                          │
│              (Zudoku/React on portal.canyougrab.it)              │
│   Usage Dashboard · API Keys · Pricing · API Reference (OAS)    │
└──────────────────┬───────────────────────────────────────────────┘
                   │  Auth0 JWT (portal)  /  Bearer API key (API)
                   ▼
┌──────────────────────────────────────────────────────────────────┐
│                    FastAPI Backend (v5.0.0)                       │
│                    api.canyougrab.it:8000                         │
│                                                                  │
│  ┌─────────────┐  ┌──────────┐  ┌──────────┐  ┌─────────────┐  │
│  │ app.py      │  │ keys.py  │  │billing.py│  │ auth.py     │  │
│  │ /check/bulk │  │ /keys    │  │/billing  │  │ API key +   │  │
│  │ /usage      │  │ CRUD     │  │/stripe   │  │ JWT auth    │  │
│  │             │  │ rotate   │  │ webhook  │  │             │  │
│  └──────┬──────┘  └──────────┘  └────┬─────┘  └─────────────┘  │
│         │                            │                           │
└─────────┼────────────────────────────┼───────────────────────────┘
          │                            │
          ▼                            ▼
┌─────────────────┐          ┌─────────────────┐
│  Valkey (Redis)  │          │   Stripe API     │
│  Job queue +     │          │   Subscriptions   │
│  Rate limiting   │          │   Webhooks        │
└────────┬────────┘          └─────────────────┘
         │
         ▼
┌─────────────────┐          ┌──────────────────────────────┐
│  RQ Worker       │──DNS──▶ │   Unbound Resolver            │
│  (worker.py)     │          │   (dedicated droplet)         │
│  ThreadPool(10)  │          │   NS queries via VPC          │
│  via RQ queue    │          └──────────────────────────────┘
└─────────────────┘
                             ┌──────────────────────────────┐
         ───────────────────▶│   PostgreSQL                  │
          (auth, usage,      │   API keys + Usage logs       │
           billing only)     └──────────────────────────────┘

Directory Structure

code

zuplo/
├── backend/                    # Python FastAPI backend
│   ├── app.py                  # Main API: /check/bulk, /usage, /health
│   ├── auth.py                 # API key auth (SHA-256) + Auth0 JWT auth (RS256)
│   ├── billing.py              # Stripe checkout, portal, webhooks, card-on-file, usage details
│   ├── keys.py                 # API key CRUD: create, list, rotate, revoke (+ Turnstile)
│   ├── antifraud.py            # Anti-fraud: Turnstile, device fingerprints, risk scoring
│   ├── email_utils.py          # Email normalization + disposable email detection
│   ├── dns_client.py           # DNS-based domain availability checking via Unbound
│   ├── queries.py              # PostgreSQL queries: usage tracking, auth, billing
│   ├── valkey_client.py        # Redis/Valkey job queue client (RQ-backed)
│   ├── rq_tasks.py             # RQ task function: process_domain_job()
│   ├── worker.py               # RQ worker process (ThreadPoolExecutor per job)
│   ├── migrations/             # SQL migrations
│   │   └── 001_free_tier_antifraud.sql
│   └── requirements.txt        # Python dependencies
├── portal/                     # Developer portal (Zuplo + Zudoku)
│   ├── config/
│   │   ├── routes.oas.json     # OpenAPI 3.1 spec (public API documentation)
│   │   └── policies.json       # Zuplo policies (empty — all routing is direct)
│   ├── docs/                   # Zudoku documentation portal
│   │   ├── src/
│   │   │   ├── config.ts       # API_BASE, Turnstile site key, Stripe PK
│   │   │   ├── UsageDashboard.tsx   # Usage + billing dashboard component
│   │   │   ├── PricingPage.tsx      # Plan selection + Stripe checkout
│   │   │   ├── PricingPlans.tsx     # Pricing card grid component
│   │   │   └── CardSetupPage.tsx    # Stripe Elements card-on-file for Free+
│   │   ├── public/             # Static assets (logos, banners, CSS overrides)
│   │   ├── zudoku.config.tsx   # Portal config: theme, nav, Auth0, API key mgmt
│   │   └── package.json        # Frontend dependencies (React 19, Zudoku)
│   ├── package.json            # Workspace root (Zuplo v6, TypeScript v5)
│   └── README.md               # Zuplo boilerplate (not project-specific)
├── mcp-server/                 # MCP package for ChatGPT, Claude, and remote MCP clients
│   ├── pyproject.toml          # MCP package metadata + version
│   ├── server.json             # MCP registry/server metadata
│   ├── uv.lock                 # Locked MCP runtime dependencies
│   └── src/canyougrab_mcp/
│       ├── __init__.py
│       └── server.py           # stdio + streamable-http MCP entrypoint
├── .github/workflows/
│   ├── deploy.yml              # Production deploy (on tag push v*)
│   └── deploy-dev.yml          # Dev deploy (on push to dev branch)
├── .claude/
│   └── launch.json             # Local dev server config (port 9200)
└── package.json                # Root workspace config

API Endpoints

Public API (API key auth: `Authorization: Bearer cyg_...`)

Method	Path	Description
`POST`	`/api/check/bulk`	Check up to 100 domains. Long-polls until results ready (30s max).
`GET`	`/api/account/usage`	Usage summary for the authenticated consumer.
`GET`	`/api/account/quota-check`	Lightweight monthly + per-minute quota check.
`GET`	`/health`	Health check (no auth).

Portal API (Auth0 JWT auth)

Method	Path	Description
`POST`	`/api/keys`	Create new API key.
`GET`	`/api/keys`	List user's API keys.
`POST`	`/api/keys/{id}/rotate`	Rotate key (revoke old, create new).
`DELETE`	`/api/keys/{id}`	Revoke (soft-delete) a key.
`POST`	`/api/billing/checkout`	Create Stripe Checkout session.
`POST`	`/api/billing/portal`	Create Stripe Customer Portal session.
`POST`	`/api/billing/setup-card`	Create SetupIntent for Free+ card-on-file.
`POST`	`/api/billing/confirm-free-plus`	Verify card fingerprint and upgrade to Free+.
`GET`	`/api/billing/card-status`	Check if user has a card on file.
`GET`	`/api/billing/usage/detailed`	Per-key usage breakdown for portal dashboard.
`POST`	`/api/antifraud/turnstile/verify`	Verify Cloudflare Turnstile token.
`POST`	`/api/antifraud/device/register`	Register device fingerprint (Fingerprint Pro).
`GET`	`/api/antifraud/risk`	Get risk assessment for authenticated user.
`POST`	`/api/antifraud/assess-signup`	Run multi-signal risk assessment at signup.

Internal / Webhook

Method	Path	Description
`POST`	`/api/account/usage/detailed`	Multi-consumer usage breakdown.
`POST`	`/api/stripe/webhook`	Stripe webhook receiver (signature-verified).

Core Request Flow

Domain Availability Check

code

Client                    FastAPI                     Valkey                      Worker              Unbound
  │                          │                          │                          │                    │
  │  POST /api/check/bulk    │                          │                          │                    │
  │  { domains: [...] }      │                          │                          │                    │
  │─────────────────────────▶│                          │                          │                    │
  │                          │── validate key ──────────│                          │                    │
  │                          │── check minute rate ────▶│ INCR ratelimit:id:min   │                    │
  │                          │── check monthly quota ──▶│ (PostgreSQL)             │                    │
  │                          │── record usage ─────────▶│ (PostgreSQL)             │                    │
  │                          │── create_job() ─────────▶│ HSET job:{uuid}          │                    │
  │                          │                          │ RQ enqueue (with retry)   │                    │
  │                          │                          │                          │                    │
  │                          │                          │◀── RQ Worker dequeues ───│                    │
  │                          │                          │                          │                    │
  │                          │                          │──── claim_job() ────────▶│                    │
  │                          │                          │                          │── ThreadPool(10)   │
  │                          │                          │                          │── check_domain_dns()
  │                          │                          │                          │── NS query ───────▶│
  │                          │                          │                          │◀── NOERROR/NXDOMAIN│
  │                          │                          │◀── complete_job() ───────│                    │
  │                          │                          │                          │                    │
  │                          │◀─ poll get_job_status() ─│                          │                    │
  │                          │   (0.3s interval, 30s    │                          │                    │
  │                          │    max timeout)          │                          │                    │
  │◀─────────────────────────│                          │                          │                    │
  │  { results: [...] }      │                          │                          │                    │

Billing / Subscription Flow

code

User → Pricing Page → Select Plan → Auth0 login
  → POST /api/billing/checkout (JWT auth)
  → Find/create Stripe customer (linked by auth0_sub metadata)
  → Stripe Checkout Session created → redirect to Stripe
  → User pays → Stripe fires webhook
  → POST /api/stripe/webhook (HMAC-SHA256 verified)
  → checkout.session.completed → fetch subscription → get price ID
  → Map price to plan → UPDATE api_keys SET plan, lookups_limit

Subscription Plans

Plan	Monthly Lookups	Per-Minute Rate Limit	Domains/Request	Price
Free	500	30/min	30	$0
Free+	10,000	100/min	100	$0 (card on file)
Basic	20,000	300/min	100	$10/mo
Pro	50,000	1,000/min	100	$20/mo
Business	300,000	3,000/min	100	$30/mo

Authentication

API Key Auth (public API consumers)

Format: Authorization: Bearer cyg_<token>
Keys are SHA-256 hashed in the api_keys table (plaintext never stored)
Key prefix (cyg_XXXXXXXX...) is stored for display in the portal
Full key returned only once at creation time
Soft-delete on revocation (revoked_at timestamp, not physically deleted)

JWT Auth (portal/dashboard)

Auth0 tenant: dev-mqe5tavp6dr62e7u
Custom domain: auth.canyougrab.it
Algorithm: RS256 with JWKS validation (1-hour cache)
Audience: https://api.canyougrab.it
Social logins: Google, Apple (Sign in with Apple)

Domain Availability via DNS

Domain availability is checked by querying a dedicated Unbound recursive DNS resolver running on a separate DigitalOcean droplet. The worker sends NS record queries over the VPC private network and interprets the response:

DNS Response	`available`	Meaning
NOERROR + NS records	`false`	Domain is registered and delegated
NXDOMAIN	`true`	Domain not in zone — probably available
NoAnswer (NOERROR, no NS)	`false`	Registered but parked/undelegated
SERVFAIL / Timeout	`null`	Ambiguous — check failed

The Unbound resolver caches aggressively (7 days for registered domains, 5 minutes for NXDOMAIN) and queries TLD authoritative servers directly, avoiding public resolver rate limits.

Database Schema

PostgreSQL is used for authentication, usage tracking, and billing — not for domain lookups.

PostgreSQL Tables

Table	Purpose	Key Columns
`api_keys`	User API keys	`id`, `user_sub`, `email_normalized`, `key_hash`, `key_prefix`, `plan`, `lookups_limit`, `revoked_at`
`usage_log_daily`	Daily usage aggregates	`consumer`, `lookups`, `recorded_at` (DATE, unique per consumer+day)
`usage_log_minute`	Per-minute usage aggregates	`consumer`, `lookups`, `minute_start` (TIMESTAMP, unique per consumer+minute)
`card_fingerprints`	One free account per card	`user_sub`, `stripe_fingerprint` (unique pair)
`device_fingerprints`	Device-based multi-account detection	`user_sub`, `visitor_id` (Fingerprint Pro)
`account_risk`	Composite risk scoring	`user_sub` (unique), `risk_score`, `risk_signals` (JSONB)

Valkey (Redis) Data Structures

Key Pattern	Type	Purpose	TTL
`job:{uuid}`	Hash	Job status, domains, results	1 hour
`rq:queue:canyougrab-jobs`	List	RQ job queue (managed by RQ)	—
`rq:job:{uuid}`	Hash	RQ job metadata (retries, status)	Configurable
`ratelimit:{consumer}:{YYYYMMDDHHmm}`	String	Per-minute rate limit counter (INCR)	60s

Infrastructure

Servers

Environment	Domain	Purpose
Production	`api.canyougrab.it`	FastAPI + Worker
Dev	`dev.canyougrab.it`	FastAPI + Worker
Portal	`portal.canyougrab.it`	Zudoku static site
Auth	`auth.canyougrab.it`	Auth0 custom domain

External Services

Service	Purpose
DigitalOcean	Droplets (API servers, Unbound resolver), Managed PostgreSQL, Managed Valkey
Cloudflare	DNS, CDN, SSL for canyougrab.it zone, Turnstile bot prevention
Auth0	User authentication, social login (Google + Apple), JWT issuance
Stripe	Subscription billing, checkout, customer portal, webhooks
GitHub Actions	CI/CD deployment pipelines

Deployment Pipelines

Production Deploy

Trigger: Push a git tag matching v* (e.g., v1.0.5)

code

git tag v1.0.5 && git push origin v1.0.5

Pipeline (.github/workflows/deploy.yml):

GitHub Actions triggers on v* tag push
SSH into production server (DEPLOY_HOST secret) using DEPLOY_SSH_KEY
Bootstraps the target ref on the server (git fetch + git checkout <tag>)
Runs the repo-managed deploy script: /opt/canyougrab-repo/scripts/deploy-host.sh

Dev Deploy

Trigger: Push to the dev branch

code

git push origin dev

Pipeline (.github/workflows/deploy-dev.yml):

GitHub Actions triggers on push to dev
SSH into dev server (DEV_DEPLOY_HOST secret) using same DEPLOY_SSH_KEY
Bootstraps the dev ref on the server (git fetch + git checkout dev)
Runs the repo-managed deploy script: /opt/canyougrab-repo/scripts/deploy-host.sh

Repo-Managed Host Deploy Script

Both pipelines use a small inline SSH bootstrap to update /opt/canyougrab-repo to the target ref, then call scripts/deploy-host.sh from that checked-out revision. The repo-managed deploy script handles:

pip install -r requirements.txt for backend dependencies
rsync of backend, portal, and MCP source trees into their runtime directories
Reinstall or refresh the mcp-server package/runtime when the host also runs canyougrab-mcp.service
Restart of FastAPI (uvicorn), worker, and MCP services via systemd

If /mcp is served by the same host as the API, backend-only deploys are not enough. The MCP service must be updated and restarted during the same deploy or the OAuth metadata and live MCP behavior can drift apart.

An existing /opt/deploy.sh can still be kept as a manual compatibility shim if desired, but the automated pipeline should treat the repo copy of scripts/deploy-host.sh as the source of truth.

Branching Strategy

Branch	Purpose	Deploys To
`main`	Production releases	Tagged → `api.canyougrab.it`
`dev`	Development/staging	Auto → `dev.canyougrab.it`
Feature branches	In-progress work	No auto-deploy

Environment Variables

Backend (required on servers)

bash

# DNS Resolver (Unbound on dedicated droplet)
DNS_RESOLVER_HOSTNAME=unbound.canyougrab.internal  # VPC internal hostname (resolved via socket.gethostbyname)
DNS_RESOLVER_PORT=53
DNS_QUERY_TIMEOUT=5.0  # Per-query timeout in seconds

# PostgreSQL (DigitalOcean Managed Database — auth, usage, billing only)
POSTGRES_HOST=         # DB cluster hostname
POSTGRES_PORT=5432
POSTGRES_DB=canyougrab
POSTGRES_USER=canyougrab
POSTGRES_PASSWORD=     # DB password
POSTGRES_SSLMODE=require

# Valkey / Redis (DigitalOcean Managed Database)
VALKEY_HOST=           # Valkey cluster hostname
VALKEY_PORT=25061
VALKEY_USERNAME=default
VALKEY_PASSWORD=       # Valkey password

# Auth0
AUTH0_DOMAIN=dev-mqe5tavp6dr62e7u.us.auth0.com
AUTH0_AUDIENCE=https://api.canyougrab.it

# Stripe
STRIPE_SECRET_KEY=     # sk_live_... (prod) or sk_test_... (dev)
STRIPE_WEBHOOK_SECRET= # whsec_... (per-environment)
STRIPE_PRICE_BASIC=    # price_... (live price ID for Basic plan)
STRIPE_PRICE_PRO=      # price_... (live price ID for Pro plan)
STRIPE_PRICE_BUSINESS= # price_... (live price ID for Business plan)

# Cloudflare Turnstile (bot prevention on key creation)
TURNSTILE_SECRET_KEY=  # 0x4AAAA... (from Cloudflare dashboard)

# Portal
PORTAL_URL=https://portal.canyougrab.it

# Worker
BATCH_CONCURRENCY=10   # Thread pool size for domain checks
VALKEY_QUEUE_NAME=canyougrab-jobs  # RQ queue name (default: canyougrab-jobs)

# Monitoring (optional — only if monitoring stack is installed)
SLACK_ALERTS_WEBHOOK_URL=  # Slack incoming webhook for Alertmanager
RQ_METRICS_PORT=9122       # Prometheus metrics exporter port

# Auto-scaler (optional — only on API host with autoscaler enabled)
DO_API_TOKEN=              # DigitalOcean API token
DO_WORKER_SNAPSHOT_ID=     # Snapshot ID for new worker droplets
AUTOSCALER_MIN_WORKERS=1
AUTOSCALER_MAX_WORKERS=5
AUTOSCALER_SCALE_UP_THRESHOLD=50
AUTOSCALER_SCALE_DOWN_IDLE_MINUTES=10

GitHub Actions Secrets

Secret	Purpose
`DEPLOY_HOST`	Production server IP
`DEV_DEPLOY_HOST`	Dev server IP
`DEPLOY_SSH_KEY`	SSH private key for both servers

Local Development

Backend

bash

cd backend
pip install -r requirements.txt

# Start API server
uvicorn app:app --reload --port 8000

# Start worker (separate terminal)
python worker.py

Requires DNS_RESOLVER_HOST pointing to an Unbound instance (or use 8.8.8.8 for basic testing), plus PostgreSQL and Valkey/Redis (local or tunneled to managed instances).

Portal

bash

cd portal/docs
npm install
npm run dev          # Starts Zudoku dev server (via zuplo)

Or use the configured launch server:

bash

# From repo root
npx zuplo dev        # Starts on port 9200

Portal dev server hardcodes API_BASE to https://api.canyougrab.it in portal/docs/src/config.ts. To develop against a local backend, change this to http://localhost:8000.

SEO: .it TLD Geo-Targeting Requirements

Because canyougrab.it uses a .it country-code TLD (Italy), all user-facing HTML (portal, docs) must include signals that the content targets English speakers:

<html lang="en"> on every page.

Hreflang tags in <head>:

html

<link rel="alternate" hreflang="en-US" href="{page_url}" />
<link rel="alternate" hreflang="en" href="{page_url}" />
<link rel="alternate" hreflang="x-default" href="{page_url}" />

Canonical tag — Self-referencing <link rel="canonical" href="{page_url}" />.
All content in English — No Italian text on any page.

This applies to the portal (portal.canyougrab.it), API docs, and any other publicly rendered HTML.

Key Design Decisions

Live DNS lookups: Domain availability is checked via NS queries to a dedicated Unbound recursive resolver. This gives real-time results (no 24-hour zone file lag), works for any TLD automatically, and avoids the operational complexity of daily batch zone file loading. Unbound caches aggressively (7 days for registered domains) so repeated queries are ~1ms.
Nullable availability: DNS has more failure modes than a database lookup. When Unbound returns SERVFAIL or times out, the API returns available: null instead of a potentially dangerous false positive. Consumers should treat null as "could not determine."
Job queue for bulk checks: The bulk endpoint uses RQ (Redis Queue) backed by Valkey to dispatch jobs with automatic retries (2 retries, 5s/30s backoff) and failed-job tracking. Workers parallelize DNS queries across a thread pool.
Long-polling: The bulk check endpoint holds the HTTP connection open (polling Valkey every 0.3s, up to 45s) rather than requiring clients to implement polling. Job results are stored in custom Valkey hashes (not RQ's built-in result storage) so the poll logic is independent of the queue framework.
API keys with SHA-256 hashing: Keys are hashed before storage (like passwords), so a database breach doesn't expose raw keys.
Stripe metadata linking: Stripe customers are linked to Auth0 users via auth0_sub metadata on the Stripe customer object, avoiding a separate mapping table.
Usage double-tracking: Both daily (usage_log_daily) and per-minute (usage_log_minute) usage are recorded for monthly quota and per-minute rate limiting respectively.
No Zuplo gateway policies: The project originally used Zuplo as an API gateway but migrated to direct FastAPI (commit 3abef70). The Zuplo portal/docs framework is still used for the developer portal.

Processes on Servers

Core services (every host)

FastAPI (uvicorn): Serves all HTTP endpoints on port 8000
RQ Worker (worker.py): Processes domain check jobs from the RQ queue
MCP server (if /mcp is hosted on this server): Serves remote MCP clients

Monitoring stack (API host, optional)

Prometheus (localhost:9090): Scrapes metrics, evaluates alert rules
Alertmanager (localhost:9093): Routes alerts to Slack
redis_exporter (localhost:9121): Exports Valkey metrics to Prometheus
RQ metrics exporter (localhost:9122): Exports queue depth, worker count, failed jobs
Grafana (localhost:3000, nginx-proxied): Dashboards for queue health
Auto-scaler (optional): Scales worker droplets up/down via DO API

Install the monitoring stack with: sudo bash scripts/setup-monitoring.sh

All active services on the host must be managed via systemd and restarted during deployments. All automated deploys should route through scripts/deploy-host.sh, which is responsible for restarting whichever of these services are installed on the host.

Install

claude_desktop_config.json

{
  "mcpServers": {
    "canyougrab": {
      "command": "uvx",
      "args": [
        "mcp-server-canyougrab"
      ]
    }
  }
}

Topics

CanYouGrab API

Architecture Overview

Directory Structure

API Endpoints

Public API (API key auth: Authorization: Bearer cyg_...)

Portal API (Auth0 JWT auth)

Internal / Webhook

Core Request Flow

Domain Availability Check

Billing / Subscription Flow

Subscription Plans

Authentication

API Key Auth (public API consumers)

JWT Auth (portal/dashboard)

Domain Availability via DNS

Database Schema

PostgreSQL Tables

Valkey (Redis) Data Structures

Infrastructure

Servers

External Services

Deployment Pipelines

Production Deploy

Dev Deploy

Repo-Managed Host Deploy Script

Branching Strategy

Environment Variables

Backend (required on servers)

GitHub Actions Secrets

Local Development

Backend

Portal

SEO: .it TLD Geo-Targeting Requirements

Key Design Decisions

Processes on Servers

Core services (every host)

Monitoring stack (API host, optional)

Install

Public API (API key auth: `Authorization: Bearer cyg_...`)