IA-QA — 130+ QA & Dev Tools for AI Agents

Official149 toolsLive

by JcJamet

130+ QA & dev tools for AI agents: prompt injection, RAG testing, VLM eval, guardrails. Free.

The server provides a collection of 130+ QA and development tools for AI agents, covering areas such as prompt injection, RAG testing, VLM evaluation, and guardrails. It functions as a cataloged toolbox for QA and AI tooling, offering a broad suite of utilities to support testing and evaluation workflows. The repository highlights a large toolset with 147 tools documented or available for use.

🛠️ Key Features

130+ QA and dev tools for AI agents
Focused on prompt injection, RAG testing, VLM eval, and guardrails
Free access and usage

🚀 Use Cases

Evaluate prompt robustness and vulnerabilities
Test retrieval-augmented generation pipelines
Assess visual-language model performance
Define and verify guardrails for AI agents

⚡ Developer Benefits

Centralized toolbox for AI QA workflows
Extensive tool coverage for rapid prototyping
Clear focus areas to guide testing efforts

⚠️ Limitations

Specific tool details and integration instructions are not included in the excerpt
Tool availability or licensing not explicitly stated

Captured live from the server via tools/list.

format_json

Validate and pretty-print a string that is ALREADY valid JSON. Strict by design — it is a validity gate: valid JSON comes back formatted, anything else is rejected with the exact parse error. It never repairs, completes, or guesses. NOT for: plain text or prose (will fail), JSON embedded in markdown/prose (use extract_json_from_text first), JS objects (JSON.stringify them first), YAML (use yaml_to_json).

Parameters2

input	string	required	A raw JSON string, e.g. '{"key":"value"}'. Must already parse as JSON — plain text or truncated JSON is rejected, not repaired.
indent	number	optional	Indent size (default: 2)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "A raw JSON string, e.g. '{\"key\":\"value\"}'. Must already parse as JSON — plain text or truncated JSON is rejected, not repaired."
    },
    "indent": {
      "type": "number",
      "description": "Indent size (default: 2)"
    }
  },
  "required": [
    "input"
  ]
}

generate_uuid

Generate one or more cryptographically random UUID v4 identifiers. Use this when you need unique IDs for test fixtures, database records, session tokens, or any scenario requiring a guaranteed-unique string. Returns up to 100 UUIDs in one call.

Parameters1

count

number

optional

Number of UUIDs to generate (1–100, default: 1)

Raw schema

{
  "type": "object",
  "properties": {
    "count": {
      "type": "number",
      "description": "Number of UUIDs to generate (1–100, default: 1)"
    }
  }
}

hash_text

Compute a cryptographic hash of a text string. Use when you need to verify data integrity, generate content fingerprints, hash passwords (prefer SHA-256+), or produce a fixed-length digest of any input. Supports SHA-256 (default), SHA-512, SHA-1, and MD5.

Parameters2

input	string	required	Text to hash
algorithm	string	optional	Hash algorithm: sha256 (default), sha512, sha1, md5

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to hash"
    },
    "algorithm": {
      "type": "string",
      "description": "Hash algorithm: sha256 (default), sha512, sha1, md5"
    }
  },
  "required": [
    "input"
  ]
}

count_tokens

Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it fits within the model context window and to estimate cost. Returns token estimate, character count, and word count.

Parameters1

input

string

required

Text to count tokens for

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to count tokens for"
    }
  },
  "required": [
    "input"
  ]
}

base64_encode

Encode a UTF-8 string to Base64. Use when you need to embed binary data, multi-line text, or special characters safely inside JSON fields, HTTP headers, or data URIs.

Parameters1

input

string

required

Text to encode

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to encode"
    }
  },
  "required": [
    "input"
  ]
}

base64_decode

Decode a Base64 string back to UTF-8 text. Use for inspecting Base64-encoded API responses, JWT payload claims, config file values, or attachment data.

Parameters1

input

string

required

Base64 string to decode

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Base64 string to decode"
    }
  },
  "required": [
    "input"
  ]
}

url_encode

Percent-encode a string for safe use in URLs. Call this before programmatically building query strings, path segments, or form-encoded bodies to prevent injection and malformed URLs.

Parameters2

input	string	required	String to URL-encode
mode	string	optional	"component" (default) or "full" for encodeURI behavior

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "String to URL-encode"
    },
    "mode": {
      "type": "string",
      "description": "\"component\" (default) or \"full\" for encodeURI behavior"
    }
  },
  "required": [
    "input"
  ]
}

url_decode

Decode a percent-encoded URL string back to plain text. Use when parsing query parameters from raw URLs or when displaying encoded values to users.

Parameters1

input

string

required

URL-encoded string to decode

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "URL-encoded string to decode"
    }
  },
  "required": [
    "input"
  ]
}

generate_slug

Convert any string into a URL-friendly slug: lowercase, ASCII-normalized (é→e), special characters removed, spaces replaced with hyphens. Use for generating SEO-friendly URL paths, file names, or identifier keys from user-provided titles or labels.

Parameters2

input	string	required	String to slugify
separator	string	optional	Separator character (default: "-")

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "String to slugify"
    },
    "separator": {
      "type": "string",
      "description": "Separator character (default: \"-\")"
    }
  },
  "required": [
    "input"
  ]
}

validate_email

Validate an email address against RFC 5322 syntax before storing it, sending a transactional email, or adding it to a mailing list. Returns { valid, email } — use this to avoid bounces and malformed data.

Parameters1

string

required

Email address to validate

Raw schema

{
  "type": "object",
  "properties": {
    "email": {
      "type": "string",
      "description": "Email address to validate"
    }
  },
  "required": [
    "email"
  ]
}

minify_js

Minify a JavaScript snippet, function, class, or module up to 50 KB using Terser. Returns minified code and byte savings. Use when embedding scripts in HTML templates, report payloads, or injecting inline code programmatically.

Parameters1

code

string

required

JavaScript code to minify (max 50kb)

Raw schema

{
  "type": "object",
  "properties": {
    "code": {
      "type": "string",
      "description": "JavaScript code to minify (max 50kb)"
    }
  },
  "required": [
    "code"
  ]
}

decode_jwt

Decode a JWT (JSON Web Token) and return its header and payload without verifying the signature. Also reports whether the token is expired and the exact expiry date. Use to inspect claims (sub, iss, exp, roles) during debugging or when integrating with an auth provider.

Parameters1

token

string

required

The JWT string to decode (header.payload.signature)

Raw schema

{
  "type": "object",
  "properties": {
    "token": {
      "type": "string",
      "description": "The JWT string to decode (header.payload.signature)"
    }
  },
  "required": [
    "token"
  ]
}

text_stats

Compute comprehensive statistics for any text: character count (with and without spaces), word count, line count, sentence count, paragraph count, and estimated reading time in minutes. Use for validating form field lengths, evaluating LLM output verbosity, or content auditing.

Parameters1

input

string

required

The text to analyse

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "The text to analyse"
    }
  },
  "required": [
    "input"
  ]
}

generate_password

Generate a cryptographically secure random password using crypto.randomBytes. Configurable length (4–128), uppercase letters, digits, and symbols. Use when resetting user passwords, seeding test accounts, or generating API secrets.

Parameters4

length	number	optional	Password length (4–128, default: 16)
uppercase	boolean	optional	Include uppercase letters (default: true)
numbers	boolean	optional	Include digits (default: true)
symbols	boolean	optional	Include symbols like !@#$ (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "length": {
      "type": "number",
      "description": "Password length (4–128, default: 16)"
    },
    "uppercase": {
      "type": "boolean",
      "description": "Include uppercase letters (default: true)"
    },
    "numbers": {
      "type": "boolean",
      "description": "Include digits (default: true)"
    },
    "symbols": {
      "type": "boolean",
      "description": "Include symbols like !@#$ (default: false)"
    }
  }
}

parse_csv

Parse a CSV string into a JSON array of objects (or raw arrays). Handles RFC 4180 quoted fields, escaped quotes, and custom delimiters. Use when processing spreadsheet exports, data imports, or structured text pipelines where the source is CSV. Supports up to 200 KB.

Parameters3

input	string	required	CSV content to parse
delimiter	string	optional	Field delimiter character (default: ",")
header	boolean	optional	Treat the first row as headers (default: true)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "CSV content to parse"
    },
    "delimiter": {
      "type": "string",
      "description": "Field delimiter character (default: \",\")"
    },
    "header": {
      "type": "boolean",
      "description": "Treat the first row as headers (default: true)"
    }
  },
  "required": [
    "input"
  ]
}

color_convert

Convert a color between HEX, RGB, and HSL formats. Use when translating design tokens between CSS notations, verifying color accessibility, or normalizing color values from user input. Accepts #rrggbb, #rgb, rgb(r,g,b), or hsl(h,s%,l%).

Parameters1

input

string

required

Color value to convert, e.g. "#ff6b6b", "rgb(255,107,107)", "hsl(0,100%,71%)"

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Color value to convert, e.g. \"#ff6b6b\", \"rgb(255,107,107)\", \"hsl(0,100%,71%)\""
    }
  },
  "required": [
    "input"
  ]
}

regex_test

Test a regular expression pattern against an input string and return all matches with their index positions and named capture groups. Use for validating user inputs, extracting structured data from text, or debugging regex patterns. Supports flags g, i, m, s, u, y.

Parameters3

pattern	string	required	Regular expression pattern (without delimiters)
input	string	required	The string to test against (max 50 KB)
flags	string	optional	Regex flags: g (global), i (case-insensitive), m (multiline), s (dotAll) — default: ""

Raw schema

{
  "type": "object",
  "properties": {
    "pattern": {
      "type": "string",
      "description": "Regular expression pattern (without delimiters)"
    },
    "input": {
      "type": "string",
      "description": "The string to test against (max 50 KB)"
    },
    "flags": {
      "type": "string",
      "description": "Regex flags: g (global), i (case-insensitive), m (multiline), s (dotAll) — default: \"\""
    }
  },
  "required": [
    "pattern",
    "input"
  ]
}

lorem_ipsum

Generate Lorem Ipsum placeholder text for UI mockups, design prototypes, or test data population. Configurable paragraphs (1–10), sentences per paragraph (1–20), and approximate words per sentence (3–30).

Parameters3

paragraphs	number	optional	Number of paragraphs to generate (1–10, default: 1)
sentences_per_paragraph	number	optional	Sentences per paragraph (1–20, default: 5)
words_per_sentence	number	optional	Approximate words per sentence (3–30, default: 10)

Raw schema

{
  "type": "object",
  "properties": {
    "paragraphs": {
      "type": "number",
      "description": "Number of paragraphs to generate (1–10, default: 1)"
    },
    "sentences_per_paragraph": {
      "type": "number",
      "description": "Sentences per paragraph (1–20, default: 5)"
    },
    "words_per_sentence": {
      "type": "number",
      "description": "Approximate words per sentence (3–30, default: 10)"
    }
  }
}

timestamp_convert

Convert between Unix timestamps (seconds or milliseconds) and ISO-8601 / UTC date strings. Auto-detects epoch vs. millisecond format. Omit input to get the current time. Returns iso, unix_s, unix_ms, utc, date, and time fields.

Parameters1

input

any

optional

Unix timestamp (number, seconds or ms) or ISO date string. Omit to get the current time.

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "description": "Unix timestamp (number, seconds or ms) or ISO date string. Omit to get the current time."
    }
  }
}

diff_text

Compute a unified line-by-line diff between two text strings (LCS algorithm). Returns added/removed/unchanged line counts and formatted diff hunks with configurable context lines (0–20). Use to compare versions of prompts, configs, code snippets, or any text where you need to see exactly what changed.

Parameters3

a	string	required	Original (before) text
b	string	required	Modified (after) text
context	number	optional	Context lines around each change (0–20, default: 3)

Raw schema

{
  "type": "object",
  "properties": {
    "a": {
      "type": "string",
      "description": "Original (before) text"
    },
    "b": {
      "type": "string",
      "description": "Modified (after) text"
    },
    "context": {
      "type": "number",
      "description": "Context lines around each change (0–20, default: 3)"
    }
  },
  "required": [
    "a",
    "b"
  ]
}

truncate_to_tokens

Truncate text to at most N tokens (cl100k_base: ~4 chars/token) to avoid exceeding an LLM context window. Optionally keeps the end of the text instead of the start (useful for keeping recent conversation history). Reports whether truncation occurred and the estimated token count.

Parameters3

input	string	required	Text to truncate
max_tokens	number	required	Maximum number of tokens to keep
from_end	boolean	optional	Keep the end of the text instead of the start (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to truncate"
    },
    "max_tokens": {
      "type": "number",
      "description": "Maximum number of tokens to keep"
    },
    "from_end": {
      "type": "boolean",
      "description": "Keep the end of the text instead of the start (default: false)"
    }
  },
  "required": [
    "input",
    "max_tokens"
  ]
}

split_chunks

Split text into chunks of at most N tokens (cl100k_base: ~4 chars/token) with optional overlap. Designed for RAG ingestion pipelines.

Parameters3

input	string	required	Text to split into chunks
chunk_tokens	number	required	Maximum tokens per chunk (10–8000)
overlap	number	optional	Token overlap between consecutive chunks (default: 0)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to split into chunks"
    },
    "chunk_tokens": {
      "type": "number",
      "description": "Maximum tokens per chunk (10–8000)"
    },
    "overlap": {
      "type": "number",
      "description": "Token overlap between consecutive chunks (default: 0)"
    }
  },
  "required": [
    "input",
    "chunk_tokens"
  ]
}

extract_json_from_text

Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON. Call this whenever an LLM returns structured data mixed with explanation text instead of raw JSON.

Parameters1

input

string

required

Raw text (e.g., LLM output) that may contain a JSON object or array

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Raw text (e.g., LLM output) that may contain a JSON object or array"
    }
  },
  "required": [
    "input"
  ]
}

strip_markdown

Strip all Markdown formatting (headers, bold, italic, code fences, links, lists) from text and return clean plain text. Run this before injecting scraped documentation, README files, or user content into an LLM prompt to eliminate redundant markup tokens and reduce cost.

Parameters1

input

string

required

Markdown text to convert to plain text

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Markdown text to convert to plain text"
    }
  },
  "required": [
    "input"
  ]
}

estimate_llm_cost

Estimate the API cost in USD for a given model and token counts. Supports all major 2024–2026 models: GPT-4o, GPT-4.1, o3, o4-mini, Claude Opus 4, Claude Sonnet 4/4.5, Gemini 2.5 Pro/Flash, DeepSeek V3/R1, Grok 3, and legacy models.

Parameters3

model	string	required	Model name, e.g. "gpt-4o", "claude-3.5-sonnet", "deepseek-v3"
input_tokens	number	required	Number of input/prompt tokens
output_tokens	number	optional	Number of output/completion tokens (default: 0)

Raw schema

{
  "type": "object",
  "properties": {
    "model": {
      "type": "string",
      "description": "Model name, e.g. \"gpt-4o\", \"claude-3.5-sonnet\", \"deepseek-v3\""
    },
    "input_tokens": {
      "type": "number",
      "description": "Number of input/prompt tokens"
    },
    "output_tokens": {
      "type": "number",
      "description": "Number of output/completion tokens (default: 0)"
    }
  },
  "required": [
    "model",
    "input_tokens"
  ]
}

escape_html

Escape HTML special characters (&, <, >, ", ') to their safe HTML entities. ALWAYS call this before inserting any user-provided or LLM-generated content into an HTML template to prevent cross-site scripting (XSS) attacks.

Parameters1

input

string

required

String to HTML-escape

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "String to HTML-escape"
    }
  },
  "required": [
    "input"
  ]
}

unescape_html

Convert HTML entities (&, <, >, ", ', and numeric &#NNN;) back to plain characters. Use when processing HTML-encoded text from APIs, email content, or legacy database fields before passing to an LLM or displaying to users.

Parameters1

input

string

required

HTML-encoded string to unescape

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "HTML-encoded string to unescape"
    }
  },
  "required": [
    "input"
  ]
}

fetch_veille_feed

Fetch the latest QA & AI/LLM articles aggregated from curated RSS sources (Google Testing Blog, DEV.to Testing/QA/AI/LLM/Agents, Hugging Face Blog, Simon Willison). Perfect for agents monitoring the QA & AI landscape.

Parameters2

category	string	optional	Filter: "qa" (testing/quality), "ai" (AI/LLM/agents), "all" (default — both)
limit	number	optional	Max articles to return (default: 20, max: 50)

Raw schema

{
  "type": "object",
  "properties": {
    "category": {
      "type": "string",
      "description": "Filter: \"qa\" (testing/quality), \"ai\" (AI/LLM/agents), \"all\" (default — both)"
    },
    "limit": {
      "type": "number",
      "description": "Max articles to return (default: 20, max: 50)"
    }
  }
}

score_geo_signals

Analyze a webpage <head> HTML (or full HTML) for GEO (Generative Engine Optimization) signals. Returns a score /60 with per-check results and improvement tips. GEO = optimizing pages for AI-powered search engines (ChatGPT Search, Perplexity, etc.).

Parameters1

head_html

string

required

Raw HTML of the <head> section (or full page HTML) to analyze

Raw schema

{
  "type": "object",
  "properties": {
    "head_html": {
      "type": "string",
      "description": "Raw HTML of the <head> section (or full page HTML) to analyze"
    }
  },
  "required": [
    "head_html"
  ]
}

extract_json_path

Extract a value from a JSON string using dot-notation path (e.g., "user.address.city", "items.0.name", "meta.tags"). Supports array index access via numeric path segments.

Parameters2

input	string	required	A valid JSON string to traverse
path	string	required	Dot-notation path, e.g. "user.address.city" or "items.0.name"

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "A valid JSON string to traverse"
    },
    "path": {
      "type": "string",
      "description": "Dot-notation path, e.g. \"user.address.city\" or \"items.0.name\""
    }
  },
  "required": [
    "input",
    "path"
  ]
}

generate_json_ld

Generate a ready-to-paste <script type="application/ld+json"> snippet for GEO / structured data optimization. Supported types: WebSite, FAQPage, Article, Person, Organization, SoftwareApplication, HowTo.

Parameters3

type	string	required	Schema @type: "WebSite", "FAQPage", "Article", "Person", "Organization", "SoftwareApplication", "HowTo"
fields	object	optional	Schema fields as key-value pairs (name, url, description, author, datePublished, etc.)
faq_items	array	optional	For FAQPage/HowTo: array of { question, answer } objects

Raw schema

{
  "type": "object",
  "properties": {
    "type": {
      "type": "string",
      "description": "Schema @type: \"WebSite\", \"FAQPage\", \"Article\", \"Person\", \"Organization\", \"SoftwareApplication\", \"HowTo\""
    },
    "fields": {
      "type": "object",
      "additionalProperties": true,
      "description": "Schema fields as key-value pairs (name, url, description, author, datePublished, etc.)"
    },
    "faq_items": {
      "type": "array",
      "description": "For FAQPage/HowTo: array of { question, answer } objects",
      "items": {
        "type": "object",
        "properties": {
          "question": {
            "type": "string"
          },
          "answer": {
            "type": "string"
          }
        }
      }
    }
  },
  "required": [
    "type"
  ]
}

analyze_diff_bugs

Detect potential bugs and code smells from a git diff or two code versions. Returns a list of issues with severity levels and test suggestions.

Parameters3

version1	string	optional	Original code (before changes). If omitted, only the new version is analysed.
version2	string	required	New/modified code (after changes)
context	string	optional	Optional PR title or feature context for better analysis

Raw schema

{
  "type": "object",
  "properties": {
    "version1": {
      "type": "string",
      "description": "Original code (before changes). If omitted, only the new version is analysed."
    },
    "version2": {
      "type": "string",
      "description": "New/modified code (after changes)"
    },
    "context": {
      "type": "string",
      "description": "Optional PR title or feature context for better analysis"
    }
  },
  "required": [
    "version2"
  ]
}

generate_test_cases

Generate a set of test cases (valid, edge, invalid) for a given feature description. Returns test matrix with Gherkin scenarios ready to use.

Parameters2

feature	string	required	Feature or function to test. Be specific: describe inputs, expected behaviour, context.
inputs	string	optional	Optional: list of input parameters (one per line, e.g. "email: string [required]")

Raw schema

{
  "type": "object",
  "properties": {
    "feature": {
      "type": "string",
      "description": "Feature or function to test. Be specific: describe inputs, expected behaviour, context."
    },
    "inputs": {
      "type": "string",
      "description": "Optional: list of input parameters (one per line, e.g. \"email: string [required]\")"
    }
  },
  "required": [
    "feature"
  ]
}

run_pr_gate_pipeline

Full automated QA pipeline for a pull request. Takes a unified git diff (output of `git diff HEAD`) and returns: bug hotspots, regression impact areas, risk score (0–100), generated test cases, severity assessment, and a merge recommendation (PASS / CONDITIONAL / BLOCK). This is the highest-value QA tool — use it when reviewing any code change.

Parameters2

git_diff	string	required	Unified git diff (output of `git diff HEAD` or copied from GitHub diff view)
context	string	optional	Optional PR title or description for richer analysis

Raw schema

{
  "type": "object",
  "properties": {
    "git_diff": {
      "type": "string",
      "description": "Unified git diff (output of `git diff HEAD` or copied from GitHub diff view)"
    },
    "context": {
      "type": "string",
      "description": "Optional PR title or description for richer analysis"
    }
  },
  "required": [
    "git_diff"
  ]
}

validate_mcp_response

Validate that an MCP tool response conforms to expected format, schema, and content rules. Use this to QA-test any MCP server tool. Supply the tool's actual JSON result and a set of checks to perform.

Parameters8

response	string	required	The MCP tool result as a JSON string to validate
expected_type	string	optional	Expected top-level type: "object", "array", "string", "number"
required_keys	string	optional	Comma-separated list of keys that MUST exist in the response (dot-notation for nested: "data.id, data.name")
forbidden_keys	string	optional	Comma-separated list of keys that MUST NOT exist (e.g. "password, secret, token")
max_response_ms	number	optional	Maximum acceptable latency in ms (will be compared if provided)
actual_latency	number	optional	Actual measured latency in ms (from the call)
min_items	number	optional	If response is an array, minimum number of items expected
max_size_bytes	number	optional	Maximum acceptable response size in bytes

Raw schema

{
  "type": "object",
  "properties": {
    "response": {
      "type": "string",
      "description": "The MCP tool result as a JSON string to validate"
    },
    "expected_type": {
      "type": "string",
      "description": "Expected top-level type: \"object\", \"array\", \"string\", \"number\"",
      "enum": [
        "object",
        "array",
        "string",
        "number"
      ]
    },
    "required_keys": {
      "type": "string",
      "description": "Comma-separated list of keys that MUST exist in the response (dot-notation for nested: \"data.id, data.name\")"
    },
    "forbidden_keys": {
      "type": "string",
      "description": "Comma-separated list of keys that MUST NOT exist (e.g. \"password, secret, token\")"
    },
    "max_response_ms": {
      "type": "number",
      "description": "Maximum acceptable latency in ms (will be compared if provided)"
    },
    "actual_latency": {
      "type": "number",
      "description": "Actual measured latency in ms (from the call)"
    },
    "min_items": {
      "type": "number",
      "description": "If response is an array, minimum number of items expected"
    },
    "max_size_bytes": {
      "type": "number",
      "description": "Maximum acceptable response size in bytes"
    }
  },
  "required": [
    "response"
  ]
}

llm_output_validator

Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.

Parameters9

output	string	required	The LLM output text to validate
expected_format	string	optional	Expected output format
must_include	string	optional	Comma-separated strings that MUST appear in the output
must_not_include	string	optional	Comma-separated strings that must NOT appear (e.g. "TODO, FIXME, undefined, NaN")
max_length	number	optional	Maximum character length for the output
min_length	number	optional	Minimum character length for the output
expected_language	string	optional	Expected language of the output (en, fr, es, de…). Checks for common words.
check_safety	boolean	optional	Check for PII patterns (emails, phones, SSN), profanity signals, and prompt leakage
check_json_schema	string	optional	If expected_format is JSON, provide required keys as comma-separated list to validate the structure

Raw schema

{
  "type": "object",
  "properties": {
    "output": {
      "type": "string",
      "description": "The LLM output text to validate"
    },
    "expected_format": {
      "type": "string",
      "description": "Expected output format",
      "enum": [
        "json",
        "markdown",
        "code",
        "plain",
        "any"
      ]
    },
    "must_include": {
      "type": "string",
      "description": "Comma-separated strings that MUST appear in the output"
    },
    "must_not_include": {
      "type": "string",
      "description": "Comma-separated strings that must NOT appear (e.g. \"TODO, FIXME, undefined, NaN\")"
    },
    "max_length": {
      "type": "number",
      "description": "Maximum character length for the output"
    },
    "min_length": {
      "type": "number",
      "description": "Minimum character length for the output"
    },
    "expected_language": {
      "type": "string",
      "description": "Expected language of the output (en, fr, es, de…). Checks for common words."
    },
    "check_safety": {
      "type": "boolean",
      "description": "Check for PII patterns (emails, phones, SSN), profanity signals, and prompt leakage"
    },
    "check_json_schema": {
      "type": "string",
      "description": "If expected_format is JSON, provide required keys as comma-separated list to validate the structure"
    }
  },
  "required": [
    "output"
  ]
}

compare_responses

Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.

Parameters9

response_a	string	required	First output (e.g. model A's answer)
response_b	string	required	Second output (e.g. model B's answer)
label_a	string	optional	Label for output A (e.g. "GPT-4o", "v1.0")
label_b	string	optional	Label for output B (e.g. "GPT-5-nano", "v1.1")
task	string	optional	The task/prompt both outputs were answering — used by the LLM judge for context
reference	string	optional	Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic).
check_json	boolean	optional	Try to parse as JSON and compare structurally (keys, types, values)
model	string	optional	Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner.
api_key	string	optional	Optional API key for the judge model (BYOK). Used only for the judge call; never stored.

Raw schema

{
  "type": "object",
  "properties": {
    "response_a": {
      "type": "string",
      "description": "First output (e.g. model A's answer)"
    },
    "response_b": {
      "type": "string",
      "description": "Second output (e.g. model B's answer)"
    },
    "label_a": {
      "type": "string",
      "description": "Label for output A (e.g. \"GPT-4o\", \"v1.0\")"
    },
    "label_b": {
      "type": "string",
      "description": "Label for output B (e.g. \"GPT-5-nano\", \"v1.1\")"
    },
    "task": {
      "type": "string",
      "description": "The task/prompt both outputs were answering — used by the LLM judge for context"
    },
    "reference": {
      "type": "string",
      "description": "Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic)."
    },
    "check_json": {
      "type": "boolean",
      "description": "Try to parse as JSON and compare structurally (keys, types, values)"
    },
    "model": {
      "type": "string",
      "description": "Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner."
    },
    "api_key": {
      "type": "string",
      "description": "Optional API key for the judge model (BYOK). Used only for the judge call; never stored."
    }
  },
  "required": [
    "response_a",
    "response_b"
  ]
}

analyze_responses

Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. Deterministic, no LLM, no key — gate-able in CI. You bring the outputs (2+). For a 2-way head-to-head with structural JSON diff use compare_responses instead.

Parameters2

responses	array	required	The outputs to analyze (same task, N models/prompts/versions). Each item is a plain string or { "label": "GPT-4o", "text": "..." }. At least 2 required.
reference	string	optional	Optional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named.

Raw schema

{
  "type": "object",
  "properties": {
    "responses": {
      "type": "array",
      "minItems": 2,
      "description": "The outputs to analyze (same task, N models/prompts/versions). Each item is a plain string or { \"label\": \"GPT-4o\", \"text\": \"...\" }. At least 2 required.",
      "items": {
        "type": "object",
        "required": [
          "text"
        ],
        "properties": {
          "label": {
            "type": "string",
            "description": "Human name for this candidate (e.g. model id)"
          },
          "text": {
            "type": "string",
            "description": "The produced output"
          }
        }
      }
    },
    "reference": {
      "type": "string",
      "description": "Optional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named."
    }
  },
  "required": [
    "responses"
  ]
}

prompt_test_suite

Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.

Parameters10

system_prompt	string	required	The system prompt under test
user_prompt	string	required	The user prompt to send
expected_behavior	string	optional	Description of what the LLM should do (free text)
expected_format	string	optional	Expected output format
must_include	string	optional	Required content (comma-separated)
must_not_include	string	optional	Forbidden content (comma-separated)
max_tokens	number	optional	Max token budget for the test
temperature	number	optional	Temperature to use
check_safety	boolean	optional	Include safety/PII checks in the rubric
adversarial_prompts	boolean	optional	Auto-generate adversarial test variants (jailbreak, injection, edge cases)

Raw schema

{
  "type": "object",
  "properties": {
    "system_prompt": {
      "type": "string",
      "description": "The system prompt under test"
    },
    "user_prompt": {
      "type": "string",
      "description": "The user prompt to send"
    },
    "expected_behavior": {
      "type": "string",
      "description": "Description of what the LLM should do (free text)"
    },
    "expected_format": {
      "type": "string",
      "description": "Expected output format",
      "enum": [
        "json",
        "markdown",
        "code",
        "plain",
        "any"
      ]
    },
    "must_include": {
      "type": "string",
      "description": "Required content (comma-separated)"
    },
    "must_not_include": {
      "type": "string",
      "description": "Forbidden content (comma-separated)"
    },
    "max_tokens": {
      "type": "number",
      "description": "Max token budget for the test"
    },
    "temperature": {
      "type": "number",
      "description": "Temperature to use"
    },
    "check_safety": {
      "type": "boolean",
      "description": "Include safety/PII checks in the rubric"
    },
    "adversarial_prompts": {
      "type": "boolean",
      "description": "Auto-generate adversarial test variants (jailbreak, injection, edge cases)"
    }
  },
  "required": [
    "system_prompt",
    "user_prompt"
  ]
}

mcp_server_health_check

Generate a health check report for an MCP server's tool manifest. Validates tool definitions, schema quality, naming conventions, and documentation completeness. Paste the server manifest JSON to audit.

Parameters2

manifest	string	required	MCP server manifest JSON (the response from GET /mcp or tools/list)
strict	boolean	optional	Enable strict mode: also check for optional best practices (examples, default values, descriptions > 20 chars)

Raw schema

{
  "type": "object",
  "properties": {
    "manifest": {
      "type": "string",
      "description": "MCP server manifest JSON (the response from GET /mcp or tools/list)"
    },
    "strict": {
      "type": "boolean",
      "description": "Enable strict mode: also check for optional best practices (examples, default values, descriptions > 20 chars)"
    }
  },
  "required": [
    "manifest"
  ]
}

mcp_server_evaluate

Run a full compliance evaluation against a live MCP server URL. Tests: server reachability (ping), manifest discovery (GET /mcp), schema quality (snake_case names, descriptions, inputSchema), JSON-RPC 2.0 test call, and P50/P95 latency. Returns a PASS/FIX/BLOCK verdict with a 0-100 score and per-check details.

Parameters2

url	string	required	Base URL of the MCP server (e.g. https://ia-qa.com or http://localhost:3001)
test_tool_name	string	optional	Specific tool name to use in the JSON-RPC test call (defaults to the first tool in the manifest)

Raw schema

{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "description": "Base URL of the MCP server (e.g. https://ia-qa.com or http://localhost:3001)"
    },
    "test_tool_name": {
      "type": "string",
      "description": "Specific tool name to use in the JSON-RPC test call (defaults to the first tool in the manifest)"
    }
  },
  "required": [
    "url"
  ]
}

json_schema_validate

Validate a JSON value against a JSON Schema (draft-07 subset). Supports type, required, properties, items, enum, const, pattern, format (email/uri/date), minimum/maximum, minLength/maxLength, minItems/maxItems, uniqueItems, additionalProperties, anyOf, allOf, oneOf. Returns all validation errors with dot-notation paths.

Parameters2

value	string	required	JSON string to validate
schema	string	required	JSON Schema as a JSON string

Raw schema

{
  "type": "object",
  "properties": {
    "value": {
      "type": "string",
      "description": "JSON string to validate"
    },
    "schema": {
      "type": "string",
      "description": "JSON Schema as a JSON string"
    }
  },
  "required": [
    "value",
    "schema"
  ]
}

flatten_json

Flatten a nested JSON object to single-level dot-notation keys (e.g. {"a":{"b":1}} → {"a.b":1}), or unflatten dot-notation keys back to a nested object. Supports custom separators.

Parameters3

input	string	required	JSON string to flatten or unflatten
mode	string	optional	"flatten" (default) or "unflatten"
separator	string	optional	Key separator (default: ".")

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "JSON string to flatten or unflatten"
    },
    "mode": {
      "type": "string",
      "description": "\"flatten\" (default) or \"unflatten\""
    },
    "separator": {
      "type": "string",
      "description": "Key separator (default: \".\")"
    }
  },
  "required": [
    "input"
  ]
}

xml_to_json

Convert an XML string to a JSON object. Supports attributes, nested elements, arrays, CDATA, and namespaces. Options: parse numbers, parse booleans, ignore attributes.

Parameters4

input	string	required	XML string to convert
parse_values	boolean	optional	Auto-parse numbers and booleans (default: true)
ignore_attrs	boolean	optional	Ignore XML attributes (default: false)
attr_prefix	string	optional	Prefix for attribute keys (default: "@_")

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "XML string to convert"
    },
    "parse_values": {
      "type": "boolean",
      "description": "Auto-parse numbers and booleans (default: true)"
    },
    "ignore_attrs": {
      "type": "boolean",
      "description": "Ignore XML attributes (default: false)"
    },
    "attr_prefix": {
      "type": "string",
      "description": "Prefix for attribute keys (default: \"@_\")"
    }
  },
  "required": [
    "input"
  ]
}

redact_pii

Automatically detect and redact Personally Identifiable Information (PII) from text. Replaces emails, phone numbers, SSNs, credit cards, IP addresses, and JWT tokens with [REDACTED_TYPE] placeholders. Safe to use before logging or sending to an LLM.

Parameters3

input	string	required	Text to redact PII from
types	string	optional	Comma-separated types to redact (default: all). Options: email, phone, ssn, credit_card, ip_address, jwt
marker	string	optional	Custom replacement marker (default: "REDACTED"). Result: [REDACTED_EMAIL]

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to redact PII from"
    },
    "types": {
      "type": "string",
      "description": "Comma-separated types to redact (default: all). Options: email, phone, ssn, credit_card, ip_address, jwt"
    },
    "marker": {
      "type": "string",
      "description": "Custom replacement marker (default: \"REDACTED\"). Result: [REDACTED_EMAIL]"
    }
  },
  "required": [
    "input"
  ]
}

mock_from_schema

Generate realistic mock data from a JSON Schema. Supports all common types (string, number, integer, boolean, array, object, null), format hints (email, date, date-time, uri, uuid), enum, const, and nested schemas. Perfect for testing MCP tools with realistic data.

Parameters3

schema	string	required	JSON Schema as a JSON string
count	number	optional	Number of mock objects to generate (default: 1, max: 20)
seed	string	optional	Optional seed string for deterministic output (uses first char codes)

Raw schema

{
  "type": "object",
  "properties": {
    "schema": {
      "type": "string",
      "description": "JSON Schema as a JSON string"
    },
    "count": {
      "type": "number",
      "description": "Number of mock objects to generate (default: 1, max: 20)"
    },
    "seed": {
      "type": "string",
      "description": "Optional seed string for deterministic output (uses first char codes)"
    }
  },
  "required": [
    "schema"
  ]
}

transform_json_array

Transform a JSON array using common operations: pluck (extract specific fields), filter (by field value), sort_by (field), group_by (field), count_by (field), uniq_by (field). Useful for processing MCP tool results and LLM structured outputs.

Parameters9

input	string	required	JSON string containing an array (or object with an array at path)
path	string	optional	Optional dot-notation path to the array within the JSON object (e.g. "data.items")
operation	string	required	Operation: "pluck", "filter", "sort_by", "group_by", "count_by", "uniq_by", "reverse", "first_n", "last_n", "flatten"
field	string	optional	Field to operate on (for sort_by, group_by, count_by, uniq_by, filter)
fields	string	optional	Comma-separated field list for "pluck" (e.g. "id,name,email")
filter_op	string	optional	For "filter": "==" \| "!=" \| ">" \| ">=" \| "<" \| "<=" \| "contains" \| "exists" \| "!exists"
filter_value	string	optional	For "filter": value to compare against
n	number	optional	For first_n / last_n: number of items
sort_order	string	optional	For sort_by: "asc" (default) or "desc"

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "JSON string containing an array (or object with an array at path)"
    },
    "path": {
      "type": "string",
      "description": "Optional dot-notation path to the array within the JSON object (e.g. \"data.items\")"
    },
    "operation": {
      "type": "string",
      "description": "Operation: \"pluck\", \"filter\", \"sort_by\", \"group_by\", \"count_by\", \"uniq_by\", \"reverse\", \"first_n\", \"last_n\", \"flatten\""
    },
    "field": {
      "type": "string",
      "description": "Field to operate on (for sort_by, group_by, count_by, uniq_by, filter)"
    },
    "fields": {
      "type": "string",
      "description": "Comma-separated field list for \"pluck\" (e.g. \"id,name,email\")"
    },
    "filter_op": {
      "type": "string",
      "description": "For \"filter\": \"==\" | \"!=\" | \">\" | \">=\" | \"<\" | \"<=\" | \"contains\" | \"exists\" | \"!exists\""
    },
    "filter_value": {
      "type": "string",
      "description": "For \"filter\": value to compare against"
    },
    "n": {
      "type": "number",
      "description": "For first_n / last_n: number of items"
    },
    "sort_order": {
      "type": "string",
      "description": "For sort_by: \"asc\" (default) or \"desc\""
    }
  },
  "required": [
    "input",
    "operation"
  ]
}

json_to_csv

Convert a JSON array of objects to CSV format. Automatically detects columns from all object keys. Handles quoting and escaping per RFC 4180.

Parameters3

input	string	required	JSON string containing an array of objects
delimiter	string	optional	Column delimiter (default: ",")
headers	boolean	optional	Include header row (default: true)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "JSON string containing an array of objects"
    },
    "delimiter": {
      "type": "string",
      "description": "Column delimiter (default: \",\")"
    },
    "headers": {
      "type": "boolean",
      "description": "Include header row (default: true)"
    }
  },
  "required": [
    "input"
  ]
}

case_convert

Convert a string between naming conventions: camelCase, PascalCase, snake_case, kebab-case, UPPER_SNAKE_CASE, dot.case, Title Case. Essential for code generation and refactoring.

Parameters2

input	string	required	String to convert (e.g., "myVariableName", "my-css-class")
to	string	required	Target case: "camel", "pascal", "snake", "kebab", "upper_snake", "dot", "title"

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "String to convert (e.g., \"myVariableName\", \"my-css-class\")"
    },
    "to": {
      "type": "string",
      "description": "Target case: \"camel\", \"pascal\", \"snake\", \"kebab\", \"upper_snake\", \"dot\", \"title\""
    }
  },
  "required": [
    "input",
    "to"
  ]
}

sort_lines

Sort, deduplicate, reverse, or filter lines of text. Useful for cleaning import lists, dependencies, log files, and config entries.

Parameters5

input	string	required	Multi-line text to process
operation	string	optional	"sort" (default), "sort_desc", "reverse", "deduplicate", "unique_sort", "filter"
filter	string	optional	For "filter": keep lines containing this substring (case-insensitive)
trim	boolean	optional	Trim whitespace from each line (default: true)
remove_empty	boolean	optional	Remove empty lines (default: true)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Multi-line text to process"
    },
    "operation": {
      "type": "string",
      "description": "\"sort\" (default), \"sort_desc\", \"reverse\", \"deduplicate\", \"unique_sort\", \"filter\""
    },
    "filter": {
      "type": "string",
      "description": "For \"filter\": keep lines containing this substring (case-insensitive)"
    },
    "trim": {
      "type": "boolean",
      "description": "Trim whitespace from each line (default: true)"
    },
    "remove_empty": {
      "type": "boolean",
      "description": "Remove empty lines (default: true)"
    }
  },
  "required": [
    "input"
  ]
}

number_base_convert

Convert numbers between bases: decimal, binary, octal, hexadecimal, or any base 2–36. Auto-detects 0x, 0b, 0o prefixes.

Parameters3

input	string	required	Number to convert (e.g., "255", "0xFF", "0b1010", "0o77")
from_base	number	optional	Source base 2–36 (auto-detects prefix if omitted)
to_base	number	optional	Target base 2–36 (omit to get all common bases)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Number to convert (e.g., \"255\", \"0xFF\", \"0b1010\", \"0o77\")"
    },
    "from_base": {
      "type": "number",
      "description": "Source base 2–36 (auto-detects prefix if omitted)"
    },
    "to_base": {
      "type": "number",
      "description": "Target base 2–36 (omit to get all common bases)"
    }
  },
  "required": [
    "input"
  ]
}

validate_url

Parse and validate a URL. Returns decomposed components: protocol, hostname, port, path, query parameters, hash, and origin.

Parameters1

input

string

required

URL to validate and parse

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "URL to validate and parse"
    }
  },
  "required": [
    "input"
  ]
}

check_contrast_ratio

Calculate WCAG 2.1 contrast ratio between two colors. Returns ratio and compliance for AA/AAA normal and large text.

Parameters2

foreground	string	required	Foreground color in hex (e.g., "#333333")
background	string	required	Background color in hex (e.g., "#ffffff")

Raw schema

{
  "type": "object",
  "properties": {
    "foreground": {
      "type": "string",
      "description": "Foreground color in hex (e.g., \"#333333\")"
    },
    "background": {
      "type": "string",
      "description": "Background color in hex (e.g., \"#ffffff\")"
    }
  },
  "required": [
    "foreground",
    "background"
  ]
}

html_to_markdown

Convert HTML to clean Markdown. Strips scripts, styles, nav, ads, and comments. Converts headings, lists, links, images, code blocks. Ideal for preparing web content as LLM context.

Parameters2

input	string	required	HTML string to convert
strip_links	boolean	optional	Strip link URLs, keep text only (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "HTML string to convert"
    },
    "strip_links": {
      "type": "boolean",
      "description": "Strip link URLs, keep text only (default: false)"
    }
  },
  "required": [
    "input"
  ]
}

cron_parse

Parse a cron expression into a human-readable schedule description. Supports standard 5-field cron (minute hour day month weekday).

Parameters1

expression

string

required

Cron expression (e.g., "0 9 * * 1-5", "*/15 * * * *")

Raw schema

{
  "type": "object",
  "properties": {
    "expression": {
      "type": "string",
      "description": "Cron expression (e.g., \"0 9 * * 1-5\", \"*/15 * * * *\")"
    }
  },
  "required": [
    "expression"
  ]
}

cron_validator

Validate a 5-field cron expression, explain the schedule, and preview the next execution times. Use this to debug cron jobs before they reach production. Returns parsed fields, a human-readable description, and upcoming ISO timestamps.

Parameters2

expression	string	required	Cron expression with 5 fields, e.g. "/15 9-18 * 1-5"
next_runs_count	number	optional	How many upcoming runs to return (1-50, default: 10)

Raw schema

{
  "type": "object",
  "properties": {
    "expression": {
      "type": "string",
      "description": "Cron expression with 5 fields, e.g. \"*/15 9-18 * * 1-5\""
    },
    "next_runs_count": {
      "type": "number",
      "description": "How many upcoming runs to return (1-50, default: 10)"
    }
  },
  "required": [
    "expression"
  ]
}

calculate_readability

Calculate readability scores: Flesch Reading Ease, Flesch-Kincaid Grade Level, Coleman-Liau Index, and Automated Readability Index. Useful for evaluating LLM output quality.

Parameters1

input

string

required

Text to analyze for readability

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to analyze for readability"
    }
  },
  "required": [
    "input"
  ]
}

normalize_whitespace

Normalize whitespace: trim trailing spaces, collapse blank lines, normalize line endings (LF/CRLF), convert tabs to spaces. Useful for cleaning code, configs, and text before processing.

Parameters6

input	string	required	Text to normalize
line_ending	string	optional	"lf" (default), "crlf", or "cr"
collapse_blanks	boolean	optional	Collapse 3+ consecutive blank lines to 2 (default: true)
trim_lines	boolean	optional	Trim trailing whitespace from each line (default: true)
trim_file	boolean	optional	Trim leading/trailing blank lines (default: true)
tab_to_spaces	number	optional	Convert tabs to N spaces (omit to keep tabs)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to normalize"
    },
    "line_ending": {
      "type": "string",
      "description": "\"lf\" (default), \"crlf\", or \"cr\""
    },
    "collapse_blanks": {
      "type": "boolean",
      "description": "Collapse 3+ consecutive blank lines to 2 (default: true)"
    },
    "trim_lines": {
      "type": "boolean",
      "description": "Trim trailing whitespace from each line (default: true)"
    },
    "trim_file": {
      "type": "boolean",
      "description": "Trim leading/trailing blank lines (default: true)"
    },
    "tab_to_spaces": {
      "type": "number",
      "description": "Convert tabs to N spaces (omit to keep tabs)"
    }
  },
  "required": [
    "input"
  ]
}

embedding_similarity

Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for true semantic similarity with vector embeddings, use run_semantic_tests with mode="embeddings" and your OpenAI API key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.

Parameters4

text_a	string	optional	First text to compare (single-pair mode)
text_b	string	optional	Second text to compare (single-pair mode)
batch	array	optional	Batch mode: array of { text_a, text_b } pairs. Overrides text_a/text_b if provided.
methods	array	optional	Algorithms to use (default: all three). Options: "bow", "tfidf", "ngram"

Raw schema

{
  "type": "object",
  "properties": {
    "text_a": {
      "type": "string",
      "description": "First text to compare (single-pair mode)"
    },
    "text_b": {
      "type": "string",
      "description": "Second text to compare (single-pair mode)"
    },
    "batch": {
      "type": "array",
      "description": "Batch mode: array of { text_a, text_b } pairs. Overrides text_a/text_b if provided.",
      "items": {
        "type": "object",
        "required": [
          "text_a",
          "text_b"
        ],
        "properties": {
          "text_a": {
            "type": "string"
          },
          "text_b": {
            "type": "string"
          }
        }
      }
    },
    "methods": {
      "type": "array",
      "description": "Algorithms to use (default: all three). Options: \"bow\", \"tfidf\", \"ngram\"",
      "items": {
        "type": "string",
        "enum": [
          "bow",
          "tfidf",
          "ngram"
        ]
      }
    }
  }
}

llm_format_check

Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.

Parameters3

output	string	required	The LLM output to validate
expected_format	string	required	Expected format
regex_pattern	string	optional	Custom regex pattern (only when expected_format is "regex")

Raw schema

{
  "type": "object",
  "properties": {
    "output": {
      "type": "string",
      "description": "The LLM output to validate"
    },
    "expected_format": {
      "type": "string",
      "description": "Expected format",
      "enum": [
        "json",
        "markdown_heading",
        "code_block",
        "bullet_list",
        "numbered_list",
        "table",
        "yaml",
        "xml",
        "regex"
      ]
    },
    "regex_pattern": {
      "type": "string",
      "description": "Custom regex pattern (only when expected_format is \"regex\")"
    }
  },
  "required": [
    "output",
    "expected_format"
  ]
}

hallucination_check

Word-overlap based hallucination check: verifies if an LLM answer's words and numbers appear in the provided source/context. Fast, deterministic, no API key needed. Limitations: not semantic — does not understand synonyms or paraphrases. For true semantic grounding, use run_semantic_tests with embedding mode. Essential for quick RAG accuracy testing.

Parameters3

answer	string	required	The LLM-generated answer to verify
context	string	required	The source/reference text that should ground the answer
strict	boolean	optional	If true, every sentence in the answer must be supported (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "answer": {
      "type": "string",
      "description": "The LLM-generated answer to verify"
    },
    "context": {
      "type": "string",
      "description": "The source/reference text that should ground the answer"
    },
    "strict": {
      "type": "boolean",
      "description": "If true, every sentence in the answer must be supported (default: false)"
    }
  },
  "required": [
    "answer",
    "context"
  ]
}

prompt_injection_scan

Scan user input or prompts for common prompt injection patterns. Detects system prompt overrides, jailbreak attempts, role manipulation, encoding tricks, delimiter attacks, template/interpolation injection ({{...}}, ${...}), and context-exfiltration attempts ("repeat everything above").

Parameters2

input	string	required	The user input or prompt to scan for injection patterns
sensitivity	string	optional	Detection sensitivity (default: medium)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "The user input or prompt to scan for injection patterns"
    },
    "sensitivity": {
      "type": "string",
      "enum": [
        "low",
        "medium",
        "high"
      ],
      "description": "Detection sensitivity (default: medium)"
    }
  },
  "required": [
    "input"
  ]
}

token_budget_calculator

Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds model context window. Supports 25+ models.

Parameters8

model	string	required	Model name (e.g. gpt-4o, claude-3.5-sonnet, gemini-2.0-flash)
system_prompt_tokens	number	optional	Token count for system prompt
user_input_tokens	number	optional	Token count for user message
context_tokens	number	optional	Token count for RAG context / documents
expected_output_tokens	number	optional	Expected max output tokens
system_prompt	string	optional	Actual system prompt text (will estimate tokens)
user_input	string	optional	Actual user input text (will estimate tokens)
context	string	optional	Actual context text (will estimate tokens)

Raw schema

{
  "type": "object",
  "properties": {
    "model": {
      "type": "string",
      "description": "Model name (e.g. gpt-4o, claude-3.5-sonnet, gemini-2.0-flash)"
    },
    "system_prompt_tokens": {
      "type": "number",
      "description": "Token count for system prompt"
    },
    "user_input_tokens": {
      "type": "number",
      "description": "Token count for user message"
    },
    "context_tokens": {
      "type": "number",
      "description": "Token count for RAG context / documents"
    },
    "expected_output_tokens": {
      "type": "number",
      "description": "Expected max output tokens"
    },
    "system_prompt": {
      "type": "string",
      "description": "Actual system prompt text (will estimate tokens)"
    },
    "user_input": {
      "type": "string",
      "description": "Actual user input text (will estimate tokens)"
    },
    "context": {
      "type": "string",
      "description": "Actual context text (will estimate tokens)"
    }
  },
  "required": [
    "model"
  ]
}

consistency_check

Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. For true semantic consistency, use run_semantic_tests with embedding mode. Essential for determinism testing.

Parameters2

responses	array	required	Array of 2+ LLM responses to compare (same prompt, different runs)
check_facts	boolean	optional	Check for contradictory numbers/facts across responses (default: true)

Raw schema

{
  "type": "object",
  "properties": {
    "responses": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of 2+ LLM responses to compare (same prompt, different runs)"
    },
    "check_facts": {
      "type": "boolean",
      "description": "Check for contradictory numbers/facts across responses (default: true)"
    }
  },
  "required": [
    "responses"
  ]
}

llm_json_schema_check

Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.

Parameters2

output	string	required	The LLM JSON output (raw string, will be parsed)
schema	object	required	JSON Schema (draft-07 subset) to validate against

Raw schema

{
  "type": "object",
  "properties": {
    "output": {
      "type": "string",
      "description": "The LLM JSON output (raw string, will be parsed)"
    },
    "schema": {
      "type": "object",
      "additionalProperties": true,
      "description": "JSON Schema (draft-07 subset) to validate against"
    }
  },
  "required": [
    "output",
    "schema"
  ]
}

latency_benchmark

Measure response time of one or more HTTP endpoints (GET/POST). Runs N iterations and returns min/max/avg/p95 latency. Useful for API and MCP server benchmarking.

Parameters2

endpoints	string \| array	required	Endpoints to benchmark. Accepts a single URL string, an array of URL strings, or an array of {url, method?, body?, headers?, label?} objects.
iterations	number	optional	Number of iterations per endpoint (default: 3, max: 10)

Raw schema

{
  "type": "object",
  "properties": {
    "endpoints": {
      "type": [
        "string",
        "array"
      ],
      "items": {
        "type": "object",
        "properties": {
          "url": {
            "type": "string",
            "description": "Full URL to test"
          },
          "method": {
            "type": "string",
            "enum": [
              "GET",
              "POST"
            ],
            "description": "HTTP method (default: GET)"
          },
          "body": {
            "type": "object",
            "additionalProperties": true,
            "description": "Request body for POST"
          },
          "headers": {
            "type": "object",
            "additionalProperties": true,
            "description": "Custom headers"
          },
          "label": {
            "type": "string",
            "description": "Optional label for this endpoint"
          }
        }
      },
      "description": "Endpoints to benchmark. Accepts a single URL string, an array of URL strings, or an array of {url, method?, body?, headers?, label?} objects."
    },
    "iterations": {
      "type": "number",
      "description": "Number of iterations per endpoint (default: 3, max: 10)"
    }
  },
  "required": [
    "endpoints"
  ]
}

response_quality_score

Score an LLM response on multiple quality dimensions: relevance, completeness, clarity, conciseness, formatting. Returns a weighted 0-100 score with detailed breakdown.

Parameters4

question	string	required	The original question/prompt
response	string	required	The LLM response to score
expected_keywords	array	optional	Keywords that should appear in a good answer
max_length	number	optional	Ideal max character length (penalize if exceeded)

Raw schema

{
  "type": "object",
  "properties": {
    "question": {
      "type": "string",
      "description": "The original question/prompt"
    },
    "response": {
      "type": "string",
      "description": "The LLM response to score"
    },
    "expected_keywords": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Keywords that should appear in a good answer"
    },
    "max_length": {
      "type": "number",
      "description": "Ideal max character length (penalize if exceeded)"
    }
  },
  "required": [
    "question",
    "response"
  ]
}

rag_relevance_rank

Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.

Parameters3

query	string	required	The user query
chunks	array	required	Array of text chunks to rank
top_k	number	optional	Return top K results (default: all)

Raw schema

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "The user query"
    },
    "chunks": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of text chunks to rank"
    },
    "top_k": {
      "type": "number",
      "description": "Return top K results (default: all)"
    }
  },
  "required": [
    "query",
    "chunks"
  ]
}

toxicity_scan

Scan text for toxic language, bias indicators, profanity, and harmful content categories. Returns risk scores per category. Useful for LLM safety guardrail testing.

Parameters2

text	string	required	Text to scan
categories	array	optional	Categories to check (default: all)

Raw schema

{
  "type": "object",
  "properties": {
    "text": {
      "type": "string",
      "description": "Text to scan"
    },
    "categories": {
      "type": "array",
      "items": {
        "type": "string",
        "enum": [
          "profanity",
          "hate_speech",
          "violence",
          "sexual",
          "self_harm",
          "bias"
        ]
      },
      "description": "Categories to check (default: all)"
    }
  },
  "required": [
    "text"
  ]
}

guardrail_test

Test an LLM response against a set of guardrail rules: must-include, must-not-include, max length, required format, language, forbidden patterns, and custom regex. Returns pass/fail per rule.

Parameters2

response	string	required	The LLM response to test
rules	array	required	Array of guardrail rules to check

Raw schema

{
  "type": "object",
  "properties": {
    "response": {
      "type": "string",
      "description": "The LLM response to test"
    },
    "rules": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string",
            "enum": [
              "must_include",
              "must_not_include",
              "max_length",
              "min_length",
              "format",
              "regex_match",
              "regex_not_match",
              "starts_with",
              "ends_with",
              "word_count_max",
              "word_count_min"
            ]
          },
          "value": {
            "type": "string",
            "description": "Value for the rule (text, number as string, regex, or format name)"
          },
          "label": {
            "type": "string",
            "description": "Optional human-readable label"
          }
        }
      },
      "description": "Array of guardrail rules to check"
    }
  },
  "required": [
    "response",
    "rules"
  ]
}

function_call_validate

Validate an LLM function call / tool_use output: check that function name is in allowed list, arguments match expected schema, no extra/missing args. For OpenAI function calling & MCP tool_use testing.

Parameters2

function_call	object	required	The function call object from LLM (e.g. { "name": "get_weather", "arguments": {"city":"Paris"} })
allowed_functions	array	required	List of allowed function definitions

Raw schema

{
  "type": "object",
  "properties": {
    "function_call": {
      "type": "object",
      "additionalProperties": true,
      "description": "The function call object from LLM (e.g. { \"name\": \"get_weather\", \"arguments\": {\"city\":\"Paris\"} })"
    },
    "allowed_functions": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string"
          },
          "required_args": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "optional_args": {
            "type": "array",
            "items": {
              "type": "string"
            }
          }
        }
      },
      "description": "List of allowed function definitions"
    }
  },
  "required": [
    "function_call",
    "allowed_functions"
  ]
}

conversation_analyze

Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.

Parameters1

messages

array

required

Conversation messages in order

Raw schema

{
  "type": "object",
  "properties": {
    "messages": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "role": {
            "type": "string",
            "enum": [
              "system",
              "user",
              "assistant"
            ]
          },
          "content": {
            "type": "string"
          }
        }
      },
      "description": "Conversation messages in order"
    }
  },
  "required": [
    "messages"
  ]
}

mcp_schema_lint

Lint an MCP tool definition for best practices: naming conventions, description quality, schema completeness, required fields consistency, description length. Returns actionable warnings.

Parameters1

tool_definition

object

required

MCP tool definition object with name, description, inputSchema

Raw schema

{
  "type": "object",
  "properties": {
    "tool_definition": {
      "type": "object",
      "additionalProperties": true,
      "description": "MCP tool definition object with name, description, inputSchema"
    }
  },
  "required": [
    "tool_definition"
  ]
}

cot_analyzer

Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.

Parameters2

reasoning	string	required	The CoT / reasoning trace text (e.g. from <think> tags or step-by-step output)
expected_conclusion	string	optional	Expected final answer to check against (optional)

Raw schema

{
  "type": "object",
  "properties": {
    "reasoning": {
      "type": "string",
      "description": "The CoT / reasoning trace text (e.g. from <think> tags or step-by-step output)"
    },
    "expected_conclusion": {
      "type": "string",
      "description": "Expected final answer to check against (optional)"
    }
  },
  "required": [
    "reasoning"
  ]
}

ab_test_report

Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns statistical comparison: mean, median, std deviation, winner, and improvement percentage.

Parameters2

variant_a	object	required	First variant configuration with name and score array
variant_b	object	required	Second variant configuration with name and score array

Raw schema

{
  "type": "object",
  "properties": {
    "variant_a": {
      "type": "object",
      "description": "First variant configuration with name and score array",
      "properties": {
        "name": {
          "type": "string",
          "description": "Name/label for variant A"
        },
        "scores": {
          "type": "array",
          "items": {
            "type": "number"
          },
          "description": "Array of scores (0-100)"
        }
      }
    },
    "variant_b": {
      "type": "object",
      "description": "Second variant configuration with name and score array",
      "properties": {
        "name": {
          "type": "string",
          "description": "Name/label for variant B"
        },
        "scores": {
          "type": "array",
          "items": {
            "type": "number"
          },
          "description": "Array of scores (0-100)"
        }
      }
    }
  },
  "required": [
    "variant_a",
    "variant_b"
  ]
}

context_window_check

Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.

Parameters3

messages	array	required	Array of messages (system/user/assistant)
model	string	required	Target model name (e.g. gpt-4o, claude-3.5-sonnet)
max_output_tokens	number	optional	Reserved tokens for output (default: 4096)

Raw schema

{
  "type": "object",
  "properties": {
    "messages": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "role": {
            "type": "string"
          },
          "content": {
            "type": "string"
          }
        }
      },
      "description": "Array of messages (system/user/assistant)"
    },
    "model": {
      "type": "string",
      "description": "Target model name (e.g. gpt-4o, claude-3.5-sonnet)"
    },
    "max_output_tokens": {
      "type": "number",
      "description": "Reserved tokens for output (default: 4096)"
    }
  },
  "required": [
    "messages",
    "model"
  ]
}

vector_similarity

Compute similarity/distance between two float vectors: cosine similarity, dot product, Euclidean and Manhattan distance. Essential for vector DB relevance scoring, embedding evaluation, and nearest-neighbor testing.

Parameters3

vector_a	array	required	First vector as array of floats
vector_b	array	required	Second vector as array of floats
metric	string	optional	Distance metric (default: all)

Raw schema

{
  "type": "object",
  "properties": {
    "vector_a": {
      "type": "array",
      "items": {
        "type": "number"
      },
      "description": "First vector as array of floats"
    },
    "vector_b": {
      "type": "array",
      "items": {
        "type": "number"
      },
      "description": "Second vector as array of floats"
    },
    "metric": {
      "type": "string",
      "enum": [
        "cosine",
        "dot_product",
        "euclidean",
        "manhattan",
        "all"
      ],
      "description": "Distance metric (default: all)"
    }
  },
  "required": [
    "vector_a",
    "vector_b"
  ]
}

normalize_vector

L2-normalize a float vector (produce a unit vector with norm=1). Required by many vector DBs (Pinecone, Qdrant cosine). Supports batch normalization of up to 1000 vectors.

Parameters2

vector	array	optional	Single vector to normalize
batch	array	optional	Batch of vectors to normalize (overrides vector)

Raw schema

{
  "type": "object",
  "properties": {
    "vector": {
      "type": "array",
      "items": {
        "type": "number"
      },
      "description": "Single vector to normalize"
    },
    "batch": {
      "type": "array",
      "items": {
        "type": "array",
        "items": {
          "type": "number"
        }
      },
      "description": "Batch of vectors to normalize (overrides vector)"
    }
  }
}

vector_quantize

Simulate int8 or int4 quantization of float32 embedding vectors. Reduces storage by 4x (int8) or 8x (int4). Returns quantized values, scale factor, and precision loss (MSE). Useful for understanding vector DB compression trade-offs.

Parameters2

vector	array	required	Float32 vector to quantize
bits	number	optional	Quantization bits: 8 (int8, default) or 4 (int4)

Raw schema

{
  "type": "object",
  "properties": {
    "vector": {
      "type": "array",
      "items": {
        "type": "number"
      },
      "description": "Float32 vector to quantize"
    },
    "bits": {
      "type": "number",
      "description": "Quantization bits: 8 (int8, default) or 4 (int4)"
    }
  },
  "required": [
    "vector"
  ]
}

vector_stats

Compute statistics for a float vector or matrix of vectors: mean, std, L2 norm, min, max, sparsity, top-K indices. Useful for debugging embedding quality and analyzing vector distributions in a vector DB.

Parameters3

vector	array	optional	Single vector to analyze
matrix	array	optional	Matrix of vectors (overrides vector). Returns per-vector + matrix-level stats.
top_k	number	optional	Return indices of top K absolute values (default: 5)

Raw schema

{
  "type": "object",
  "properties": {
    "vector": {
      "type": "array",
      "items": {
        "type": "number"
      },
      "description": "Single vector to analyze"
    },
    "matrix": {
      "type": "array",
      "items": {
        "type": "array",
        "items": {
          "type": "number"
        }
      },
      "description": "Matrix of vectors (overrides vector). Returns per-vector + matrix-level stats."
    },
    "top_k": {
      "type": "number",
      "description": "Return indices of top K absolute values (default: 5)"
    }
  }
}

bm25_score

Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.

Parameters5

query	string	required	The search query
documents	array	required	Array of documents to rank
k1	number	optional	Term frequency saturation (default: 1.5)
b	number	optional	Length normalization factor (default: 0.75)
top_k	number	optional	Return top K results (default: all)

Raw schema

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "The search query"
    },
    "documents": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of documents to rank"
    },
    "k1": {
      "type": "number",
      "description": "Term frequency saturation (default: 1.5)"
    },
    "b": {
      "type": "number",
      "description": "Length normalization factor (default: 0.75)"
    },
    "top_k": {
      "type": "number",
      "description": "Return top K results (default: all)"
    }
  },
  "required": [
    "query",
    "documents"
  ]
}

build_rag_prompt

Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.

Parameters6

query	string	required	The user question to answer
chunks	array	required	Retrieved context chunks with .text (required), .source (optional), .score (optional)
system_instruction	string	optional	Custom system instruction (default: standard RAG grounding instruction)
max_context_tokens	number	optional	Max tokens for context section (default: 2000)
cite_sources	boolean	optional	Add [1], [2] citation numbers (default: true)
language	string	optional	Response language instruction (e.g. "French", "Spanish")

Raw schema

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "The user question to answer"
    },
    "chunks": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "text": {
            "type": "string"
          },
          "source": {
            "type": "string"
          },
          "score": {
            "type": "number"
          }
        }
      },
      "description": "Retrieved context chunks with .text (required), .source (optional), .score (optional)"
    },
    "system_instruction": {
      "type": "string",
      "description": "Custom system instruction (default: standard RAG grounding instruction)"
    },
    "max_context_tokens": {
      "type": "number",
      "description": "Max tokens for context section (default: 2000)"
    },
    "cite_sources": {
      "type": "boolean",
      "description": "Add [1], [2] citation numbers (default: true)"
    },
    "language": {
      "type": "string",
      "description": "Response language instruction (e.g. \"French\", \"Spanish\")"
    }
  },
  "required": [
    "query",
    "chunks"
  ]
}

prompt_template_fill

Fill a prompt template with variables. Supports {{variable}} syntax and {{#if key}}...{{/if}} conditional blocks. Returns the filled prompt and lists unfilled variables.

Parameters3

template	string	required	Prompt template with {{variable}} placeholders
variables	object	optional	Key-value pairs to fill (e.g. {"name":"Alice","role":"engineer"})
strict	boolean	optional	Throw error if any variable is not provided (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "template": {
      "type": "string",
      "description": "Prompt template with {{variable}} placeholders"
    },
    "variables": {
      "type": "object",
      "additionalProperties": true,
      "description": "Key-value pairs to fill (e.g. {\"name\":\"Alice\",\"role\":\"engineer\"})"
    },
    "strict": {
      "type": "boolean",
      "description": "Throw error if any variable is not provided (default: false)"
    }
  },
  "required": [
    "template"
  ]
}

few_shot_formatter

Format few-shot examples for LLM prompts. Converts example pairs into formatted blocks. Supports chat format (User/Assistant), XML tags, Markdown, or plain text.

Parameters4

examples	array	required	Array of {input, output} pairs
format	string	optional	Output format (default: chat)
input_label	string	optional	Label for input (default: User / <input>)
output_label	string	optional	Label for output (default: Assistant / <output>)

Raw schema

{
  "type": "object",
  "properties": {
    "examples": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "input": {
            "type": "string"
          },
          "output": {
            "type": "string"
          },
          "label": {
            "type": "string"
          }
        }
      },
      "description": "Array of {input, output} pairs"
    },
    "format": {
      "type": "string",
      "enum": [
        "chat",
        "xml",
        "markdown",
        "plain"
      ],
      "description": "Output format (default: chat)"
    },
    "input_label": {
      "type": "string",
      "description": "Label for input (default: User / <input>)"
    },
    "output_label": {
      "type": "string",
      "description": "Label for output (default: Assistant / <output>)"
    }
  },
  "required": [
    "examples"
  ]
}

system_prompt_builder

Build a structured system prompt from components: role, task, constraints, output format, tone, language, and examples. Generates a production-ready system prompt with token estimate.

Parameters7

role	string	required	Role/persona (e.g. "Senior QA Engineer", "JSON extraction assistant")
task	string	optional	Main task or objective
constraints	array	optional	Rules and constraints to follow
output_format	string	optional	Expected output format description
tone	string	optional	Communication tone
language	string	optional	Response language (e.g. "French")
examples	string	optional	Brief examples to include

Raw schema

{
  "type": "object",
  "properties": {
    "role": {
      "type": "string",
      "description": "Role/persona (e.g. \"Senior QA Engineer\", \"JSON extraction assistant\")"
    },
    "task": {
      "type": "string",
      "description": "Main task or objective"
    },
    "constraints": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Rules and constraints to follow"
    },
    "output_format": {
      "type": "string",
      "description": "Expected output format description"
    },
    "tone": {
      "type": "string",
      "enum": [
        "professional",
        "friendly",
        "concise",
        "technical",
        "educational"
      ],
      "description": "Communication tone"
    },
    "language": {
      "type": "string",
      "description": "Response language (e.g. \"French\")"
    },
    "examples": {
      "type": "string",
      "description": "Brief examples to include"
    }
  },
  "required": [
    "role"
  ]
}

model_info

Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.

Parameters1

model

string

required

Model name (e.g. "gpt-4o", "claude-3.5-sonnet", "gemini-2.5-pro")

Raw schema

{
  "type": "object",
  "properties": {
    "model": {
      "type": "string",
      "description": "Model name (e.g. \"gpt-4o\", \"claude-3.5-sonnet\", \"gemini-2.5-pro\")"
    }
  },
  "required": [
    "model"
  ]
}

compare_models

Compare 2-5 AI models side by side: context window, pricing, multimodal, reasoning capabilities, and provider. Returns a comparison table with a recommendation based on your use case.

Parameters2

models	array	required	Array of 2-5 model names (e.g. ["gpt-4o","claude-3.5-sonnet","gemini-2.0-flash"])
use_case	string	optional	Optimize recommendation for this criterion

Raw schema

{
  "type": "object",
  "properties": {
    "models": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of 2-5 model names (e.g. [\"gpt-4o\",\"claude-3.5-sonnet\",\"gemini-2.0-flash\"])"
    },
    "use_case": {
      "type": "string",
      "enum": [
        "cost",
        "context",
        "reasoning",
        "multimodal",
        "speed"
      ],
      "description": "Optimize recommendation for this criterion"
    }
  },
  "required": [
    "models"
  ]
}

http_status_lookup

Look up detailed information about any HTTP status code: class, name, description, cacheability, typical causes, and handling best practices. Covers all standard 1xx-5xx codes.

Parameters1

code

number

required

HTTP status code (e.g. 200, 404, 429, 503)

Raw schema

{
  "type": "object",
  "properties": {
    "code": {
      "type": "number",
      "description": "HTTP status code (e.g. 200, 404, 429, 503)"
    }
  },
  "required": [
    "code"
  ]
}

parse_http_headers

Parse a raw HTTP headers block into a structured JSON object. Detects multi-value headers, masks Authorization values, and optionally audits for missing security headers (HSTS, CSP, X-Frame-Options, etc.).

Parameters2

headers	string	required	Raw HTTP headers (one "Name: Value" per line)
analyze_security	boolean	optional	Audit for missing security headers (default: true)

Raw schema

{
  "type": "object",
  "properties": {
    "headers": {
      "type": "string",
      "description": "Raw HTTP headers (one \"Name: Value\" per line)"
    },
    "analyze_security": {
      "type": "boolean",
      "description": "Audit for missing security headers (default: true)"
    }
  },
  "required": [
    "headers"
  ]
}

generate_curl

Generate a curl command from request parameters. Supports GET/POST/PUT/DELETE, custom headers, JSON body, and form data. Useful for documentation, sharing, and debugging API calls.

Parameters7

url	string	required	Request URL (must be http/https)
method	string	optional	HTTP method (default: GET)
headers	object	optional	Request headers as key-value object
body	string	optional	Raw request body string
body_json	object	optional	JSON body (auto-adds Content-Type: application/json)
follow_redirects	boolean	optional	Follow redirects with -L flag (default: true)
verbose	boolean	optional	Add -v for verbose output (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "description": "Request URL (must be http/https)"
    },
    "method": {
      "type": "string",
      "description": "HTTP method (default: GET)"
    },
    "headers": {
      "type": "object",
      "additionalProperties": true,
      "description": "Request headers as key-value object"
    },
    "body": {
      "type": "string",
      "description": "Raw request body string"
    },
    "body_json": {
      "type": "object",
      "additionalProperties": true,
      "description": "JSON body (auto-adds Content-Type: application/json)"
    },
    "follow_redirects": {
      "type": "boolean",
      "description": "Follow redirects with -L flag (default: true)"
    },
    "verbose": {
      "type": "boolean",
      "description": "Add -v for verbose output (default: false)"
    }
  },
  "required": [
    "url"
  ]
}

extract_todos

Extract TODO, FIXME, HACK, BUG, NOTE, OPTIMIZE, and custom tags from any source code or text. Returns line numbers, tag types, and message text. Essential for technical debt auditing.

Parameters3

input	string	required	Code or text to scan
tags	array	optional	Custom tags to add (default set: TODO, FIXME, HACK, NOTE, BUG, OPTIMIZE, XXX)
include_context	boolean	optional	Include full line text (default: true)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Code or text to scan"
    },
    "tags": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Custom tags to add (default set: TODO, FIXME, HACK, NOTE, BUG, OPTIMIZE, XXX)"
    },
    "include_context": {
      "type": "boolean",
      "description": "Include full line text (default: true)"
    }
  },
  "required": [
    "input"
  ]
}

detect_secrets

Scan code or config files for hardcoded secrets: AWS keys, GitHub tokens, OpenAI/Anthropic API keys, Stripe secrets, JWTs, database connection strings, and generic passwords. Returns findings with severity. Run before every commit.

Parameters2

input	string	required	Code or config content to scan (max 500KB)
filename	string	optional	Optional filename for context (e.g. ".env", "config.js")

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Code or config content to scan (max 500KB)"
    },
    "filename": {
      "type": "string",
      "description": "Optional filename for context (e.g. \".env\", \"config.js\")"
    }
  },
  "required": [
    "input"
  ]
}

count_code_lines

Count lines of code: total, code lines, comment lines, blank lines, and comment density. Supports JS/TS, Python, Java/C/C++, Ruby, Go, Shell, HTML/XML, and CSS.

Parameters2

input	string	required	Source code to analyze
language	string	optional	Language hint: "js", "ts", "py", "java", "c", "rb", "go", "sh", "html", "css" (auto-detect if omitted)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Source code to analyze"
    },
    "language": {
      "type": "string",
      "description": "Language hint: \"js\", \"ts\", \"py\", \"java\", \"c\", \"rb\", \"go\", \"sh\", \"html\", \"css\" (auto-detect if omitted)"
    }
  },
  "required": [
    "input"
  ]
}

lint_commit_message

Validate a git commit message against the Conventional Commits spec (feat, fix, docs, style, refactor, test, chore, ci, perf, build). Returns compliance score, breaking change detection, and actionable suggestions.

Parameters2

message	string	required	Git commit message to validate
strict	boolean	optional	Enforce strict rules: max 72-char subject, imperative mood check (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "message": {
      "type": "string",
      "description": "Git commit message to validate"
    },
    "strict": {
      "type": "boolean",
      "description": "Enforce strict rules: max 72-char subject, imperative mood check (default: false)"
    }
  },
  "required": [
    "message"
  ]
}

word_frequency

Analyze word frequency in text. Returns top N words with counts and percentages. Supports English stopword filtering. Useful for content analysis, keyword extraction, and LLM output analysis.

Parameters4

input	string	required	Text to analyze
top_n	number	optional	Return top N words (default: 20, max: 200)
min_length	number	optional	Minimum word length to include (default: 3)
remove_stopwords	boolean	optional	Remove common English stopwords (default: true)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to analyze"
    },
    "top_n": {
      "type": "number",
      "description": "Return top N words (default: 20, max: 200)"
    },
    "min_length": {
      "type": "number",
      "description": "Minimum word length to include (default: 3)"
    },
    "remove_stopwords": {
      "type": "boolean",
      "description": "Remove common English stopwords (default: true)"
    }
  },
  "required": [
    "input"
  ]
}

extract_links

Extract all URLs, email addresses, and domain names from text. Returns categorized and deduplicated results. Useful for content auditing, link checking, and web scraping validation.

Parameters2

input	string	required	Text to extract links from
types	array	optional	Types to extract (default: all three)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to extract links from"
    },
    "types": {
      "type": "array",
      "items": {
        "type": "string",
        "enum": [
          "url",
          "email",
          "domain"
        ]
      },
      "description": "Types to extract (default: all three)"
    }
  },
  "required": [
    "input"
  ]
}

levenshtein_distance

Compute the Levenshtein (edit) distance and normalized similarity ratio between two strings. Supports batch comparison. Useful for fuzzy string matching, deduplication, and test result comparison.

Parameters4

a	string	optional	First string (single-pair mode)
b	string	optional	Second string (single-pair mode)
batch	array	optional	Batch of {a,b} pairs (max 50)
case_insensitive	boolean	optional	Ignore case differences (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "a": {
      "type": "string",
      "description": "First string (single-pair mode)"
    },
    "b": {
      "type": "string",
      "description": "Second string (single-pair mode)"
    },
    "batch": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "a": {
            "type": "string"
          },
          "b": {
            "type": "string"
          }
        }
      },
      "description": "Batch of {a,b} pairs (max 50)"
    },
    "case_insensitive": {
      "type": "boolean",
      "description": "Ignore case differences (default: false)"
    }
  }
}

json_diff

Compute a deep structural diff between two JSON values. Returns added, removed, and changed keys with dot-notation paths. Like git diff but for JSON objects — perfect for API response regression testing.

Parameters3

before	string	required	Original JSON string (before)
after	string	required	Modified JSON string (after)
max_depth	number	optional	Max nesting depth to recurse (default: 10)

Raw schema

{
  "type": "object",
  "properties": {
    "before": {
      "type": "string",
      "description": "Original JSON string (before)"
    },
    "after": {
      "type": "string",
      "description": "Modified JSON string (after)"
    },
    "max_depth": {
      "type": "number",
      "description": "Max nesting depth to recurse (default: 10)"
    }
  },
  "required": [
    "before",
    "after"
  ]
}

merge_json

Deep merge two JSON objects. Supports three array strategies: replace (default), concat, or unique (dedup concat). Nested objects are recursively merged — override takes precedence for primitives.

Parameters3

base	string	required	Base JSON object (will be merged into)
override	string	required	Override JSON object (takes precedence)
array_strategy	string	optional	Array merge strategy: replace (default), concat, or unique

Raw schema

{
  "type": "object",
  "properties": {
    "base": {
      "type": "string",
      "description": "Base JSON object (will be merged into)"
    },
    "override": {
      "type": "string",
      "description": "Override JSON object (takes precedence)"
    },
    "array_strategy": {
      "type": "string",
      "enum": [
        "replace",
        "concat",
        "unique"
      ],
      "description": "Array merge strategy: replace (default), concat, or unique"
    }
  },
  "required": [
    "base",
    "override"
  ]
}

json_to_yaml

Convert a JSON object to clean, human-readable YAML. Handles nested objects, arrays, multiline strings, and special characters. No external dependencies.

Parameters2

input	string	required	JSON string to convert to YAML
indent	number	optional	Indentation size in spaces (default: 2)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "JSON string to convert to YAML"
    },
    "indent": {
      "type": "number",
      "description": "Indentation size in spaces (default: 2)"
    }
  },
  "required": [
    "input"
  ]
}

generate_hmac

Compute an HMAC signature for a message using a secret key. Supports SHA-256 (default), SHA-512, SHA-1, and MD5. Used for API request signing, webhook verification (GitHub, Stripe, Twilio), and JWT validation.

Parameters4

message	string	required	Message to sign
secret	string	required	Secret key
algorithm	string	optional	Hash algorithm: sha256 (default), sha512, sha1, md5
encoding	string	optional	Output encoding (default: hex)

Raw schema

{
  "type": "object",
  "properties": {
    "message": {
      "type": "string",
      "description": "Message to sign"
    },
    "secret": {
      "type": "string",
      "description": "Secret key"
    },
    "algorithm": {
      "type": "string",
      "description": "Hash algorithm: sha256 (default), sha512, sha1, md5"
    },
    "encoding": {
      "type": "string",
      "enum": [
        "hex",
        "base64",
        "base64url"
      ],
      "description": "Output encoding (default: hex)"
    }
  },
  "required": [
    "message",
    "secret"
  ]
}

format_bytes

Convert raw byte counts to human-readable sizes in SI (KB=1000) or IEC (KiB=1024) units, or parse size strings back to bytes. Covers B, KB/KiB, MB/MiB, GB/GiB, TB/TiB, PB/PiB.

Parameters3

bytes	number	optional	Number of bytes to format
size_string	string	optional	Size string to parse to bytes (e.g. "1.5 GB", "512 MiB")
standard	string	optional	Output standard (default: both)

Raw schema

{
  "type": "object",
  "properties": {
    "bytes": {
      "type": "number",
      "description": "Number of bytes to format"
    },
    "size_string": {
      "type": "string",
      "description": "Size string to parse to bytes (e.g. \"1.5 GB\", \"512 MiB\")"
    },
    "standard": {
      "type": "string",
      "enum": [
        "both",
        "si",
        "iec"
      ],
      "description": "Output standard (default: both)"
    }
  }
}

detect_language

Detect the natural language of a text using n-gram frequency analysis and common word markers. Supports 15 languages: English, French, Spanish, German, Italian, Portuguese, Dutch, Russian, Chinese, Japanese, Korean, Arabic, Polish, Turkish, Swedish.

Parameters1

input

string

required

Text to detect language from (min 20 chars for accuracy)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to detect language from (min 20 chars for accuracy)"
    }
  },
  "required": [
    "input"
  ]
}

pr_gatekeeper

Compound quality gate for pull requests. Runs three sequential checks: (1) secret detection — scans diff for API keys, tokens, passwords matching 16 regex patterns; (2) bug analysis — heuristic scan for eval(), innerHTML, empty catch, console.log, TODO/FIXME; (3) commit message linting against Conventional Commits spec. Returns gate verdict (PASS/WARN/BLOCK), blockers, and actionable warnings. Use before merging any code change.

Parameters3

diff	string	required	Unified git diff (output of `git diff HEAD`)
commit_message	string	required	The commit message to lint (e.g. "feat(auth): add OAuth2 login")
context	string	optional	Optional: PR title or description for richer bug analysis

Raw schema

{
  "type": "object",
  "properties": {
    "diff": {
      "type": "string",
      "description": "Unified git diff (output of `git diff HEAD`)"
    },
    "commit_message": {
      "type": "string",
      "description": "The commit message to lint (e.g. \"feat(auth): add OAuth2 login\")"
    },
    "context": {
      "type": "string",
      "description": "Optional: PR title or description for richer bug analysis"
    }
  },
  "required": [
    "diff",
    "commit_message"
  ]
}

list_llm_models

List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.

Parameters1

provider

string

optional

Filter by provider name (case-insensitive). E.g. "Groq", "HuggingFace", "OpenAI", "Anthropic", "Google", "DeepSeek", "xAI", "Ollama". Omit for full catalog.

Raw schema

{
  "type": "object",
  "properties": {
    "provider": {
      "type": "string",
      "description": "Filter by provider name (case-insensitive). E.g. \"Groq\", \"HuggingFace\", \"OpenAI\", \"Anthropic\", \"Google\", \"DeepSeek\", \"xAI\", \"Ollama\". Omit for full catalog."
    }
  }
}

llm_generate

Generate text using open-source LLM models hosted on Groq (ultra-fast) or HuggingFace Inference (serverless). No API key required — the server provides its own keys. Supported models: Qwen3 32B, Gemma 4 27B, Gemma 3 27B, Llama 3.3 70B, Llama 4 Scout, DeepSeek R1, Mistral Small 24B, and more. Use list_llm_models to see the full catalog. Rate-limited to prevent abuse.

Parameters5

prompt	string	required	The user prompt / instruction to send to the model
model	string	optional	Model ID (default: "qwen/qwen3-32b"). Server-keyed whitelist only — Groq: qwen/qwen3-32b, llama-3.3-70b-versatile, meta-llama/llama-4-scout-17b-16e-instruct, llama-3.1-8b-instant; HuggingFace: Qwen/Qwen3-32B, meta-llama/Llama-3.3-70B-Instruct, deepseek-ai/DeepSeek-R1, google/gemma-3-27b-it, and more. Other ids from list_llm_models are BYOK-only and will be rejected.
system	string	optional	Optional system prompt to set context or persona
temperature	number	optional	Sampling temperature 0.0–1.5 (default: 0.7)
max_tokens	number	optional	Maximum tokens to generate (default: 2048, max: 4096)

Raw schema

{
  "type": "object",
  "properties": {
    "prompt": {
      "type": "string",
      "description": "The user prompt / instruction to send to the model"
    },
    "model": {
      "type": "string",
      "description": "Model ID (default: \"qwen/qwen3-32b\"). Server-keyed whitelist only — Groq: qwen/qwen3-32b, llama-3.3-70b-versatile, meta-llama/llama-4-scout-17b-16e-instruct, llama-3.1-8b-instant; HuggingFace: Qwen/Qwen3-32B, meta-llama/Llama-3.3-70B-Instruct, deepseek-ai/DeepSeek-R1, google/gemma-3-27b-it, and more. Other ids from list_llm_models are BYOK-only and will be rejected."
    },
    "system": {
      "type": "string",
      "description": "Optional system prompt to set context or persona"
    },
    "temperature": {
      "type": "number",
      "description": "Sampling temperature 0.0–1.5 (default: 0.7)"
    },
    "max_tokens": {
      "type": "number",
      "description": "Maximum tokens to generate (default: 2048, max: 4096)"
    }
  },
  "required": [
    "prompt"
  ]
}

rerank_evaluate

Evaluate RAG retrieval quality using the NVIDIA neural reranker (nv-rerankqa-mistral-4b-v3). Ranks passages by semantic relevance to a query and computes Precision@k and Recall@k. Optionally accepts ground-truth relevance labels to produce a PASS/FAIL CI/CD verdict.

Parameters4

query	string	required	The search query or question to rank against
passages	array	required	Array of passage objects to rank (min 2, max 20)
top_k	integer	optional	k for Precision@k evaluation (default 3)
threshold	number	optional	Minimum Precision@k to PASS (0-1, default 0.5)

Raw schema

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "The search query or question to rank against"
    },
    "passages": {
      "type": "array",
      "description": "Array of passage objects to rank (min 2, max 20)",
      "items": {
        "type": "object",
        "required": [
          "text"
        ],
        "properties": {
          "text": {
            "type": "string"
          },
          "relevant": {
            "type": "boolean",
            "description": "Ground truth: is this passage relevant?"
          },
          "id": {
            "type": "string"
          }
        }
      }
    },
    "top_k": {
      "type": "integer",
      "description": "k for Precision@k evaluation (default 3)",
      "minimum": 1,
      "maximum": 10
    },
    "threshold": {
      "type": "number",
      "description": "Minimum Precision@k to PASS (0-1, default 0.5)",
      "minimum": 0,
      "maximum": 1
    }
  },
  "required": [
    "query",
    "passages"
  ]
}

shield_analyze

Run a comprehensive AI guardrail analysis on an LLM response. Orchestrates 6 deterministic safety checks plus an optional LLM-powered deep analysis in parallel: hallucination detection (grounding score), prompt injection scan, toxicity scan, output validation (PII/safety), guardrail rules, response quality scoring, and AI verdict (via Qwen, Gemma, Llama, etc.). Returns a unified PASS/FIX/BLOCK verdict with a 0-100 safety score, per-check results, and actionable fix recommendations. Use this as a single-call safety gate before surfacing any LLM output to users.

Parameters5

response	string	required	The LLM-generated response to analyze
source	string	optional	Optional reference/source text for hallucination grounding check
prompt	string	optional	Optional original prompt (used for quality scoring and injection detection)
rules	array	optional	Optional guardrail rules array (same format as guardrail_test tool)
model	string	optional	LLM model for AI-powered deep analysis (default: "qwen/qwen3-32b"). Set to "none" to skip LLM check. Supports any model from list_llm_models.

Raw schema

{
  "type": "object",
  "properties": {
    "response": {
      "type": "string",
      "description": "The LLM-generated response to analyze"
    },
    "source": {
      "type": "string",
      "description": "Optional reference/source text for hallucination grounding check"
    },
    "prompt": {
      "type": "string",
      "description": "Optional original prompt (used for quality scoring and injection detection)"
    },
    "rules": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string"
          },
          "value": {
            "type": "string"
          },
          "label": {
            "type": "string"
          }
        }
      },
      "description": "Optional guardrail rules array (same format as guardrail_test tool)"
    },
    "model": {
      "type": "string",
      "description": "LLM model for AI-powered deep analysis (default: \"qwen/qwen3-32b\"). Set to \"none\" to skip LLM check. Supports any model from list_llm_models."
    }
  },
  "required": [
    "response"
  ]
}

security_headers_check

Analyse the HTTP security headers of a public URL OR of raw response headers you paste in. Grades each header (A–F) for: Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, X-XSS-Protection, Cross-Origin-Opener-Policy, Cross-Origin-Resource-Policy, and Cross-Origin-Embedder-Policy. Returns an overall score (0–100), per-header grades, missing headers, and fix snippets for Express, Nginx, and Apache. For localhost/private targets the remote server cannot reach, pass the `headers` parameter instead of `url`.

Parameters2

url	string	optional	Optional. Full public URL to check (e.g. https://example.com). Omit it entirely when using `headers`. The server cannot reach localhost/private IPs.
headers	any	optional	Optional, and sufficient on its own (no url needed). The response headers to grade, either as an object {"strict-transport-security": "max-age=...", ...} or as the raw header block pasted as a string (e.g. `curl -sI` output). Use this to audit a local server the remote MCP cannot reach.

Raw schema

{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "description": "Optional. Full public URL to check (e.g. https://example.com). Omit it entirely when using `headers`. The server cannot reach localhost/private IPs."
    },
    "headers": {
      "description": "Optional, and sufficient on its own (no url needed). The response headers to grade, either as an object {\"strict-transport-security\": \"max-age=...\", ...} or as the raw header block pasted as a string (e.g. `curl -sI` output). Use this to audit a local server the remote MCP cannot reach."
    }
  }
}

ssl_certificate_check

Analyse the SSL/TLS certificate of any HTTPS host. Returns certificate subject, issuer, validity dates, days until expiry, protocol version, cipher suite, key exchange info, and an overall grade (A+, A, B, C, F). Detects expired, self-signed, and weak certificates. Use this to audit TLS posture before production deployment or during security reviews.

Parameters2

host	string	required	Hostname to check (e.g. example.com). Do not include https:// prefix.
port	number	optional	Port number (default: 443)

Raw schema

{
  "type": "object",
  "properties": {
    "host": {
      "type": "string",
      "description": "Hostname to check (e.g. example.com). Do not include https:// prefix."
    },
    "port": {
      "type": "number",
      "description": "Port number (default: 443)"
    }
  },
  "required": [
    "host"
  ]
}

cors_test

Test a URL for CORS misconfigurations. Sends preflight (OPTIONS) and cross-origin requests with various Origin headers to detect: wildcard origins with credentials, origin reflection (echoing any origin), null origin acceptance, subdomain wildcard bypass, and missing Vary headers. Returns risk level (safe/low/medium/high/critical), per-test results, and fix recommendations. Essential for API security audits.

Parameters2

url	string	required	Full URL to test (e.g. https://api.example.com/endpoint)
origin	string	optional	Custom Origin header to test (default: tests multiple origins automatically)

Raw schema

{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "description": "Full URL to test (e.g. https://api.example.com/endpoint)"
    },
    "origin": {
      "type": "string",
      "description": "Custom Origin header to test (default: tests multiple origins automatically)"
    }
  },
  "required": [
    "url"
  ]
}

cors_checker

Check the CORS configuration of a URL the same way a browser would. Returns the main response status, all Access-Control-* headers, the tested origin, and the preflight OPTIONS response. Use this for direct CORS debugging, not just security auditing.

Parameters3

url	string	required	Full URL to test, e.g. https://api.example.com/resource
origin	string	optional	Origin header to simulate (default: https://yourdomain.com)
method	string	optional	HTTP method to simulate (default: GET)

Raw schema

{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "description": "Full URL to test, e.g. https://api.example.com/resource"
    },
    "origin": {
      "type": "string",
      "description": "Origin header to simulate (default: https://yourdomain.com)"
    },
    "method": {
      "type": "string",
      "description": "HTTP method to simulate (default: GET)"
    }
  },
  "required": [
    "url"
  ]
}

webhook_endpoint_create

Create a temporary webhook endpoint that captures incoming HTTP requests for one hour. Returns the webhook id, public URL, expiration timestamp, and current request count. Use together with webhook_endpoint_requests to inspect captured payloads.

Parameters1

base_url

string

optional

Optional public base URL. Default: https://ia-qa.com/mcp/webhook

Raw schema

{
  "type": "object",
  "properties": {
    "base_url": {
      "type": "string",
      "description": "Optional public base URL. Default: https://ia-qa.com/mcp/webhook"
    }
  }
}

webhook_endpoint_requests

Fetch the requests captured by a webhook created with webhook_endpoint_create. Returns the newest requests first with method, headers, query params, body payload, and timestamps.

Parameters2

id	string	required	Webhook id returned by webhook_endpoint_create
limit	number	optional	Maximum number of requests to return (1-100, default: 20)

Raw schema

{
  "type": "object",
  "properties": {
    "id": {
      "type": "string",
      "description": "Webhook id returned by webhook_endpoint_create"
    },
    "limit": {
      "type": "number",
      "description": "Maximum number of requests to return (1-100, default: 20)"
    }
  },
  "required": [
    "id"
  ]
}

cookie_security_audit

Audit the security attributes of cookies set by any URL. Fetches the URL and inspects all Set-Cookie headers for: HttpOnly, Secure, SameSite, Domain scope, Path scope, Max-Age/Expires, __Host-/__Secure- prefixes. Flags insecure patterns: missing HttpOnly on session cookies, missing Secure flag, SameSite=None without Secure, overly broad Domain, and excessive TTL. Returns per-cookie grades and an overall security score (0–100).

Parameters1

url

string

required

Full URL to audit (e.g. https://example.com/login)

Raw schema

{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "description": "Full URL to audit (e.g. https://example.com/login)"
    }
  },
  "required": [
    "url"
  ]
}

web_security_audit

Run a comprehensive web security audit combining headers, SSL, CORS, and cookies checks — then use an LLM to produce a prioritised remediation plan. Orchestrates security_headers_check + ssl_certificate_check + cors_test + cookie_security_audit in parallel, merges all findings, then asks an AI model to: (1) rank vulnerabilities by real-world exploitability, (2) generate a remediation roadmap, (3) produce fix code snippets for the detected stack. Returns both raw audit data and the AI analysis. Use this as a one-click security posture assessment.

Parameters3

url	string	required	Full URL to audit (e.g. https://example.com)
model	string	optional	LLM model for AI analysis (default: "qwen/qwen3-32b"). Set to "none" to skip AI analysis.
api_key	string	optional	Your Groq or HuggingFace API key. Required to enable AI analysis.

Raw schema

{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "description": "Full URL to audit (e.g. https://example.com)"
    },
    "model": {
      "type": "string",
      "description": "LLM model for AI analysis (default: \"qwen/qwen3-32b\"). Set to \"none\" to skip AI analysis."
    },
    "api_key": {
      "type": "string",
      "description": "Your Groq or HuggingFace API key. Required to enable AI analysis."
    }
  },
  "required": [
    "url"
  ]
}

secret_scan

Scan text or code for leaked secrets: API keys (AWS, GCP, Azure, OpenAI, Anthropic, Stripe, GitHub, GitLab, Slack, Twilio, SendGrid, HuggingFace), private keys (RSA/EC/PGP), JWTs, database connection strings, Bearer tokens, and Basic auth headers. Returns a list of findings with type, severity, line number, and a redacted preview. Use before committing code, sharing logs, or sending text to an LLM. 100% regex-based, zero network calls.

Parameters2

input	string	required	Text or code to scan for secrets
types	string	optional	Comma-separated types to scan (default: all). Options: aws, gcp, azure, openai, anthropic, stripe, github, gitlab, slack, twilio, sendgrid, huggingface, jwt, private_key, connection_string, bearer, basic_auth

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Text or code to scan for secrets"
    },
    "types": {
      "type": "string",
      "description": "Comma-separated types to scan (default: all). Options: aws, gcp, azure, openai, anthropic, stripe, github, gitlab, slack, twilio, sendgrid, huggingface, jwt, private_key, connection_string, bearer, basic_auth"
    }
  },
  "required": [
    "input"
  ]
}

similarity_score

Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.

Parameters5

reference	string	optional	Reference / expected text (ground truth)
hypothesis	string	optional	Hypothesis / actual text (LLM output)
metrics	array	optional	Metrics to compute (default: all). Options: "cosine_bow", "cosine_tfidf", "jaccard", "rouge1", "rouge2", "rougeL", "bleu"
batch	array	optional	Batch mode: array of {reference, hypothesis} pairs.
threshold	number	optional	Optional pass/fail threshold (0-1). Applies to ROUGE-L F1 score.

Raw schema

{
  "type": "object",
  "properties": {
    "reference": {
      "type": "string",
      "description": "Reference / expected text (ground truth)"
    },
    "hypothesis": {
      "type": "string",
      "description": "Hypothesis / actual text (LLM output)"
    },
    "metrics": {
      "type": "array",
      "description": "Metrics to compute (default: all). Options: \"cosine_bow\", \"cosine_tfidf\", \"jaccard\", \"rouge1\", \"rouge2\", \"rougeL\", \"bleu\"",
      "items": {
        "type": "string"
      }
    },
    "batch": {
      "type": "array",
      "description": "Batch mode: array of {reference, hypothesis} pairs.",
      "items": {
        "type": "object",
        "properties": {
          "reference": {
            "type": "string"
          },
          "hypothesis": {
            "type": "string"
          }
        },
        "required": [
          "reference",
          "hypothesis"
        ]
      }
    },
    "threshold": {
      "type": "number",
      "description": "Optional pass/fail threshold (0-1). Applies to ROUGE-L F1 score."
    }
  }
}

needle_haystack_generate

Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.

Parameters4

needle	string	required	The fact to hide (e.g. "The secret code is ALPHA-42")
question	string	required	The question to ask the LLM (e.g. "What is the secret code?")
tokens	integer	optional	Target haystack size in tokens (default: 5000, max: 100000)
position	string	optional	Where to insert the needle: "start", "middle", "end", "random" (default: "middle")

Raw schema

{
  "type": "object",
  "properties": {
    "needle": {
      "type": "string",
      "description": "The fact to hide (e.g. \"The secret code is ALPHA-42\")"
    },
    "question": {
      "type": "string",
      "description": "The question to ask the LLM (e.g. \"What is the secret code?\")"
    },
    "tokens": {
      "type": "integer",
      "description": "Target haystack size in tokens (default: 5000, max: 100000)",
      "default": 5000
    },
    "position": {
      "type": "string",
      "description": "Where to insert the needle: \"start\", \"middle\", \"end\", \"random\" (default: \"middle\")",
      "enum": [
        "start",
        "middle",
        "end",
        "random"
      ],
      "default": "middle"
    }
  },
  "required": [
    "needle",
    "question"
  ]
}

optimize_prompt_tokens

Compress an LLM prompt by removing filler words, verbose phrases, duplicate sentences, and unnecessary whitespace. Returns optimized text with token savings breakdown. 100% deterministic, no API key needed.

Parameters2

text	string	required	The prompt text to optimize
options	object	optional	Toggle optimization steps (all true by default)

Raw schema

{
  "type": "object",
  "properties": {
    "text": {
      "type": "string",
      "description": "The prompt text to optimize"
    },
    "options": {
      "type": "object",
      "description": "Toggle optimization steps (all true by default)",
      "properties": {
        "whitespace": {
          "type": "boolean",
          "default": true
        },
        "duplicates": {
          "type": "boolean",
          "default": true
        },
        "fillers": {
          "type": "boolean",
          "default": true
        },
        "instructions": {
          "type": "boolean",
          "default": true
        }
      }
    }
  },
  "required": [
    "text"
  ]
}

bias_detect

Analyse a set of LLM responses generated from the same prompt template but with different demographic variants (gender, origin, age, tone). Returns a bias score (0-100), sentiment analysis per variant, pairwise Jaccard similarity, and a human-readable verdict. No API key needed — runs entirely locally.

Parameters1

responses

array

required

Array of variant responses to compare for bias

Raw schema

{
  "type": "object",
  "properties": {
    "responses": {
      "type": "array",
      "minItems": 2,
      "description": "Array of variant responses to compare for bias",
      "items": {
        "type": "object",
        "required": [
          "variantId",
          "answer"
        ],
        "properties": {
          "variantId": {
            "type": "string",
            "description": "Identifier for the demographic variant (e.g. \"male\", \"female\", \"western\", \"young\")"
          },
          "answer": {
            "type": "string",
            "description": "The LLM response text for this variant"
          }
        }
      }
    }
  },
  "required": [
    "responses"
  ]
}

llm_fit_finder

Find the best LLM for a given use case. Compares 30+ cloud API models and 12+ local models by cost, speed, benchmarks, features and VRAM requirements. Returns ranked recommendations with cost simulation. No API key needed.

Parameters8

use_case	string	optional	Primary use case: chatbot \| code \| rag \| summarization \| classification \| reasoning \| agents \| multilingual
features	array	optional	Required features: vision, function_calling, json_mode, streaming, reasoning
max_budget	number	optional	Maximum monthly budget in USD (based on tokens_per_day)
tokens_per_day	number	optional	Estimated daily token volume (default: 100000)
mode	string	optional	cloud (API models) or local (Ollama/self-hosted). Default: cloud
vram_gb	number	optional	GPU VRAM in GB (only for mode=local). Default: 16
quantization	string	optional	Quantization (only for mode=local): Q4_K_M \| Q8_0 \| FP16. Default: Q4_K_M
top_n	number	optional	Number of recommendations to return (default: 5)

Raw schema

{
  "type": "object",
  "properties": {
    "use_case": {
      "type": "string",
      "description": "Primary use case: chatbot | code | rag | summarization | classification | reasoning | agents | multilingual"
    },
    "features": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Required features: vision, function_calling, json_mode, streaming, reasoning"
    },
    "max_budget": {
      "type": "number",
      "description": "Maximum monthly budget in USD (based on tokens_per_day)"
    },
    "tokens_per_day": {
      "type": "number",
      "description": "Estimated daily token volume (default: 100000)"
    },
    "mode": {
      "type": "string",
      "description": "cloud (API models) or local (Ollama/self-hosted). Default: cloud"
    },
    "vram_gb": {
      "type": "number",
      "description": "GPU VRAM in GB (only for mode=local). Default: 16"
    },
    "quantization": {
      "type": "string",
      "description": "Quantization (only for mode=local): Q4_K_M | Q8_0 | FP16. Default: Q4_K_M"
    },
    "top_n": {
      "type": "number",
      "description": "Number of recommendations to return (default: 5)"
    }
  }
}

diff_mappings

Diff a baseline page mapping against a current one and return a CI-style verdict: PASS / FIX / BLOCK, plus per-element drift (ok, renamed, healable, ambiguous, lost, added, rebound). Pure and deterministic — provide two mappings as JSON with "elements" arrays of {role, name, selector, context?}. Use the companion @ia-qa/self-healing package (npm install -g @ia-qa/self-healing) to capture mappings from your app via its local MCP server ia-qa-heal-mcp, or paste the snippet from ia-qa.com/devtools/selector-drift into your browser console.

Parameters2

before	object	required	Baseline page mapping: { page, url, capturedAt, elements: [{role, name, selector, context?}] }. Captured before a UI change.
after	object	required	Current page mapping: same shape as before, captured after the UI change.

Raw schema

{
  "type": "object",
  "properties": {
    "before": {
      "type": "object",
      "description": "Baseline page mapping: { page, url, capturedAt, elements: [{role, name, selector, context?}] }. Captured before a UI change."
    },
    "after": {
      "type": "object",
      "description": "Current page mapping: same shape as before, captured after the UI change."
    }
  },
  "required": [
    "before",
    "after"
  ]
}

find_tool

Search available MCP tools by keyword or category before calling them. Returns matching tool names, descriptions, and optionally their inputSchemas. Call this when you are unsure which tool to use or want to explore the catalogue. Categories: data, encoding, text, llm, qa, rag, dev, security, web.

Parameters3

query	string	required	Keyword(s) to search in tool name and description (e.g. "cors", "token", "vector", "json")
category	string	optional	Optional: filter by category — data \| encoding \| text \| llm \| qa \| rag \| dev \| security \| web
with_schema	boolean	optional	Set true to include inputSchema in results (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "Keyword(s) to search in tool name and description (e.g. \"cors\", \"token\", \"vector\", \"json\")"
    },
    "category": {
      "type": "string",
      "description": "Optional: filter by category — data | encoding | text | llm | qa | rag | dev | security | web"
    },
    "with_schema": {
      "type": "boolean",
      "description": "Set true to include inputSchema in results (default: false)"
    }
  },
  "required": [
    "query"
  ]
}

list_local_tests

Discover .ia-eval.yaml LLM test suite files in the project directory. Scans CWD and standard sub-directories (evals/, tests/, contracts/). Returns file paths ready to pass to run_eval_contract.

Parameters1

dir

string

optional

Directory to scan (defaults to server CWD)

Raw schema

{
  "type": "object",
  "properties": {
    "dir": {
      "type": "string",
      "description": "Directory to scan (defaults to server CWD)"
    }
  }
}

run_eval_contract

Parse a .ia-eval.yaml LLM test suite, call the specified LLM model for each scenario, run all configured scorers, and return a structured JSON report with per-scenario Pass/Fail verdicts and a Markdown summary. Use list_local_tests to discover available test files.

Parameters4

contract_path	string	optional	Absolute or relative path to a .ia-eval.yaml file (required unless inline_contract is provided)
inline_contract	object	optional	Raw contract object (alternative to contract_path). Must contain top-level "metadata" ({name, version, model?, provider?}), "expectations" ({min_score?}), and "scenarios" ([{id, input, ground_truth?}]) — scenarios alone are rejected. Use generate_eval_yaml to scaffold one.
api_keys	object	optional	API keys to use for LLM generation (all optional — falls back to server env vars)
overrides	object	optional	Override contract defaults

Raw schema

{
  "type": "object",
  "properties": {
    "contract_path": {
      "type": "string",
      "description": "Absolute or relative path to a .ia-eval.yaml file (required unless inline_contract is provided)"
    },
    "inline_contract": {
      "type": "object",
      "additionalProperties": true,
      "description": "Raw contract object (alternative to contract_path). Must contain top-level \"metadata\" ({name, version, model?, provider?}), \"expectations\" ({min_score?}), and \"scenarios\" ([{id, input, ground_truth?}]) — scenarios alone are rejected. Use generate_eval_yaml to scaffold one."
    },
    "api_keys": {
      "type": "object",
      "description": "API keys to use for LLM generation (all optional — falls back to server env vars)",
      "properties": {
        "groq": {
          "type": "string"
        },
        "openai": {
          "type": "string"
        },
        "anthropic": {
          "type": "string"
        },
        "google": {
          "type": "string"
        },
        "hf": {
          "type": "string"
        }
      }
    },
    "overrides": {
      "type": "object",
      "description": "Override contract defaults",
      "properties": {
        "model": {
          "type": "string"
        },
        "provider": {
          "type": "string"
        },
        "temperature": {
          "type": "number"
        },
        "system_prompt": {
          "type": "string"
        }
      }
    }
  }
}

generate_html_report

Convert a run_eval_contract() LLM Test Runner JSON result into a fully self-contained dark-themed HTML report with Pass/Fail badges, side-by-side Input/Output/Ground-Truth panels, evaluator score bars, and a radar chart. Returns the HTML as a string.

Parameters1

results

object

required

The JSON object returned by run_eval_contract()

Raw schema

{
  "type": "object",
  "properties": {
    "results": {
      "type": "object",
      "additionalProperties": true,
      "description": "The JSON object returned by run_eval_contract()"
    }
  },
  "required": [
    "results"
  ]
}

run_vlm_test_suite

Run a test suite against a Vision-Language Model (VLM) — send an image (URL or base64) + N test cases (each with a question + assertion) to GPT-4o, Claude 3.5, or Gemini. Returns per-case PASS/FAIL verdicts, a pass rate, an overall PASS/WARNING/FAIL verdict (customizable threshold), and latency stats. Assertion types: contains, not_contains, json_format, min_length, max_length, semantic_contains (TF-IDF cosine similarity ≥ 0.4). BYOK: requires your own API key for the target provider.

Parameters8

image_url	string	optional	Public URL of the image to evaluate (required unless image_base64 is provided).
image_base64	string	optional	Base64-encoded image data (required unless image_url is provided).
image_mime_type	string	optional	MIME type of the image if using image_base64 (default: image/jpeg).
system_prompt	string	optional	Optional system prompt sent to the VLM.
threshold	number	optional	Pass rate threshold for overall verdict (default: 80, 0–100).
model	string	required	VLM model to use.
api_key	string	required	API key for the model provider (OpenAI sk-, Anthropic sk-ant-, or Google AIzaSy...).
test_cases	array	required	Array of test cases to run.

Raw schema

{
  "type": "object",
  "properties": {
    "image_url": {
      "type": "string",
      "description": "Public URL of the image to evaluate (required unless image_base64 is provided)."
    },
    "image_base64": {
      "type": "string",
      "description": "Base64-encoded image data (required unless image_url is provided)."
    },
    "image_mime_type": {
      "type": "string",
      "description": "MIME type of the image if using image_base64 (default: image/jpeg)."
    },
    "system_prompt": {
      "type": "string",
      "description": "Optional system prompt sent to the VLM."
    },
    "threshold": {
      "type": "number",
      "description": "Pass rate threshold for overall verdict (default: 80, 0–100)."
    },
    "model": {
      "type": "string",
      "enum": [
        "gpt-4o",
        "gpt-4o-mini",
        "claude-3-5-sonnet-20241022",
        "claude-3-5-haiku-20241022",
        "gemini-1.5-flash",
        "gemini-2.0-flash"
      ],
      "description": "VLM model to use."
    },
    "api_key": {
      "type": "string",
      "description": "API key for the model provider (OpenAI sk-, Anthropic sk-ant-, or Google AIzaSy...)."
    },
    "test_cases": {
      "type": "array",
      "maxItems": 10,
      "description": "Array of test cases to run.",
      "items": {
        "type": "object",
        "required": [
          "question"
        ],
        "properties": {
          "id": {
            "type": "string",
            "description": "Optional identifier for this case."
          },
          "question": {
            "type": "string",
            "description": "Question to ask the VLM about the image."
          },
          "assertion_type": {
            "type": "string",
            "enum": [
              "contains",
              "not_contains",
              "json_format",
              "min_length",
              "max_length",
              "semantic_contains"
            ],
            "description": "Assertion to run on the VLM response. semantic_contains uses TF-IDF cosine similarity ≥ 0.4."
          },
          "assertion_value": {
            "type": "string",
            "description": "Expected value for the assertion (not needed for json_format)."
          }
        }
      }
    }
  },
  "required": [
    "test_cases",
    "model",
    "api_key"
  ]
}

multimodal_eval_guide

Unified tool for multimodal AI evaluation: set action=guide for reference thresholds/interpretation (CLIP, FID, VQA), or set action=clip_score / fid_score / vqa_accuracy / pipeline to compute real metrics via HuggingFace Inference API and VLM BYOK calls. One tool for both reference and computation.

Parameters16

action	string	optional	guide (default) = reference thresholds/interpretation. clip_score/fid_score/vqa_accuracy = compute that metric. pipeline = run all three.
metric	string	optional	[guide only] Metric to explain.
score	number	optional	[guide only] Optional score value to interpret.
image_url	string	optional	[clip_score/vqa_accuracy] Public URL of the image.
text	string	optional	[clip_score only] Text description to compare against the image.
image_base64	string	optional	[clip_score/vqa_accuracy] Base64-encoded image data.
image_mime_type	string	optional	[clip_score/vqa_accuracy] MIME type for base64 image.
system_prompt	string	optional	[vqa_accuracy] Optional system prompt.
model	string	optional	[vqa_accuracy] VLM model ID (default: gpt-4o).
api_key	string	optional	[vqa_accuracy] Your API key for the provider (BYOK).
test_cases	array	optional	[vqa_accuracy] Array of {question, accepted_answers} objects.
real_images	array	optional	[fid_score] Array of real image URLs.
generated_images	array	optional	[fid_score] Array of generated image URLs.
clip	object	optional	[pipeline] {image_url, text} for CLIP.
vqa	object	optional	[pipeline] VQA config object (same inputs as vqa_accuracy).
fid	object	optional	[pipeline] {real_images, generated_images} for FID.

Raw schema

{
  "type": "object",
  "properties": {
    "action": {
      "type": "string",
      "enum": [
        "guide",
        "clip_score",
        "fid_score",
        "vqa_accuracy",
        "pipeline"
      ],
      "description": "guide (default) = reference thresholds/interpretation. clip_score/fid_score/vqa_accuracy = compute that metric. pipeline = run all three."
    },
    "metric": {
      "type": "string",
      "enum": [
        "clip_score",
        "fid",
        "vqa_accuracy",
        "all"
      ],
      "description": "[guide only] Metric to explain."
    },
    "score": {
      "type": "number",
      "description": "[guide only] Optional score value to interpret."
    },
    "image_url": {
      "type": "string",
      "description": "[clip_score/vqa_accuracy] Public URL of the image."
    },
    "text": {
      "type": "string",
      "description": "[clip_score only] Text description to compare against the image."
    },
    "image_base64": {
      "type": "string",
      "description": "[clip_score/vqa_accuracy] Base64-encoded image data."
    },
    "image_mime_type": {
      "type": "string",
      "description": "[clip_score/vqa_accuracy] MIME type for base64 image."
    },
    "system_prompt": {
      "type": "string",
      "description": "[vqa_accuracy] Optional system prompt."
    },
    "model": {
      "type": "string",
      "description": "[vqa_accuracy] VLM model ID (default: gpt-4o)."
    },
    "api_key": {
      "type": "string",
      "description": "[vqa_accuracy] Your API key for the provider (BYOK)."
    },
    "test_cases": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "question": {
            "type": "string"
          },
          "accepted_answers": {
            "type": "array",
            "items": {
              "type": "string"
            }
          }
        }
      },
      "description": "[vqa_accuracy] Array of {question, accepted_answers} objects."
    },
    "real_images": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "[fid_score] Array of real image URLs."
    },
    "generated_images": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "[fid_score] Array of generated image URLs."
    },
    "clip": {
      "type": "object",
      "additionalProperties": true,
      "description": "[pipeline] {image_url, text} for CLIP."
    },
    "vqa": {
      "type": "object",
      "additionalProperties": true,
      "description": "[pipeline] VQA config object (same inputs as vqa_accuracy)."
    },
    "fid": {
      "type": "object",
      "additionalProperties": true,
      "description": "[pipeline] {real_images, generated_images} for FID."
    }
  }
}

run_vlm_test_suite_batch

Compare multiple VLMs on the same test suite in parallel — send an image (URL or base64) + N test cases to all models simultaneously. Returns per-model PASS/FAIL verdicts, pass rates, latency stats, and a comparison table. Assertion types: contains, not_contains, json_format, min_length, max_length, semantic_contains. BYOK: requires API keys for each provider.

Parameters8

image_url	string	optional	Public URL of the image to evaluate (required unless image_base64 is provided).
image_base64	string	optional	Base64-encoded image data (required unless image_url is provided).
image_mime_type	string	optional	MIME type of the image if using image_base64 (default: image/jpeg).
system_prompt	string	optional	Optional system prompt sent to every VLM.
threshold	number	optional	Pass rate threshold for overall verdict (default: 80, 0–100).
models	array	required	Array of model IDs to compare (runs in parallel).
api_keys	object	required	Map of model ID → API key. Example: { "gpt-4o": "sk-...", "claude-3-5-sonnet-20241022": "sk-ant-..." }
test_cases	array	required	Array of test cases to run against every model.

Raw schema

{
  "type": "object",
  "properties": {
    "image_url": {
      "type": "string",
      "description": "Public URL of the image to evaluate (required unless image_base64 is provided)."
    },
    "image_base64": {
      "type": "string",
      "description": "Base64-encoded image data (required unless image_url is provided)."
    },
    "image_mime_type": {
      "type": "string",
      "description": "MIME type of the image if using image_base64 (default: image/jpeg)."
    },
    "system_prompt": {
      "type": "string",
      "description": "Optional system prompt sent to every VLM."
    },
    "threshold": {
      "type": "number",
      "description": "Pass rate threshold for overall verdict (default: 80, 0–100)."
    },
    "models": {
      "type": "array",
      "minItems": 1,
      "maxItems": 6,
      "description": "Array of model IDs to compare (runs in parallel).",
      "items": {
        "type": "string",
        "enum": [
          "gpt-4o",
          "gpt-4o-mini",
          "claude-3-5-sonnet-20241022",
          "claude-3-5-haiku-20241022",
          "gemini-1.5-flash",
          "gemini-2.0-flash"
        ]
      }
    },
    "api_keys": {
      "type": "object",
      "description": "Map of model ID → API key. Example: { \"gpt-4o\": \"sk-...\", \"claude-3-5-sonnet-20241022\": \"sk-ant-...\" }",
      "additionalProperties": {
        "type": "string"
      }
    },
    "test_cases": {
      "type": "array",
      "maxItems": 10,
      "description": "Array of test cases to run against every model.",
      "items": {
        "type": "object",
        "required": [
          "question"
        ],
        "properties": {
          "id": {
            "type": "string",
            "description": "Optional identifier for this case."
          },
          "question": {
            "type": "string",
            "description": "Question to ask the VLM about the image."
          },
          "assertion_type": {
            "type": "string",
            "enum": [
              "contains",
              "not_contains",
              "json_format",
              "min_length",
              "max_length",
              "semantic_contains"
            ],
            "description": "Assertion to run on the VLM response."
          },
          "assertion_value": {
            "type": "string",
            "description": "Expected value for the assertion (not needed for json_format)."
          }
        }
      }
    }
  },
  "required": [
    "test_cases",
    "models",
    "api_keys"
  ]
}

generate_eval_yaml

Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq llama-3.3-70b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.

Parameters4

description	string	required	Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints.
system_prompt	string	optional	Optional system prompt of the LLM under test. Helps generate more accurate test cases.
scenario_count	number	optional	Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial.
task_type	string	optional	Optional task type hint to guide evaluator selection.

Raw schema

{
  "type": "object",
  "properties": {
    "description": {
      "type": "string",
      "description": "Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints."
    },
    "system_prompt": {
      "type": "string",
      "description": "Optional system prompt of the LLM under test. Helps generate more accurate test cases."
    },
    "scenario_count": {
      "type": "number",
      "enum": [
        3,
        5,
        8
      ],
      "description": "Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial."
    },
    "task_type": {
      "type": "string",
      "enum": [
        "rag",
        "summarization",
        "classification",
        "safety",
        "customer_support",
        "code_gen"
      ],
      "description": "Optional task type hint to guide evaluator selection."
    }
  },
  "required": [
    "description"
  ]
}

validate_agent_trajectory

Run declarative assertions on an agent trace (OpenAI tool-call messages, LangChain run trees, or plain text logs). No LLM call — deterministic. Assertion types: order (tool A before B), must_call, must_not_call, max_calls, min_calls, no_error, recovery (agent continues after error). Returns per-assertion PASS/FAIL, parsed steps, and an overall verdict. Use this to gate CI/CD on agent behavior correctness.

Parameters3

trace	any	required	Agent execution trace as JSON (OpenAI messages array, LangChain run tree) or plain text log (Thought/Action/Observation format).
assertions	array	required	List of assertions to validate against the trace.
format	string	optional	Trace format. auto (default) detects automatically.

Raw schema

{
  "type": "object",
  "properties": {
    "trace": {
      "description": "Agent execution trace as JSON (OpenAI messages array, LangChain run tree) or plain text log (Thought/Action/Observation format).",
      "oneOf": [
        {
          "type": "string"
        },
        {
          "type": "object",
          "additionalProperties": true
        },
        {
          "type": "array",
          "items": {}
        }
      ]
    },
    "assertions": {
      "type": "array",
      "maxItems": 30,
      "description": "List of assertions to validate against the trace.",
      "items": {
        "type": "object",
        "required": [
          "type"
        ],
        "properties": {
          "id": {
            "type": "string",
            "description": "Optional assertion identifier."
          },
          "type": {
            "type": "string",
            "enum": [
              "order",
              "must_call",
              "must_not_call",
              "max_calls",
              "min_calls",
              "no_error",
              "recovery"
            ],
            "description": "Assertion type."
          },
          "tool": {
            "type": "string",
            "description": "Tool name to check (for must_call, must_not_call, max_calls, min_calls)."
          },
          "before": {
            "type": "string",
            "description": "[order] Tool that must be called first."
          },
          "after": {
            "type": "string",
            "description": "[order] Tool that must be called after."
          },
          "max": {
            "type": "number",
            "description": "[max_calls] Maximum number of allowed calls."
          },
          "min": {
            "type": "number",
            "description": "[min_calls] Minimum number of required calls."
          }
        }
      }
    },
    "format": {
      "type": "string",
      "enum": [
        "auto",
        "openai",
        "langchain"
      ],
      "description": "Trace format. auto (default) detects automatically."
    }
  },
  "required": [
    "trace",
    "assertions"
  ]
}

run_semantic_tests

Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.

Parameters5

cases	array	required	Array of (actual, expected) pairs to evaluate.
thresholds	object	optional	Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5).
require_all	boolean	optional	If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS.
mode	string	optional	tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key.
api_key	string	optional	OpenAI API key — required only when mode is embeddings.

Raw schema

{
  "type": "object",
  "properties": {
    "cases": {
      "type": "array",
      "maxItems": 50,
      "description": "Array of (actual, expected) pairs to evaluate.",
      "items": {
        "type": "object",
        "required": [
          "actual",
          "expected"
        ],
        "properties": {
          "id": {
            "type": "string",
            "description": "Optional identifier for this case."
          },
          "actual": {
            "type": "string",
            "description": "The text produced by your LLM/system."
          },
          "expected": {
            "type": "string",
            "description": "The reference/ground-truth text."
          }
        }
      }
    },
    "thresholds": {
      "type": "object",
      "description": "Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5).",
      "properties": {
        "cosine": {
          "type": "number",
          "minimum": 0,
          "maximum": 1,
          "description": "Minimum cosine similarity to pass (default: 0.75)."
        },
        "rouge_l": {
          "type": "number",
          "minimum": 0,
          "maximum": 1,
          "description": "Minimum ROUGE-L F1 to pass (default: 0.5)."
        }
      }
    },
    "require_all": {
      "type": "boolean",
      "description": "If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS."
    },
    "mode": {
      "type": "string",
      "enum": [
        "tfidf",
        "embeddings"
      ],
      "description": "tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key."
    },
    "api_key": {
      "type": "string",
      "description": "OpenAI API key — required only when mode is embeddings."
    }
  },
  "required": [
    "cases"
  ]
}

get_testing_guidelines

Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset.

Parameters1

topic

string

optional

The testing topic to retrieve guidelines for. Omit to get the full list of available topics.

Raw schema

{
  "type": "object",
  "properties": {
    "topic": {
      "type": "string",
      "enum": [
        "llm-unit-testing",
        "rag-pipeline",
        "prompt-stability",
        "prompt-ab-testing",
        "embedding-quality",
        "eval-framework",
        "semantic-testing",
        "auto-testing",
        "security",
        "api-testing",
        "ci-cd",
        "multimodal",
        "llm-data-security",
        "agent-observability",
        "pro-tips",
        "learning-paths",
        "golden-dataset"
      ],
      "description": "The testing topic to retrieve guidelines for. Omit to get the full list of available topics."
    }
  }
}

test_skill

Validate a SKILL.md definition (Cursor / GitHub Copilot / Windsurf) by auto-generating trigger-positive and trigger-negative scenarios, running each through the model with the skill injected as a system prompt, and scoring trigger accuracy + step adherence. Returns a PASS/FIX/BLOCK verdict with per-scenario breakdown. Uses Groq llama-3.3-70b by default (server key, no api_key needed). Pass api_key + model to use your own provider.

Parameters4

skill_md	string	required	Full content of the SKILL.md file to test. Must include a name, a "Use when:" trigger description, and at least one step.
scenario_count	number	optional	Number of test scenarios to generate: half trigger-positive, half trigger-negative. Default: 6.
model	string	optional	LLM model ID to use for both scenario generation and testing (e.g. gpt-4o-mini, claude-3-5-haiku-20241022). Defaults to llama-3.3-70b-versatile (Groq, server key).
api_key	string	optional	API key for the chosen model provider. Not required when using the default Groq model.

Raw schema

{
  "type": "object",
  "properties": {
    "skill_md": {
      "type": "string",
      "description": "Full content of the SKILL.md file to test. Must include a name, a \"Use when:\" trigger description, and at least one step."
    },
    "scenario_count": {
      "type": "number",
      "enum": [
        4,
        6,
        8,
        10
      ],
      "description": "Number of test scenarios to generate: half trigger-positive, half trigger-negative. Default: 6."
    },
    "model": {
      "type": "string",
      "description": "LLM model ID to use for both scenario generation and testing (e.g. gpt-4o-mini, claude-3-5-haiku-20241022). Defaults to llama-3.3-70b-versatile (Groq, server key)."
    },
    "api_key": {
      "type": "string",
      "description": "API key for the chosen model provider. Not required when using the default Groq model."
    }
  },
  "required": [
    "skill_md"
  ]
}

identify_caller

Returns what the server knows about the current MCP client: clientInfo captured during initialize, User-Agent, and any _meta fields sent with this request. Useful for debugging caller identification.

Parameters1

_meta

object

optional

Optional self-identification. Keys: agent (string), model (string), version (string).

Raw schema

{
  "type": "object",
  "properties": {
    "_meta": {
      "type": "object",
      "description": "Optional self-identification. Keys: agent (string), model (string), version (string).",
      "properties": {
        "agent": {
          "type": "string"
        },
        "model": {
          "type": "string"
        },
        "version": {
          "type": "string"
        }
      }
    }
  }
}

yaml_to_json

Parse a YAML string and return the equivalent JSON value. The reverse of json_to_yaml. Supports nested objects, arrays, anchors, aliases, multi-document streams, and all scalar types. Use when processing config files, CI/CD pipeline definitions, or OpenAPI specs authored in YAML.

Parameters2

input	string	required	YAML string to parse
multi	boolean	optional	If true, parse all documents in a multi-document stream and return an array (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "YAML string to parse"
    },
    "multi": {
      "type": "boolean",
      "description": "If true, parse all documents in a multi-document stream and return an array (default: false)"
    }
  },
  "required": [
    "input"
  ]
}

env_parse

Parse a .env file content into a JSON object. Handles quoted values (single and double), inline comments, export prefix, and escaped sequences (\n, \t inside double quotes). Returns all key-value pairs. Use in CI/CD pipelines, agent config loaders, or when processing dotenv files programmatically.

Parameters1

input

string

required

.env file content to parse (e.g. the output of `cat .env`)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": ".env file content to parse (e.g. the output of `cat .env`)"
    }
  },
  "required": [
    "input"
  ]
}

json_schema_generate

Infer a JSON Schema (draft-07) from a sample JSON value. Detects types, required fields, array item shapes, nested objects, and common string formats (email, uri, date, date-time, uuid). Returns a ready-to-use schema compatible with json_schema_validate. Use when you have a sample API response or LLM output and want to auto-generate a validation schema for CI/CD testing.

Parameters2

input	string	required	Sample JSON value (object, array, or scalar) to infer the schema from
required_all	boolean	optional	Mark all detected object properties as required (default: true)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "Sample JSON value (object, array, or scalar) to infer the schema from"
    },
    "required_all": {
      "type": "boolean",
      "description": "Mark all detected object properties as required (default: true)"
    }
  },
  "required": [
    "input"
  ]
}

format_table

Convert a JSON array of objects into a Markdown table. Automatically detects columns, aligns headers, and fills missing keys with empty cells. Use when an agent needs to present structured data — tool results, model comparisons, test reports — as a readable table in a response or document.

Parameters2

input	string	required	JSON array of objects to convert to a Markdown table
columns	array	optional	Column names and order (default: all keys from first row)

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "JSON array of objects to convert to a Markdown table"
    },
    "columns": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Column names and order (default: all keys from first row)"
    }
  },
  "required": [
    "input"
  ]
}

openapi_validate

Validate the structure of an OpenAPI 3.x specification (JSON or YAML). Checks required top-level fields (openapi, info.title, info.version, paths), validates each operation (responses, operationId uniqueness), detects undeclared $ref components, and flags missing 2xx responses. Returns a PASS/FAIL verdict, a 0–100 compliance score, and a list of errors and warnings with JSON-pointer locations. Use before publishing an API spec or generating SDK code.

Parameters1

input

string

required

OpenAPI 3.x specification as a JSON or YAML string

Raw schema

{
  "type": "object",
  "properties": {
    "input": {
      "type": "string",
      "description": "OpenAPI 3.x specification as a JSON or YAML string"
    }
  },
  "required": [
    "input"
  ]
}

post_jira_comment

Post the output of jira_to_test_suite as a formatted comment on the source Jira ticket. Converts Gherkin, E2E steps, API tests, and ambiguities into Atlassian Document Format (ADF). STATEFUL — creates a comment on the issue.

Parameters5

issue_key	string	required	Jira issue key, e.g. "PROJ-123"
jira_base_url	string	required	Atlassian base URL
jira_email	string	required	Atlassian account email
jira_token	string	required	Atlassian API token
test_suite	object	required	The test_suite object from jira_to_test_suite result

Raw schema

{
  "type": "object",
  "properties": {
    "issue_key": {
      "type": "string",
      "description": "Jira issue key, e.g. \"PROJ-123\""
    },
    "jira_base_url": {
      "type": "string",
      "description": "Atlassian base URL"
    },
    "jira_email": {
      "type": "string",
      "description": "Atlassian account email"
    },
    "jira_token": {
      "type": "string",
      "description": "Atlassian API token"
    },
    "test_suite": {
      "type": "object",
      "additionalProperties": true,
      "description": "The test_suite object from jira_to_test_suite result"
    }
  },
  "required": [
    "issue_key",
    "jira_base_url",
    "jira_email",
    "jira_token",
    "test_suite"
  ]
}

create_confluence_page

Create a new Confluence page from the output of jira_to_test_suite. Formats Gherkin, E2E steps, API tests, and test data as a properly structured Confluence page with code blocks and tables. STATEFUL — creates a new page in the specified space.

Parameters9

confluence_base_url	string	required	Atlassian base URL
confluence_email	string	required	Atlassian account email
confluence_token	string	required	Atlassian API token
space_key	string	required	Confluence space key where the page will be created, e.g. "QA", "ENG"
title	string	optional	Page title. Defaults to "Test Plan: {issue_key}"
parent_page_id	string	optional	Optional parent page ID — page will be created as a child of this page
test_suite	object	required	The test_suite object from jira_to_test_suite result
issue_key	string	optional	Source Jira issue key (for the page title and source link)
issue_url	string	optional	Source Jira issue URL (added as a link in the page)

Raw schema

{
  "type": "object",
  "properties": {
    "confluence_base_url": {
      "type": "string",
      "description": "Atlassian base URL"
    },
    "confluence_email": {
      "type": "string",
      "description": "Atlassian account email"
    },
    "confluence_token": {
      "type": "string",
      "description": "Atlassian API token"
    },
    "space_key": {
      "type": "string",
      "description": "Confluence space key where the page will be created, e.g. \"QA\", \"ENG\""
    },
    "title": {
      "type": "string",
      "description": "Page title. Defaults to \"Test Plan: {issue_key}\""
    },
    "parent_page_id": {
      "type": "string",
      "description": "Optional parent page ID — page will be created as a child of this page"
    },
    "test_suite": {
      "type": "object",
      "additionalProperties": true,
      "description": "The test_suite object from jira_to_test_suite result"
    },
    "issue_key": {
      "type": "string",
      "description": "Source Jira issue key (for the page title and source link)"
    },
    "issue_url": {
      "type": "string",
      "description": "Source Jira issue URL (added as a link in the page)"
    }
  },
  "required": [
    "confluence_base_url",
    "confluence_email",
    "confluence_token",
    "space_key",
    "test_suite"
  ]
}

fetch_jira_issue

Fetch a complete Jira issue: summary, description converted to Markdown, status, assignee, priority, labels, custom fields, and optionally comments and attachment metadata. BYOK — credentials transit in-memory only, never stored on ia-qa.com.

Parameters7

issue_key	string	required	Jira issue key, e.g. "PROJ-123"
jira_base_url	string	required	Atlassian base URL, e.g. "https://mycompany.atlassian.net"
jira_email	string	required	Atlassian account email
jira_token	string	required	Atlassian API token (from id.atlassian.com > Security > API tokens)
fields	array	optional	Specific Jira field names to return. Omit for all standard fields.
include_comments	boolean	optional	Include issue comments, up to 20 (default: true)
include_attachments	boolean	optional	Include attachment metadata list (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "issue_key": {
      "type": "string",
      "description": "Jira issue key, e.g. \"PROJ-123\""
    },
    "jira_base_url": {
      "type": "string",
      "description": "Atlassian base URL, e.g. \"https://mycompany.atlassian.net\""
    },
    "jira_email": {
      "type": "string",
      "description": "Atlassian account email"
    },
    "jira_token": {
      "type": "string",
      "description": "Atlassian API token (from id.atlassian.com > Security > API tokens)"
    },
    "fields": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Specific Jira field names to return. Omit for all standard fields."
    },
    "include_comments": {
      "type": "boolean",
      "description": "Include issue comments, up to 20 (default: true)"
    },
    "include_attachments": {
      "type": "boolean",
      "description": "Include attachment metadata list (default: false)"
    }
  },
  "required": [
    "issue_key",
    "jira_base_url",
    "jira_email",
    "jira_token"
  ]
}

search_jira_issues

Search Jira using JQL (Jira Query Language). Returns matching issues with key fields. Ideal for finding open bugs, sprint tickets, or issues by label/assignee/component. BYOK — credentials transit in-memory only, never stored.

Parameters6

jql	string	required	JQL query string, e.g. "project = PROJ AND status = Open AND assignee = currentUser() ORDER BY priority DESC"
jira_base_url	string	required	Atlassian base URL, e.g. "https://mycompany.atlassian.net"
jira_email	string	required	Atlassian account email
jira_token	string	required	Atlassian API token
max_results	number	optional	Max issues to return (default: 10, max: 50)
fields	array	optional	Fields per issue. Default: summary, status, assignee, priority, issuetype, labels, created, updated

Raw schema

{
  "type": "object",
  "properties": {
    "jql": {
      "type": "string",
      "description": "JQL query string, e.g. \"project = PROJ AND status = Open AND assignee = currentUser() ORDER BY priority DESC\""
    },
    "jira_base_url": {
      "type": "string",
      "description": "Atlassian base URL, e.g. \"https://mycompany.atlassian.net\""
    },
    "jira_email": {
      "type": "string",
      "description": "Atlassian account email"
    },
    "jira_token": {
      "type": "string",
      "description": "Atlassian API token"
    },
    "max_results": {
      "type": "number",
      "description": "Max issues to return (default: 10, max: 50)"
    },
    "fields": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Fields per issue. Default: summary, status, assignee, priority, issuetype, labels, created, updated"
    }
  },
  "required": [
    "jql",
    "jira_base_url",
    "jira_email",
    "jira_token"
  ]
}

jira_to_test_suite

Transform a Jira ticket into a complete test suite: Gherkin scenarios, E2E steps, API test cases, test data matrix, and ambiguity detection. Accepts either Jira credentials (auto-fetch) or a pre-fetched issue object. The returned test_suite includes _gherkin_warnings (deterministic syntax validation — empty if clean). Requires BYOK LLM key (OpenAI, Anthropic, etc.).

Parameters9

issue_key	string	optional	Jira issue key to fetch automatically, e.g. "PROJ-123". Required if issue is not provided.
jira_base_url	string	optional	Atlassian base URL. Required for auto-fetch mode.
jira_email	string	optional	Atlassian account email. Required for auto-fetch mode.
jira_token	string	optional	Atlassian API token. Required for auto-fetch mode.
issue	object	optional	Pre-fetched issue object from fetch_jira_issue, OR a mock object with fields: key, summary, description (plain text or Markdown), status, issue_type, priority, labels, comments. Use this for offline/CI testing without Jira credentials.
confluence_pages	array	optional	Optional array of pre-fetched Confluence page objects from fetch_confluence_page, used as documentation context.
api_key	string	required	Your LLM provider API key (OpenAI sk-, Anthropic sk-ant-, Google AIzaSy-, etc.).
model	string	required	LLM model to use, e.g. "gpt-4o-mini", "claude-3-5-haiku-20241022", "gemini-2.0-flash".
max_tokens	integer	optional	Maximum tokens for the LLM response. Default: 8192. Increase for large tickets with many ACs; decrease to reduce cost on simple tickets.

Raw schema

{
  "type": "object",
  "properties": {
    "issue_key": {
      "type": "string",
      "description": "Jira issue key to fetch automatically, e.g. \"PROJ-123\". Required if issue is not provided."
    },
    "jira_base_url": {
      "type": "string",
      "description": "Atlassian base URL. Required for auto-fetch mode."
    },
    "jira_email": {
      "type": "string",
      "description": "Atlassian account email. Required for auto-fetch mode."
    },
    "jira_token": {
      "type": "string",
      "description": "Atlassian API token. Required for auto-fetch mode."
    },
    "issue": {
      "type": "object",
      "additionalProperties": true,
      "description": "Pre-fetched issue object from fetch_jira_issue, OR a mock object with fields: key, summary, description (plain text or Markdown), status, issue_type, priority, labels, comments. Use this for offline/CI testing without Jira credentials."
    },
    "confluence_pages": {
      "type": "array",
      "items": {
        "type": "object",
        "additionalProperties": true
      },
      "description": "Optional array of pre-fetched Confluence page objects from fetch_confluence_page, used as documentation context."
    },
    "api_key": {
      "type": "string",
      "description": "Your LLM provider API key (OpenAI sk-, Anthropic sk-ant-, Google AIzaSy-, etc.)."
    },
    "model": {
      "type": "string",
      "description": "LLM model to use, e.g. \"gpt-4o-mini\", \"claude-3-5-haiku-20241022\", \"gemini-2.0-flash\"."
    },
    "max_tokens": {
      "type": "integer",
      "description": "Maximum tokens for the LLM response. Default: 8192. Increase for large tickets with many ACs; decrease to reduce cost on simple tickets.",
      "default": 8192
    }
  },
  "required": [
    "api_key",
    "model"
  ]
}

fix_gherkin

Fix Gherkin syntax warnings from a jira_to_test_suite result. Takes the current gherkin text and the _gherkin_warnings array, calls your LLM to fix ONLY the flagged issues (adds missing Given/When/Then steps, etc.), and returns the corrected Gherkin. Lightweight — uses ~300-500 tokens vs ~5k for a full regeneration. Requires BYOK LLM key.

Parameters4

gherkin	string	required	The current Gherkin text from the jira_to_test_suite result (test_suite.gherkin).
warnings	array	required	The _gherkin_warnings array from the jira_to_test_suite result.
api_key	string	required	Your LLM provider API key.
model	string	required	LLM model to use for the fix, e.g. "gpt-4o-mini".

Raw schema

{
  "type": "object",
  "properties": {
    "gherkin": {
      "type": "string",
      "description": "The current Gherkin text from the jira_to_test_suite result (test_suite.gherkin)."
    },
    "warnings": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "The _gherkin_warnings array from the jira_to_test_suite result."
    },
    "api_key": {
      "type": "string",
      "description": "Your LLM provider API key."
    },
    "model": {
      "type": "string",
      "description": "LLM model to use for the fix, e.g. \"gpt-4o-mini\"."
    }
  },
  "required": [
    "gherkin",
    "warnings",
    "api_key",
    "model"
  ]
}

fetch_confluence_page

Fetch a Confluence page and return its content as clean Markdown. Accepts a numeric page_id or a full page URL. Optionally lists direct child pages. BYOK — credentials transit in-memory only, never stored.

Parameters6

page_id	string	optional	Confluence page ID (numeric string), e.g. "123456789"
page_url	string	optional	Full Confluence page URL (alternative to page_id), e.g. "https://mycompany.atlassian.net/wiki/spaces/ENG/pages/123456789"
confluence_base_url	string	required	Atlassian base URL, e.g. "https://mycompany.atlassian.net"
confluence_email	string	required	Atlassian account email (same credentials as Jira)
confluence_token	string	required	Atlassian API token
include_children	boolean	optional	List direct child pages (id + title) (default: false)

Raw schema

{
  "type": "object",
  "properties": {
    "page_id": {
      "type": "string",
      "description": "Confluence page ID (numeric string), e.g. \"123456789\""
    },
    "page_url": {
      "type": "string",
      "description": "Full Confluence page URL (alternative to page_id), e.g. \"https://mycompany.atlassian.net/wiki/spaces/ENG/pages/123456789\""
    },
    "confluence_base_url": {
      "type": "string",
      "description": "Atlassian base URL, e.g. \"https://mycompany.atlassian.net\""
    },
    "confluence_email": {
      "type": "string",
      "description": "Atlassian account email (same credentials as Jira)"
    },
    "confluence_token": {
      "type": "string",
      "description": "Atlassian API token"
    },
    "include_children": {
      "type": "boolean",
      "description": "List direct child pages (id + title) (default: false)"
    }
  },
  "required": [
    "confluence_base_url",
    "confluence_email",
    "confluence_token"
  ]
}

rate_tool

Give honest usage feedback on an IA-QA MCP tool. Provide a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Ratings are aggregated on a public dashboard at /devtools/mcp-ratings. Skip rating routine successes — we want signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })

Parameters3

tool_name	string	required	Name of the MCP tool to rate (e.g. "format_json", "shield_analyze")
score	number	required	Rating from 1 (poor) to 5 (excellent)
comment	string	optional	Strongly encouraged — explain what you were trying to do and whether the tool got you there. Be specific about what was missing, wrong, or a poor fit. This is the most valuable part of the rating (max 500 chars).

Raw schema

{
  "type": "object",
  "properties": {
    "tool_name": {
      "type": "string",
      "description": "Name of the MCP tool to rate (e.g. \"format_json\", \"shield_analyze\")"
    },
    "score": {
      "type": "number",
      "description": "Rating from 1 (poor) to 5 (excellent)",
      "minimum": 1,
      "maximum": 5
    },
    "comment": {
      "type": "string",
      "description": "Strongly encouraged — explain what you were trying to do and whether the tool got you there. Be specific about what was missing, wrong, or a poor fit. This is the most valuable part of the rating (max 500 chars)."
    }
  },
  "required": [
    "tool_name",
    "score"
  ]
}

Install

Tools (149)

format_jsongenerate_uuidhash_textcount_tokensbase64_encodebase64_decodeurl_encodeurl_decode

Remote endpoint

Streamable HTTP

Hosted server - connect over the network, no local install.

url

https://www.ia-qa.com/mcp

Add to VS Code Add to Cursor

claude_desktop_config.json

json

{
  "mcpServers": {
    "ia-qa-toolbox": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://www.ia-qa.com/mcp"
      ]
    }
  }
}

Desktop config is stdio-only; this bridges via mcp-remote. Native remote: Settings > Connectors.