io.github.JcJamet/ia-qa-toolbox
Official147 toolsIA-QA — 130+ QA & Dev Tools for AI Agents
130+ QA & dev tools for AI agents: prompt injection, RAG testing, VLM eval, guardrails. Free.
130+ QA tools for AI agents including prompt injection, RAG, and model evaluation.
Captured live from the server via tools/list.
format_json
Format, validate, and pretty-print a JSON string. Returns the formatted JSON or a detailed parse error.
Parameters (2)
- inputstringrequired
Raw JSON string to format
- indentnumber
Indent size (default: 2)
generate_uuid
Generate one or more cryptographically random UUID v4 identifiers. Use this when you need unique IDs for test fixtures, database records, session tokens, or any scenario requiring a guaranteed-unique string. Returns up to 100 UUIDs in one call.
Parameters (1)
- countnumber
Number of UUIDs to generate (1–100, default: 1)
hash_text
Compute a cryptographic hash of a text string. Use when you need to verify data integrity, generate content fingerprints, hash passwords (prefer SHA-256+), or produce a fixed-length digest of any input. Supports SHA-256 (default), SHA-512, SHA-1, and MD5.
Parameters (2)
- inputstringrequired
Text to hash
- algorithmstring
Hash algorithm: sha256 (default), sha512, sha1, md5
count_tokens
Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it fits within the model context window and to estimate cost. Returns token estimate, character count, and word count.
Parameters (1)
- inputstringrequired
Text to count tokens for
base64_encode
Encode a UTF-8 string to Base64. Use when you need to embed binary data, multi-line text, or special characters safely inside JSON fields, HTTP headers, or data URIs.
Parameters (1)
- inputstringrequired
Text to encode
base64_decode
Decode a Base64 string back to UTF-8 text. Use for inspecting Base64-encoded API responses, JWT payload claims, config file values, or attachment data.
Parameters (1)
- inputstringrequired
Base64 string to decode
url_encode
Percent-encode a string for safe use in URLs. Call this before programmatically building query strings, path segments, or form-encoded bodies to prevent injection and malformed URLs.
Parameters (2)
- inputstringrequired
String to URL-encode
- modestring
"component" (default) or "full" for encodeURI behavior
url_decode
Decode a percent-encoded URL string back to plain text. Use when parsing query parameters from raw URLs or when displaying encoded values to users.
Parameters (1)
- inputstringrequired
URL-encoded string to decode
generate_slug
Convert any string into a URL-friendly slug: lowercase, ASCII-normalized (é→e), special characters removed, spaces replaced with hyphens. Use for generating SEO-friendly URL paths, file names, or identifier keys from user-provided titles or labels.
Parameters (2)
- inputstringrequired
String to slugify
- separatorstring
Separator character (default: "-")
validate_email
Validate an email address against RFC 5322 syntax before storing it, sending a transactional email, or adding it to a mailing list. Returns { valid, email } — use this to avoid bounces and malformed data.
Parameters (1)
- emailstringrequired
Email address to validate
minify_js
Minify a JavaScript snippet, function, class, or module up to 50 KB using Terser. Returns minified code and byte savings. Use when embedding scripts in HTML templates, report payloads, or injecting inline code programmatically.
Parameters (1)
- codestringrequired
JavaScript code to minify (max 50kb)
decode_jwt
Decode a JWT (JSON Web Token) and return its header and payload without verifying the signature. Also reports whether the token is expired and the exact expiry date. Use to inspect claims (sub, iss, exp, roles) during debugging or when integrating with an auth provider.
Parameters (1)
- tokenstringrequired
The JWT string to decode (header.payload.signature)
text_stats
Compute comprehensive statistics for any text: character count (with and without spaces), word count, line count, sentence count, paragraph count, and estimated reading time in minutes. Use for validating form field lengths, evaluating LLM output verbosity, or content auditing.
Parameters (1)
- inputstringrequired
The text to analyse
generate_password
Generate a cryptographically secure random password using crypto.randomBytes. Configurable length (4–128), uppercase letters, digits, and symbols. Use when resetting user passwords, seeding test accounts, or generating API secrets.
Parameters (4)
- lengthnumber
Password length (4–128, default: 16)
- uppercaseboolean
Include uppercase letters (default: true)
- numbersboolean
Include digits (default: true)
- symbolsboolean
Include symbols like !@#$ (default: false)
parse_csv
Parse a CSV string into a JSON array of objects (or raw arrays). Handles RFC 4180 quoted fields, escaped quotes, and custom delimiters. Use when processing spreadsheet exports, data imports, or structured text pipelines where the source is CSV. Supports up to 200 KB.
Parameters (3)
- inputstringrequired
CSV content to parse
- delimiterstring
Field delimiter character (default: ",")
- headerboolean
Treat the first row as headers (default: true)
color_convert
Convert a color between HEX, RGB, and HSL formats. Use when translating design tokens between CSS notations, verifying color accessibility, or normalizing color values from user input. Accepts #rrggbb, #rgb, rgb(r,g,b), or hsl(h,s%,l%).
Parameters (1)
- inputstringrequired
Color value to convert, e.g. "#ff6b6b", "rgb(255,107,107)", "hsl(0,100%,71%)"
regex_test
Test a regular expression pattern against an input string and return all matches with their index positions and named capture groups. Use for validating user inputs, extracting structured data from text, or debugging regex patterns. Supports flags g, i, m, s, u, y.
Parameters (3)
- patternstringrequired
Regular expression pattern (without delimiters)
- inputstringrequired
The string to test against (max 50 KB)
- flagsstring
Regex flags: g (global), i (case-insensitive), m (multiline), s (dotAll) — default: ""
lorem_ipsum
Generate Lorem Ipsum placeholder text for UI mockups, design prototypes, or test data population. Configurable paragraphs (1–10), sentences per paragraph (1–20), and approximate words per sentence (3–30).
Parameters (3)
- paragraphsnumber
Number of paragraphs to generate (1–10, default: 1)
- sentences_per_paragraphnumber
Sentences per paragraph (1–20, default: 5)
- words_per_sentencenumber
Approximate words per sentence (3–30, default: 10)
timestamp_convert
Convert between Unix timestamps (seconds or milliseconds) and ISO-8601 / UTC date strings. Auto-detects epoch vs. millisecond format. Omit input to get the current time. Returns iso, unix_s, unix_ms, utc, date, and time fields.
Parameters (1)
- inputany
Unix timestamp (number, seconds or ms) or ISO date string. Omit to get the current time.
diff_text
Compute a unified line-by-line diff between two text strings (LCS algorithm). Returns added/removed/unchanged line counts and formatted diff hunks with configurable context lines (0–20). Use to compare versions of prompts, configs, code snippets, or any text where you need to see exactly what changed.
Parameters (3)
- astringrequired
Original (before) text
- bstringrequired
Modified (after) text
- contextnumber
Context lines around each change (0–20, default: 3)
truncate_to_tokens
Truncate text to at most N tokens (cl100k_base: ~4 chars/token) to avoid exceeding an LLM context window. Optionally keeps the end of the text instead of the start (useful for keeping recent conversation history). Reports whether truncation occurred and the estimated token count.
Parameters (3)
- inputstringrequired
Text to truncate
- max_tokensnumberrequired
Maximum number of tokens to keep
- from_endboolean
Keep the end of the text instead of the start (default: false)
split_chunks
Split text into chunks of at most N tokens (cl100k_base: ~4 chars/token) with optional overlap. Designed for RAG ingestion pipelines.
Parameters (3)
- inputstringrequired
Text to split into chunks
- chunk_tokensnumberrequired
Maximum tokens per chunk (10–8000)
- overlapnumber
Token overlap between consecutive chunks (default: 0)
extract_json_from_text
Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON. Call this whenever an LLM returns structured data mixed with explanation text instead of raw JSON.
Parameters (1)
- inputstringrequired
Raw text (e.g., LLM output) that may contain a JSON object or array
strip_markdown
Strip all Markdown formatting (headers, bold, italic, code fences, links, lists) from text and return clean plain text. Run this before injecting scraped documentation, README files, or user content into an LLM prompt to eliminate redundant markup tokens and reduce cost.
Parameters (1)
- inputstringrequired
Markdown text to convert to plain text
estimate_llm_cost
Estimate the API cost in USD for a given model and token counts. Supports all major 2024–2026 models: GPT-4o, GPT-4.1, o3, o4-mini, Claude Opus 4, Claude Sonnet 4/4.5, Gemini 2.5 Pro/Flash, DeepSeek V3/R1, Grok 3, and legacy models.
Parameters (3)
- modelstringrequired
Model name, e.g. "gpt-4o", "claude-3.5-sonnet", "deepseek-v3"
- input_tokensnumberrequired
Number of input/prompt tokens
- output_tokensnumber
Number of output/completion tokens (default: 0)
escape_html
Escape HTML special characters (&, <, >, ", ') to their safe HTML entities. ALWAYS call this before inserting any user-provided or LLM-generated content into an HTML template to prevent cross-site scripting (XSS) attacks.
Parameters (1)
- inputstringrequired
String to HTML-escape
unescape_html
Convert HTML entities (&, <, >, ", ', and numeric &#NNN;) back to plain characters. Use when processing HTML-encoded text from APIs, email content, or legacy database fields before passing to an LLM or displaying to users.
Parameters (1)
- inputstringrequired
HTML-encoded string to unescape
fetch_veille_feed
Fetch the latest QA & AI/LLM articles aggregated from curated RSS sources (Google Testing Blog, DEV.to Testing/QA/AI/LLM/Agents, Hugging Face Blog, Simon Willison). Perfect for agents monitoring the QA & AI landscape.
Parameters (2)
- categorystring
Filter: "qa" (testing/quality), "ai" (AI/LLM/agents), "all" (default — both)
- limitnumber
Max articles to return (default: 20, max: 50)
score_geo_signals
Analyze a webpage <head> HTML (or full HTML) for GEO (Generative Engine Optimization) signals. Returns a score /60 with per-check results and improvement tips. GEO = optimizing pages for AI-powered search engines (ChatGPT Search, Perplexity, etc.).
Parameters (1)
- head_htmlstringrequired
Raw HTML of the <head> section (or full page HTML) to analyze
extract_json_path
Extract a value from a JSON string using dot-notation path (e.g., "user.address.city", "items.0.name", "meta.tags"). Supports array index access via numeric path segments.
Parameters (2)
- inputstringrequired
A valid JSON string to traverse
- pathstringrequired
Dot-notation path, e.g. "user.address.city" or "items.0.name"
generate_json_ld
Generate a ready-to-paste <script type="application/ld+json"> snippet for GEO / structured data optimization. Supported types: WebSite, FAQPage, Article, Person, Organization, SoftwareApplication, HowTo.
Parameters (3)
- typestringrequired
Schema @type: "WebSite", "FAQPage", "Article", "Person", "Organization", "SoftwareApplication", "HowTo"
- fieldsobject
Schema fields as key-value pairs (name, url, description, author, datePublished, etc.)
- faq_itemsarray
For FAQPage/HowTo: array of { question, answer } objects
analyze_diff_bugs
Detect potential bugs and code smells from a git diff or two code versions. Returns a list of issues with severity levels and test suggestions.
Parameters (3)
- version1string
Original code (before changes). If omitted, only the new version is analysed.
- version2stringrequired
New/modified code (after changes)
- contextstring
Optional PR title or feature context for better analysis
generate_test_cases
Generate a set of test cases (valid, edge, invalid) for a given feature description. Returns test matrix with Gherkin scenarios ready to use.
Parameters (2)
- featurestringrequired
Feature or function to test. Be specific: describe inputs, expected behaviour, context.
- inputsstring
Optional: list of input parameters (one per line, e.g. "email: string [required]")
run_pr_gate_pipeline
Full automated QA pipeline for a pull request. Takes a unified git diff (output of `git diff HEAD`) and returns: bug hotspots, regression impact areas, risk score (0–100), generated test cases, severity assessment, and a merge recommendation (PASS / CONDITIONAL / BLOCK). This is the highest-value QA tool — use it when reviewing any code change.
Parameters (2)
- git_diffstringrequired
Unified git diff (output of `git diff HEAD` or copied from GitHub diff view)
- contextstring
Optional PR title or description for richer analysis
validate_mcp_response
Validate that an MCP tool response conforms to expected format, schema, and content rules. Use this to QA-test any MCP server tool. Supply the tool's actual JSON result and a set of checks to perform.
Parameters (8)
- responsestringrequired
The MCP tool result as a JSON string to validate
- expected_typestring
Expected top-level type: "object", "array", "string", "number"
- required_keysstring
Comma-separated list of keys that MUST exist in the response (dot-notation for nested: "data.id, data.name")
- forbidden_keysstring
Comma-separated list of keys that MUST NOT exist (e.g. "password, secret, token")
- max_response_msnumber
Maximum acceptable latency in ms (will be compared if provided)
- actual_latencynumber
Actual measured latency in ms (from the call)
- min_itemsnumber
If response is an array, minimum number of items expected
- max_size_bytesnumber
Maximum acceptable response size in bytes
llm_output_validator
Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.
Parameters (9)
- outputstringrequired
The LLM output text to validate
- expected_formatstring
Expected output format
- must_includestring
Comma-separated strings that MUST appear in the output
- must_not_includestring
Comma-separated strings that must NOT appear (e.g. "TODO, FIXME, undefined, NaN")
- max_lengthnumber
Maximum character length for the output
- min_lengthnumber
Minimum character length for the output
- expected_languagestring
Expected language of the output (en, fr, es, de…). Checks for common words.
- check_safetyboolean
Check for PII patterns (emails, phones, SSN), profanity signals, and prompt leakage
- check_json_schemastring
If expected_format is JSON, provide required keys as comma-separated list to validate the structure
compare_responses
Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.
Parameters (9)
- response_astringrequired
First output (e.g. model A's answer)
- response_bstringrequired
Second output (e.g. model B's answer)
- label_astring
Label for output A (e.g. "GPT-4o", "v1.0")
- label_bstring
Label for output B (e.g. "GPT-5-nano", "v1.1")
- taskstring
The task/prompt both outputs were answering — used by the LLM judge for context
- referencestring
Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic).
- check_jsonboolean
Try to parse as JSON and compare structurally (keys, types, values)
- modelstring
Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner.
- api_keystring
Optional API key for the judge model (BYOK). Used only for the judge call; never stored.
prompt_test_suite
Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.
Parameters (10)
- system_promptstringrequired
The system prompt under test
- user_promptstringrequired
The user prompt to send
- expected_behaviorstring
Description of what the LLM should do (free text)
- expected_formatstring
Expected output format
- must_includestring
Required content (comma-separated)
- must_not_includestring
Forbidden content (comma-separated)
- max_tokensnumber
Max token budget for the test
- temperaturenumber
Temperature to use
- check_safetyboolean
Include safety/PII checks in the rubric
- adversarial_promptsboolean
Auto-generate adversarial test variants (jailbreak, injection, edge cases)
mcp_server_health_check
Generate a health check report for an MCP server's tool manifest. Validates tool definitions, schema quality, naming conventions, and documentation completeness. Paste the server manifest JSON to audit.
Parameters (2)
- manifeststringrequired
MCP server manifest JSON (the response from GET /mcp or tools/list)
- strictboolean
Enable strict mode: also check for optional best practices (examples, default values, descriptions > 20 chars)
mcp_server_evaluate
Run a full compliance evaluation against a live MCP server URL. Tests: server reachability (ping), manifest discovery (GET /mcp), schema quality (snake_case names, descriptions, inputSchema), JSON-RPC 2.0 test call, and P50/P95 latency. Returns a PASS/FIX/BLOCK verdict with a 0-100 score and per-check details.
Parameters (2)
- urlstringrequired
Base URL of the MCP server (e.g. https://ia-qa.com or http://localhost:3001)
- test_tool_namestring
Specific tool name to use in the JSON-RPC test call (defaults to the first tool in the manifest)
json_schema_validate
Validate a JSON value against a JSON Schema (draft-07 subset). Supports type, required, properties, items, enum, const, pattern, format (email/uri/date), minimum/maximum, minLength/maxLength, minItems/maxItems, uniqueItems, additionalProperties, anyOf, allOf, oneOf. Returns all validation errors with dot-notation paths.
Parameters (2)
- valuestringrequired
JSON string to validate
- schemastringrequired
JSON Schema as a JSON string
flatten_json
Flatten a nested JSON object to single-level dot-notation keys (e.g. {"a":{"b":1}} → {"a.b":1}), or unflatten dot-notation keys back to a nested object. Supports custom separators.
Parameters (3)
- inputstringrequired
JSON string to flatten or unflatten
- modestring
"flatten" (default) or "unflatten"
- separatorstring
Key separator (default: ".")
xml_to_json
Convert an XML string to a JSON object. Supports attributes, nested elements, arrays, CDATA, and namespaces. Options: parse numbers, parse booleans, ignore attributes.
Parameters (4)
- inputstringrequired
XML string to convert
- parse_valuesboolean
Auto-parse numbers and booleans (default: true)
- ignore_attrsboolean
Ignore XML attributes (default: false)
- attr_prefixstring
Prefix for attribute keys (default: "@_")
redact_pii
Automatically detect and redact Personally Identifiable Information (PII) from text. Replaces emails, phone numbers, SSNs, credit cards, IP addresses, and JWT tokens with [REDACTED_TYPE] placeholders. Safe to use before logging or sending to an LLM.
Parameters (3)
- inputstringrequired
Text to redact PII from
- typesstring
Comma-separated types to redact (default: all). Options: email, phone, ssn, credit_card, ip_address, jwt
- markerstring
Custom replacement marker (default: "REDACTED"). Result: [REDACTED_EMAIL]
mock_from_schema
Generate realistic mock data from a JSON Schema. Supports all common types (string, number, integer, boolean, array, object, null), format hints (email, date, date-time, uri, uuid), enum, const, and nested schemas. Perfect for testing MCP tools with realistic data.
Parameters (3)
- schemastringrequired
JSON Schema as a JSON string
- countnumber
Number of mock objects to generate (default: 1, max: 20)
- seedstring
Optional seed string for deterministic output (uses first char codes)
transform_json_array
Transform a JSON array using common operations: pluck (extract specific fields), filter (by field value), sort_by (field), group_by (field), count_by (field), uniq_by (field). Useful for processing MCP tool results and LLM structured outputs.
Parameters (9)
- inputstringrequired
JSON string containing an array (or object with an array at path)
- pathstring
Optional dot-notation path to the array within the JSON object (e.g. "data.items")
- operationstringrequired
Operation: "pluck", "filter", "sort_by", "group_by", "count_by", "uniq_by", "reverse", "first_n", "last_n", "flatten"
- fieldstring
Field to operate on (for sort_by, group_by, count_by, uniq_by, filter)
- fieldsstring
Comma-separated field list for "pluck" (e.g. "id,name,email")
- filter_opstring
For "filter": "==" | "!=" | ">" | ">=" | "<" | "<=" | "contains" | "exists" | "!exists"
- filter_valuestring
For "filter": value to compare against
- nnumber
For first_n / last_n: number of items
- sort_orderstring
For sort_by: "asc" (default) or "desc"
json_to_csv
Convert a JSON array of objects to CSV format. Automatically detects columns from all object keys. Handles quoting and escaping per RFC 4180.
Parameters (3)
- inputstringrequired
JSON string containing an array of objects
- delimiterstring
Column delimiter (default: ",")
- headersboolean
Include header row (default: true)
case_convert
Convert a string between naming conventions: camelCase, PascalCase, snake_case, kebab-case, UPPER_SNAKE_CASE, dot.case, Title Case. Essential for code generation and refactoring.
Parameters (2)
- inputstringrequired
String to convert (e.g., "myVariableName", "my-css-class")
- tostringrequired
Target case: "camel", "pascal", "snake", "kebab", "upper_snake", "dot", "title"
sort_lines
Sort, deduplicate, reverse, or filter lines of text. Useful for cleaning import lists, dependencies, log files, and config entries.
Parameters (5)
- inputstringrequired
Multi-line text to process
- operationstring
"sort" (default), "sort_desc", "reverse", "deduplicate", "unique_sort", "filter"
- filterstring
For "filter": keep lines containing this substring (case-insensitive)
- trimboolean
Trim whitespace from each line (default: true)
- remove_emptyboolean
Remove empty lines (default: true)
number_base_convert
Convert numbers between bases: decimal, binary, octal, hexadecimal, or any base 2–36. Auto-detects 0x, 0b, 0o prefixes.
Parameters (3)
- inputstringrequired
Number to convert (e.g., "255", "0xFF", "0b1010", "0o77")
- from_basenumber
Source base 2–36 (auto-detects prefix if omitted)
- to_basenumber
Target base 2–36 (omit to get all common bases)
validate_url
Parse and validate a URL. Returns decomposed components: protocol, hostname, port, path, query parameters, hash, and origin.
Parameters (1)
- inputstringrequired
URL to validate and parse
check_contrast_ratio
Calculate WCAG 2.1 contrast ratio between two colors. Returns ratio and compliance for AA/AAA normal and large text.
Parameters (2)
- foregroundstringrequired
Foreground color in hex (e.g., "#333333")
- backgroundstringrequired
Background color in hex (e.g., "#ffffff")
html_to_markdown
Convert HTML to clean Markdown. Strips scripts, styles, nav, ads, and comments. Converts headings, lists, links, images, code blocks. Ideal for preparing web content as LLM context.
Parameters (2)
- inputstringrequired
HTML string to convert
- strip_linksboolean
Strip link URLs, keep text only (default: false)
cron_parse
Parse a cron expression into a human-readable schedule description. Supports standard 5-field cron (minute hour day month weekday).
Parameters (1)
- expressionstringrequired
Cron expression (e.g., "0 9 * * 1-5", "*/15 * * * *")
cron_validator
Validate a 5-field cron expression, explain the schedule, and preview the next execution times. Use this to debug cron jobs before they reach production. Returns parsed fields, a human-readable description, and upcoming ISO timestamps.
Parameters (2)
- expressionstringrequired
Cron expression with 5 fields, e.g. "*/15 9-18 * * 1-5"
- next_runs_countnumber
How many upcoming runs to return (1-50, default: 10)
calculate_readability
Calculate readability scores: Flesch Reading Ease, Flesch-Kincaid Grade Level, Coleman-Liau Index, and Automated Readability Index. Useful for evaluating LLM output quality.
Parameters (1)
- inputstringrequired
Text to analyze for readability
normalize_whitespace
Normalize whitespace: trim trailing spaces, collapse blank lines, normalize line endings (LF/CRLF), convert tabs to spaces. Useful for cleaning code, configs, and text before processing.
Parameters (6)
- inputstringrequired
Text to normalize
- line_endingstring
"lf" (default), "crlf", or "cr"
- collapse_blanksboolean
Collapse 3+ consecutive blank lines to 2 (default: true)
- trim_linesboolean
Trim trailing whitespace from each line (default: true)
- trim_fileboolean
Trim leading/trailing blank lines (default: true)
- tab_to_spacesnumber
Convert tabs to N spaces (omit to keep tabs)
embedding_similarity
Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for true semantic similarity with vector embeddings, use run_semantic_tests with mode="embeddings" and your OpenAI API key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.
Parameters (4)
- text_astring
First text to compare (single-pair mode)
- text_bstring
Second text to compare (single-pair mode)
- batcharray
Batch mode: array of { text_a, text_b } pairs. Overrides text_a/text_b if provided.
- methodsarray
Algorithms to use (default: all three). Options: "bow", "tfidf", "ngram"
llm_format_check
Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.
Parameters (3)
- outputstringrequired
The LLM output to validate
- expected_formatstringrequired
Expected format
- regex_patternstring
Custom regex pattern (only when expected_format is "regex")
hallucination_check
Word-overlap based hallucination check: verifies if an LLM answer's words and numbers appear in the provided source/context. Fast, deterministic, no API key needed. Limitations: not semantic — does not understand synonyms or paraphrases. For true semantic grounding, use run_semantic_tests with embedding mode. Essential for quick RAG accuracy testing.
Parameters (3)
- answerstringrequired
The LLM-generated answer to verify
- contextstringrequired
The source/reference text that should ground the answer
- strictboolean
If true, every sentence in the answer must be supported (default: false)
prompt_injection_scan
Scan user input or prompts for common prompt injection patterns. Detects system prompt overrides, jailbreak attempts, role manipulation, encoding tricks, delimiter attacks, template/interpolation injection ({{...}}, ${...}), and context-exfiltration attempts ("repeat everything above").
Parameters (2)
- inputstringrequired
The user input or prompt to scan for injection patterns
- sensitivitystring
Detection sensitivity (default: medium)
token_budget_calculator
Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds model context window. Supports 25+ models.
Parameters (8)
- modelstringrequired
Model name (e.g. gpt-4o, claude-3.5-sonnet, gemini-2.0-flash)
- system_prompt_tokensnumber
Token count for system prompt
- user_input_tokensnumber
Token count for user message
- context_tokensnumber
Token count for RAG context / documents
- expected_output_tokensnumber
Expected max output tokens
- system_promptstring
Actual system prompt text (will estimate tokens)
- user_inputstring
Actual user input text (will estimate tokens)
- contextstring
Actual context text (will estimate tokens)
consistency_check
Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. For true semantic consistency, use run_semantic_tests with embedding mode. Essential for determinism testing.
Parameters (2)
- responsesarrayrequired
Array of 2+ LLM responses to compare (same prompt, different runs)
- check_factsboolean
Check for contradictory numbers/facts across responses (default: true)
llm_json_schema_check
Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.
Parameters (2)
- outputstringrequired
The LLM JSON output (raw string, will be parsed)
- schemaobjectrequired
JSON Schema (draft-07 subset) to validate against
latency_benchmark
Measure response time of one or more HTTP endpoints (GET/POST). Runs N iterations and returns min/max/avg/p95 latency. Useful for API and MCP server benchmarking.
Parameters (2)
- endpointsarrayrequired
Endpoints to benchmark
- iterationsnumber
Number of iterations per endpoint (default: 3, max: 10)
response_quality_score
Score an LLM response on multiple quality dimensions: relevance, completeness, clarity, conciseness, formatting. Returns a weighted 0-100 score with detailed breakdown.
Parameters (4)
- questionstringrequired
The original question/prompt
- responsestringrequired
The LLM response to score
- expected_keywordsarray
Keywords that should appear in a good answer
- max_lengthnumber
Ideal max character length (penalize if exceeded)
rag_relevance_rank
Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.
Parameters (3)
- querystringrequired
The user query
- chunksarrayrequired
Array of text chunks to rank
- top_knumber
Return top K results (default: all)
toxicity_scan
Scan text for toxic language, bias indicators, profanity, and harmful content categories. Returns risk scores per category. Useful for LLM safety guardrail testing.
Parameters (2)
- textstringrequired
Text to scan
- categoriesarray
Categories to check (default: all)
guardrail_test
Test an LLM response against a set of guardrail rules: must-include, must-not-include, max length, required format, language, forbidden patterns, and custom regex. Returns pass/fail per rule.
Parameters (2)
- responsestringrequired
The LLM response to test
- rulesarrayrequired
Array of guardrail rules to check
function_call_validate
Validate an LLM function call / tool_use output: check that function name is in allowed list, arguments match expected schema, no extra/missing args. For OpenAI function calling & MCP tool_use testing.
Parameters (2)
- function_callobjectrequired
The function call object from LLM (e.g. { "name": "get_weather", "arguments": {"city":"Paris"} })
- allowed_functionsarrayrequired
List of allowed function definitions
conversation_analyze
Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.
Parameters (1)
- messagesarrayrequired
Conversation messages in order
mcp_schema_lint
Lint an MCP tool definition for best practices: naming conventions, description quality, schema completeness, required fields consistency, description length. Returns actionable warnings.
Parameters (1)
- tool_definitionobjectrequired
MCP tool definition object with name, description, inputSchema
cot_analyzer
Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.
Parameters (2)
- reasoningstringrequired
The CoT / reasoning trace text (e.g. from <think> tags or step-by-step output)
- expected_conclusionstring
Expected final answer to check against (optional)
ab_test_report
Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns statistical comparison: mean, median, std deviation, winner, and improvement percentage.
Parameters (2)
- variant_aobjectrequired
First variant configuration with name and score array
- variant_bobjectrequired
Second variant configuration with name and score array
context_window_check
Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.
Parameters (3)
- messagesarrayrequired
Array of messages (system/user/assistant)
- modelstringrequired
Target model name (e.g. gpt-4o, claude-3.5-sonnet)
- max_output_tokensnumber
Reserved tokens for output (default: 4096)
vector_similarity
Compute similarity/distance between two float vectors: cosine similarity, dot product, Euclidean and Manhattan distance. Essential for vector DB relevance scoring, embedding evaluation, and nearest-neighbor testing.
Parameters (3)
- vector_aarrayrequired
First vector as array of floats
- vector_barrayrequired
Second vector as array of floats
- metricstring
Distance metric (default: all)
normalize_vector
L2-normalize a float vector (produce a unit vector with norm=1). Required by many vector DBs (Pinecone, Qdrant cosine). Supports batch normalization of up to 1000 vectors.
Parameters (2)
- vectorarray
Single vector to normalize
- batcharray
Batch of vectors to normalize (overrides vector)
vector_quantize
Simulate int8 or int4 quantization of float32 embedding vectors. Reduces storage by 4x (int8) or 8x (int4). Returns quantized values, scale factor, and precision loss (MSE). Useful for understanding vector DB compression trade-offs.
Parameters (2)
- vectorarrayrequired
Float32 vector to quantize
- bitsnumber
Quantization bits: 8 (int8, default) or 4 (int4)
vector_stats
Compute statistics for a float vector or matrix of vectors: mean, std, L2 norm, min, max, sparsity, top-K indices. Useful for debugging embedding quality and analyzing vector distributions in a vector DB.
Parameters (3)
- vectorarray
Single vector to analyze
- matrixarray
Matrix of vectors (overrides vector). Returns per-vector + matrix-level stats.
- top_knumber
Return indices of top K absolute values (default: 5)
bm25_score
Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.
Parameters (5)
- querystringrequired
The search query
- documentsarrayrequired
Array of documents to rank
- k1number
Term frequency saturation (default: 1.5)
- bnumber
Length normalization factor (default: 0.75)
- top_knumber
Return top K results (default: all)
build_rag_prompt
Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.
Parameters (6)
- querystringrequired
The user question to answer
- chunksarrayrequired
Retrieved context chunks with .text (required), .source (optional), .score (optional)
- system_instructionstring
Custom system instruction (default: standard RAG grounding instruction)
- max_context_tokensnumber
Max tokens for context section (default: 2000)
- cite_sourcesboolean
Add [1], [2] citation numbers (default: true)
- languagestring
Response language instruction (e.g. "French", "Spanish")
prompt_template_fill
Fill a prompt template with variables. Supports {{variable}} syntax and {{#if key}}...{{/if}} conditional blocks. Returns the filled prompt and lists unfilled variables.
Parameters (3)
- templatestringrequired
Prompt template with {{variable}} placeholders
- variablesobject
Key-value pairs to fill (e.g. {"name":"Alice","role":"engineer"})
- strictboolean
Throw error if any variable is not provided (default: false)
few_shot_formatter
Format few-shot examples for LLM prompts. Converts example pairs into formatted blocks. Supports chat format (User/Assistant), XML tags, Markdown, or plain text.
Parameters (4)
- examplesarrayrequired
Array of {input, output} pairs
- formatstring
Output format (default: chat)
- input_labelstring
Label for input (default: User / <input>)
- output_labelstring
Label for output (default: Assistant / <output>)
system_prompt_builder
Build a structured system prompt from components: role, task, constraints, output format, tone, language, and examples. Generates a production-ready system prompt with token estimate.
Parameters (7)
- rolestringrequired
Role/persona (e.g. "Senior QA Engineer", "JSON extraction assistant")
- taskstring
Main task or objective
- constraintsarray
Rules and constraints to follow
- output_formatstring
Expected output format description
- tonestring
Communication tone
- languagestring
Response language (e.g. "French")
- examplesstring
Brief examples to include
model_info
Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.
Parameters (1)
- modelstringrequired
Model name (e.g. "gpt-4o", "claude-3.5-sonnet", "gemini-2.5-pro")
compare_models
Compare 2-5 AI models side by side: context window, pricing, multimodal, reasoning capabilities, and provider. Returns a comparison table with a recommendation based on your use case.
Parameters (2)
- modelsarrayrequired
Array of 2-5 model names (e.g. ["gpt-4o","claude-3.5-sonnet","gemini-2.0-flash"])
- use_casestring
Optimize recommendation for this criterion
http_status_lookup
Look up detailed information about any HTTP status code: class, name, description, cacheability, typical causes, and handling best practices. Covers all standard 1xx-5xx codes.
Parameters (1)
- codenumberrequired
HTTP status code (e.g. 200, 404, 429, 503)
parse_http_headers
Parse a raw HTTP headers block into a structured JSON object. Detects multi-value headers, masks Authorization values, and optionally audits for missing security headers (HSTS, CSP, X-Frame-Options, etc.).
Parameters (2)
- headersstringrequired
Raw HTTP headers (one "Name: Value" per line)
- analyze_securityboolean
Audit for missing security headers (default: true)
generate_curl
Generate a curl command from request parameters. Supports GET/POST/PUT/DELETE, custom headers, JSON body, and form data. Useful for documentation, sharing, and debugging API calls.
Parameters (7)
- urlstringrequired
Request URL (must be http/https)
- methodstring
HTTP method (default: GET)
- headersobject
Request headers as key-value object
- bodystring
Raw request body string
- body_jsonobject
JSON body (auto-adds Content-Type: application/json)
- follow_redirectsboolean
Follow redirects with -L flag (default: true)
- verboseboolean
Add -v for verbose output (default: false)
extract_todos
Extract TODO, FIXME, HACK, BUG, NOTE, OPTIMIZE, and custom tags from any source code or text. Returns line numbers, tag types, and message text. Essential for technical debt auditing.
Parameters (3)
- inputstringrequired
Code or text to scan
- tagsarray
Custom tags to add (default set: TODO, FIXME, HACK, NOTE, BUG, OPTIMIZE, XXX)
- include_contextboolean
Include full line text (default: true)
detect_secrets
Scan code or config files for hardcoded secrets: AWS keys, GitHub tokens, OpenAI/Anthropic API keys, Stripe secrets, JWTs, database connection strings, and generic passwords. Returns findings with severity. Run before every commit.
Parameters (2)
- inputstringrequired
Code or config content to scan (max 500KB)
- filenamestring
Optional filename for context (e.g. ".env", "config.js")
count_code_lines
Count lines of code: total, code lines, comment lines, blank lines, and comment density. Supports JS/TS, Python, Java/C/C++, Ruby, Go, Shell, HTML/XML, and CSS.
Parameters (2)
- inputstringrequired
Source code to analyze
- languagestring
Language hint: "js", "ts", "py", "java", "c", "rb", "go", "sh", "html", "css" (auto-detect if omitted)
lint_commit_message
Validate a git commit message against the Conventional Commits spec (feat, fix, docs, style, refactor, test, chore, ci, perf, build). Returns compliance score, breaking change detection, and actionable suggestions.
Parameters (2)
- messagestringrequired
Git commit message to validate
- strictboolean
Enforce strict rules: max 72-char subject, imperative mood check (default: false)
word_frequency
Analyze word frequency in text. Returns top N words with counts and percentages. Supports English stopword filtering. Useful for content analysis, keyword extraction, and LLM output analysis.
Parameters (4)
- inputstringrequired
Text to analyze
- top_nnumber
Return top N words (default: 20, max: 200)
- min_lengthnumber
Minimum word length to include (default: 3)
- remove_stopwordsboolean
Remove common English stopwords (default: true)
extract_links
Extract all URLs, email addresses, and domain names from text. Returns categorized and deduplicated results. Useful for content auditing, link checking, and web scraping validation.
Parameters (2)
- inputstringrequired
Text to extract links from
- typesarray
Types to extract (default: all three)
levenshtein_distance
Compute the Levenshtein (edit) distance and normalized similarity ratio between two strings. Supports batch comparison. Useful for fuzzy string matching, deduplication, and test result comparison.
Parameters (4)
- astring
First string (single-pair mode)
- bstring
Second string (single-pair mode)
- batcharray
Batch of {a,b} pairs (max 50)
- case_insensitiveboolean
Ignore case differences (default: false)
json_diff
Compute a deep structural diff between two JSON values. Returns added, removed, and changed keys with dot-notation paths. Like git diff but for JSON objects — perfect for API response regression testing.
Parameters (3)
- beforestringrequired
Original JSON string (before)
- afterstringrequired
Modified JSON string (after)
- max_depthnumber
Max nesting depth to recurse (default: 10)
merge_json
Deep merge two JSON objects. Supports three array strategies: replace (default), concat, or unique (dedup concat). Nested objects are recursively merged — override takes precedence for primitives.
Parameters (3)
- basestringrequired
Base JSON object (will be merged into)
- overridestringrequired
Override JSON object (takes precedence)
- array_strategystring
Array merge strategy: replace (default), concat, or unique
json_to_yaml
Convert a JSON object to clean, human-readable YAML. Handles nested objects, arrays, multiline strings, and special characters. No external dependencies.
Parameters (2)
- inputstringrequired
JSON string to convert to YAML
- indentnumber
Indentation size in spaces (default: 2)
generate_hmac
Compute an HMAC signature for a message using a secret key. Supports SHA-256 (default), SHA-512, SHA-1, and MD5. Used for API request signing, webhook verification (GitHub, Stripe, Twilio), and JWT validation.
Parameters (4)
- messagestringrequired
Message to sign
- secretstringrequired
Secret key
- algorithmstring
Hash algorithm: sha256 (default), sha512, sha1, md5
- encodingstring
Output encoding (default: hex)
format_bytes
Convert raw byte counts to human-readable sizes in SI (KB=1000) or IEC (KiB=1024) units, or parse size strings back to bytes. Covers B, KB/KiB, MB/MiB, GB/GiB, TB/TiB, PB/PiB.
Parameters (3)
- bytesnumber
Number of bytes to format
- size_stringstring
Size string to parse to bytes (e.g. "1.5 GB", "512 MiB")
- standardstring
Output standard (default: both)
detect_language
Detect the natural language of a text using n-gram frequency analysis and common word markers. Supports 15 languages: English, French, Spanish, German, Italian, Portuguese, Dutch, Russian, Chinese, Japanese, Korean, Arabic, Polish, Turkish, Swedish.
Parameters (1)
- inputstringrequired
Text to detect language from (min 20 chars for accuracy)
pr_gatekeeper
Compound quality gate for pull requests. Runs three sequential checks: (1) secret detection — scans diff for API keys, tokens, passwords matching 16 regex patterns; (2) bug analysis — heuristic scan for eval(), innerHTML, empty catch, console.log, TODO/FIXME; (3) commit message linting against Conventional Commits spec. Returns gate verdict (PASS/WARN/BLOCK), blockers, and actionable warnings. Use before merging any code change.
Parameters (3)
- diffstringrequired
Unified git diff (output of `git diff HEAD`)
- commit_messagestringrequired
The commit message to lint (e.g. "feat(auth): add OAuth2 login")
- contextstring
Optional: PR title or description for richer bug analysis
list_llm_models
List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.
Parameters (1)
- providerstring
Filter by provider name (case-insensitive). E.g. "Groq", "HuggingFace", "OpenAI", "Anthropic", "Google", "DeepSeek", "xAI", "Ollama". Omit for full catalog.
llm_generate
Generate text using open-source LLM models hosted on Groq (ultra-fast) or HuggingFace Inference (serverless). No API key required — the server provides its own keys. Supported models: Qwen3 32B, Gemma 4 27B, Gemma 3 27B, Llama 3.3 70B, Llama 4 Scout, DeepSeek R1, Mistral Small 24B, and more. Use list_llm_models to see the full catalog. Rate-limited to prevent abuse.
Parameters (5)
- promptstringrequired
The user prompt / instruction to send to the model
- modelstring
Model ID (default: "qwen/qwen3-32b"). Use list_llm_models tool with provider "Groq" or "HuggingFace" to see available models.
- systemstring
Optional system prompt to set context or persona
- temperaturenumber
Sampling temperature 0.0–1.5 (default: 0.7)
- max_tokensnumber
Maximum tokens to generate (default: 2048, max: 4096)
rerank_evaluate
Evaluate RAG retrieval quality using the NVIDIA neural reranker (nv-rerankqa-mistral-4b-v3). Ranks passages by semantic relevance to a query and computes Precision@k and Recall@k. Optionally accepts ground-truth relevance labels to produce a PASS/FAIL CI/CD verdict.
Parameters (4)
- querystringrequired
The search query or question to rank against
- passagesarrayrequired
Array of passage objects to rank (min 2, max 20)
- top_kinteger
k for Precision@k evaluation (default 3)
- thresholdnumber
Minimum Precision@k to PASS (0-1, default 0.5)
shield_analyze
Run a comprehensive AI guardrail analysis on an LLM response. Orchestrates 6 deterministic safety checks plus an optional LLM-powered deep analysis in parallel: hallucination detection (grounding score), prompt injection scan, toxicity scan, output validation (PII/safety), guardrail rules, response quality scoring, and AI verdict (via Qwen, Gemma, Llama, etc.). Returns a unified PASS/FIX/BLOCK verdict with a 0-100 safety score, per-check results, and actionable fix recommendations. Use this as a single-call safety gate before surfacing any LLM output to users.
Parameters (5)
- responsestringrequired
The LLM-generated response to analyze
- sourcestring
Optional reference/source text for hallucination grounding check
- promptstring
Optional original prompt (used for quality scoring and injection detection)
- rulesarray
Optional guardrail rules array (same format as guardrail_test tool)
- modelstring
LLM model for AI-powered deep analysis (default: "qwen/qwen3-32b"). Set to "none" to skip LLM check. Supports any model from list_llm_models.
security_headers_check
Analyse the HTTP security headers of a public URL OR of raw response headers you paste in. Grades each header (A–F) for: Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, X-XSS-Protection, Cross-Origin-Opener-Policy, Cross-Origin-Resource-Policy, and Cross-Origin-Embedder-Policy. Returns an overall score (0–100), per-header grades, missing headers, and fix snippets for Express, Nginx, and Apache. For localhost/private targets the remote server cannot reach, pass the `headers` parameter instead of `url`.
Parameters (2)
- urlstring
Optional. Full public URL to check (e.g. https://example.com). Omit it entirely when using `headers`. The server cannot reach localhost/private IPs.
- headersany
Optional, and sufficient on its own (no url needed). The response headers to grade, either as an object {"strict-transport-security": "max-age=...", ...} or as the raw header block pasted as a string (e.g. `curl -sI` output). Use this to audit a local server the remote MCP cannot reach.
ssl_certificate_check
Analyse the SSL/TLS certificate of any HTTPS host. Returns certificate subject, issuer, validity dates, days until expiry, protocol version, cipher suite, key exchange info, and an overall grade (A+, A, B, C, F). Detects expired, self-signed, and weak certificates. Use this to audit TLS posture before production deployment or during security reviews.
Parameters (2)
- hoststringrequired
Hostname to check (e.g. example.com). Do not include https:// prefix.
- portnumber
Port number (default: 443)
cors_test
Test a URL for CORS misconfigurations. Sends preflight (OPTIONS) and cross-origin requests with various Origin headers to detect: wildcard origins with credentials, origin reflection (echoing any origin), null origin acceptance, subdomain wildcard bypass, and missing Vary headers. Returns risk level (safe/low/medium/high/critical), per-test results, and fix recommendations. Essential for API security audits.
Parameters (2)
- urlstringrequired
Full URL to test (e.g. https://api.example.com/endpoint)
- originstring
Custom Origin header to test (default: tests multiple origins automatically)
cors_checker
Check the CORS configuration of a URL the same way a browser would. Returns the main response status, all Access-Control-* headers, the tested origin, and the preflight OPTIONS response. Use this for direct CORS debugging, not just security auditing.
Parameters (3)
- urlstringrequired
Full URL to test, e.g. https://api.example.com/resource
- originstring
Origin header to simulate (default: https://yourdomain.com)
- methodstring
HTTP method to simulate (default: GET)
webhook_endpoint_create
Create a temporary webhook endpoint that captures incoming HTTP requests for one hour. Returns the webhook id, public URL, expiration timestamp, and current request count. Use together with webhook_endpoint_requests to inspect captured payloads.
Parameters (1)
- base_urlstring
Optional public base URL. Default: https://ia-qa.com/mcp/webhook
webhook_endpoint_requests
Fetch the requests captured by a webhook created with webhook_endpoint_create. Returns the newest requests first with method, headers, query params, body payload, and timestamps.
Parameters (2)
- idstringrequired
Webhook id returned by webhook_endpoint_create
- limitnumber
Maximum number of requests to return (1-100, default: 20)
cookie_security_audit
Audit the security attributes of cookies set by any URL. Fetches the URL and inspects all Set-Cookie headers for: HttpOnly, Secure, SameSite, Domain scope, Path scope, Max-Age/Expires, __Host-/__Secure- prefixes. Flags insecure patterns: missing HttpOnly on session cookies, missing Secure flag, SameSite=None without Secure, overly broad Domain, and excessive TTL. Returns per-cookie grades and an overall security score (0–100).
Parameters (1)
- urlstringrequired
Full URL to audit (e.g. https://example.com/login)
web_security_audit
Run a comprehensive web security audit combining headers, SSL, CORS, and cookies checks — then use an LLM to produce a prioritised remediation plan. Orchestrates security_headers_check + ssl_certificate_check + cors_test + cookie_security_audit in parallel, merges all findings, then asks an AI model to: (1) rank vulnerabilities by real-world exploitability, (2) generate a remediation roadmap, (3) produce fix code snippets for the detected stack. Returns both raw audit data and the AI analysis. Use this as a one-click security posture assessment.
Parameters (3)
- urlstringrequired
Full URL to audit (e.g. https://example.com)
- modelstring
LLM model for AI analysis (default: "qwen/qwen3-32b"). Set to "none" to skip AI analysis.
- api_keystring
Your Groq or HuggingFace API key. Required to enable AI analysis.
secret_scan
Scan text or code for leaked secrets: API keys (AWS, GCP, Azure, OpenAI, Anthropic, Stripe, GitHub, GitLab, Slack, Twilio, SendGrid, HuggingFace), private keys (RSA/EC/PGP), JWTs, database connection strings, Bearer tokens, and Basic auth headers. Returns a list of findings with type, severity, line number, and a redacted preview. Use before committing code, sharing logs, or sending text to an LLM. 100% regex-based, zero network calls.
Parameters (2)
- inputstringrequired
Text or code to scan for secrets
- typesstring
Comma-separated types to scan (default: all). Options: aws, gcp, azure, openai, anthropic, stripe, github, gitlab, slack, twilio, sendgrid, huggingface, jwt, private_key, connection_string, bearer, basic_auth
similarity_score
Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.
Parameters (5)
- referencestring
Reference / expected text (ground truth)
- hypothesisstring
Hypothesis / actual text (LLM output)
- metricsarray
Metrics to compute (default: all). Options: "cosine_bow", "cosine_tfidf", "jaccard", "rouge1", "rouge2", "rougeL", "bleu"
- batcharray
Batch mode: array of {reference, hypothesis} pairs.
- thresholdnumber
Optional pass/fail threshold (0-1). Applies to ROUGE-L F1 score.
needle_haystack_generate
Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.
Parameters (4)
- needlestringrequired
The fact to hide (e.g. "The secret code is ALPHA-42")
- questionstringrequired
The question to ask the LLM (e.g. "What is the secret code?")
- tokensinteger
Target haystack size in tokens (default: 5000, max: 100000)
- positionstring
Where to insert the needle: "start", "middle", "end", "random" (default: "middle")
optimize_prompt_tokens
Compress an LLM prompt by removing filler words, verbose phrases, duplicate sentences, and unnecessary whitespace. Returns optimized text with token savings breakdown. 100% deterministic, no API key needed.
Parameters (2)
- textstringrequired
The prompt text to optimize
- optionsobject
Toggle optimization steps (all true by default)
bias_detect
Analyse a set of LLM responses generated from the same prompt template but with different demographic variants (gender, origin, age, tone). Returns a bias score (0-100), sentiment analysis per variant, pairwise Jaccard similarity, and a human-readable verdict. No API key needed — runs entirely locally.
Parameters (1)
- responsesarrayrequired
Array of variant responses to compare for bias
llm_fit_finder
Find the best LLM for a given use case. Compares 30+ cloud API models and 12+ local models by cost, speed, benchmarks, features and VRAM requirements. Returns ranked recommendations with cost simulation. No API key needed.
Parameters (8)
- use_casestring
Primary use case: chatbot | code | rag | summarization | classification | reasoning | agents | multilingual
- featuresarray
Required features: vision, function_calling, json_mode, streaming, reasoning
- max_budgetnumber
Maximum monthly budget in USD (based on tokens_per_day)
- tokens_per_daynumber
Estimated daily token volume (default: 100000)
- modestring
cloud (API models) or local (Ollama/self-hosted). Default: cloud
- vram_gbnumber
GPU VRAM in GB (only for mode=local). Default: 16
- quantizationstring
Quantization (only for mode=local): Q4_K_M | Q8_0 | FP16. Default: Q4_K_M
- top_nnumber
Number of recommendations to return (default: 5)
find_tool
Search available MCP tools by keyword or category before calling them. Returns matching tool names, descriptions, and optionally their inputSchemas. Call this when you are unsure which tool to use or want to explore the catalogue. Categories: data, encoding, text, llm, qa, rag, dev, security, web.
Parameters (3)
- querystringrequired
Keyword(s) to search in tool name and description (e.g. "cors", "token", "vector", "json")
- categorystring
Optional: filter by category — data | encoding | text | llm | qa | rag | dev | security | web
- with_schemaboolean
Set true to include inputSchema in results (default: false)
list_local_tests
Discover .ia-eval.yaml LLM test suite files in the project directory. Scans CWD and standard sub-directories (evals/, tests/, contracts/). Returns file paths ready to pass to run_eval_contract.
Parameters (1)
- dirstring
Directory to scan (defaults to server CWD)
run_eval_contract
Parse a .ia-eval.yaml LLM test suite, call the specified LLM model for each scenario, run all configured scorers, and return a structured JSON report with per-scenario Pass/Fail verdicts and a Markdown summary. Use list_local_tests to discover available test files.
Parameters (4)
- contract_pathstring
Absolute or relative path to a .ia-eval.yaml file (required unless inline_contract is provided)
- inline_contractobject
Raw contract object (alternative to contract_path)
- api_keysobject
API keys to use for LLM generation (all optional — falls back to server env vars)
- overridesobject
Override contract defaults
generate_html_report
Convert a run_eval_contract() LLM Test Runner JSON result into a fully self-contained dark-themed HTML report with Pass/Fail badges, side-by-side Input/Output/Ground-Truth panels, evaluator score bars, and a radar chart. Returns the HTML as a string.
Parameters (1)
- resultsobjectrequired
The JSON object returned by run_eval_contract()
run_vlm_test_suite
Run a test suite against a Vision-Language Model (VLM) — send an image (URL or base64) + N test cases (each with a question + assertion) to GPT-4o, Claude 3.5, or Gemini. Returns per-case PASS/FAIL verdicts, a pass rate, an overall PASS/WARNING/FAIL verdict (customizable threshold), and latency stats. Assertion types: contains, not_contains, json_format, min_length, max_length, semantic_contains (TF-IDF cosine similarity ≥ 0.4). BYOK: requires your own API key for the target provider.
Parameters (8)
- image_urlstring
Public URL of the image to evaluate (required unless image_base64 is provided).
- image_base64string
Base64-encoded image data (required unless image_url is provided).
- image_mime_typestring
MIME type of the image if using image_base64 (default: image/jpeg).
- system_promptstring
Optional system prompt sent to the VLM.
- thresholdnumber
Pass rate threshold for overall verdict (default: 80, 0–100).
- modelstringrequired
VLM model to use.
- api_keystringrequired
API key for the model provider (OpenAI sk-, Anthropic sk-ant-, or Google AIzaSy...).
- test_casesarrayrequired
Array of test cases to run.
multimodal_eval_guide
Unified tool for multimodal AI evaluation: set action=guide for reference thresholds/interpretation (CLIP, FID, VQA), or set action=clip_score / fid_score / vqa_accuracy / pipeline to compute real metrics via HuggingFace Inference API and VLM BYOK calls. One tool for both reference and computation.
Parameters (16)
- actionstring
guide (default) = reference thresholds/interpretation. clip_score/fid_score/vqa_accuracy = compute that metric. pipeline = run all three.
- metricstring
[guide only] Metric to explain.
- scorenumber
[guide only] Optional score value to interpret.
- image_urlstring
[clip_score/vqa_accuracy] Public URL of the image.
- textstring
[clip_score only] Text description to compare against the image.
- image_base64string
[clip_score/vqa_accuracy] Base64-encoded image data.
- image_mime_typestring
[clip_score/vqa_accuracy] MIME type for base64 image.
- system_promptstring
[vqa_accuracy] Optional system prompt.
- modelstring
[vqa_accuracy] VLM model ID (default: gpt-4o).
- api_keystring
[vqa_accuracy] Your API key for the provider (BYOK).
- test_casesarray
[vqa_accuracy] Array of {question, accepted_answers} objects.
- real_imagesarray
[fid_score] Array of real image URLs.
- generated_imagesarray
[fid_score] Array of generated image URLs.
- clipobject
[pipeline] {image_url, text} for CLIP.
- vqaobject
[pipeline] VQA config object (same inputs as vqa_accuracy).
- fidobject
[pipeline] {real_images, generated_images} for FID.
run_vlm_test_suite_batch
Compare multiple VLMs on the same test suite in parallel — send an image (URL or base64) + N test cases to all models simultaneously. Returns per-model PASS/FAIL verdicts, pass rates, latency stats, and a comparison table. Assertion types: contains, not_contains, json_format, min_length, max_length, semantic_contains. BYOK: requires API keys for each provider.
Parameters (8)
- image_urlstring
Public URL of the image to evaluate (required unless image_base64 is provided).
- image_base64string
Base64-encoded image data (required unless image_url is provided).
- image_mime_typestring
MIME type of the image if using image_base64 (default: image/jpeg).
- system_promptstring
Optional system prompt sent to every VLM.
- thresholdnumber
Pass rate threshold for overall verdict (default: 80, 0–100).
- modelsarrayrequired
Array of model IDs to compare (runs in parallel).
- api_keysobjectrequired
Map of model ID → API key. Example: { "gpt-4o": "sk-...", "claude-3-5-sonnet-20241022": "sk-ant-..." }
- test_casesarrayrequired
Array of test cases to run against every model.
generate_eval_yaml
Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq llama-3.3-70b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.
Parameters (4)
- descriptionstringrequired
Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints.
- system_promptstring
Optional system prompt of the LLM under test. Helps generate more accurate test cases.
- scenario_countnumber
Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial.
- task_typestring
Optional task type hint to guide evaluator selection.
validate_agent_trajectory
Run declarative assertions on an agent trace (OpenAI tool-call messages, LangChain run trees, or plain text logs). No LLM call — deterministic. Assertion types: order (tool A before B), must_call, must_not_call, max_calls, min_calls, no_error, recovery (agent continues after error). Returns per-assertion PASS/FAIL, parsed steps, and an overall verdict. Use this to gate CI/CD on agent behavior correctness.
Parameters (3)
- traceanyrequired
Agent execution trace as JSON (OpenAI messages array, LangChain run tree) or plain text log (Thought/Action/Observation format).
- assertionsarrayrequired
List of assertions to validate against the trace.
- formatstring
Trace format. auto (default) detects automatically.
run_semantic_tests
Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.
Parameters (5)
- casesarrayrequired
Array of (actual, expected) pairs to evaluate.
- thresholdsobject
Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5).
- require_allboolean
If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS.
- modestring
tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key.
- api_keystring
OpenAI API key — required only when mode is embeddings.
get_testing_guidelines
Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset.
Parameters (1)
- topicstring
The testing topic to retrieve guidelines for. Omit to get the full list of available topics.
test_skill
Validate a SKILL.md definition (Cursor / GitHub Copilot / Windsurf) by auto-generating trigger-positive and trigger-negative scenarios, running each through the model with the skill injected as a system prompt, and scoring trigger accuracy + step adherence. Returns a PASS/FIX/BLOCK verdict with per-scenario breakdown. Uses Groq llama-3.3-70b by default (server key, no api_key needed). Pass api_key + model to use your own provider.
Parameters (4)
- skill_mdstringrequired
Full content of the SKILL.md file to test. Must include a name, a "Use when:" trigger description, and at least one step.
- scenario_countnumber
Number of test scenarios to generate: half trigger-positive, half trigger-negative. Default: 6.
- modelstring
LLM model ID to use for both scenario generation and testing (e.g. gpt-4o-mini, claude-3-5-haiku-20241022). Defaults to llama-3.3-70b-versatile (Groq, server key).
- api_keystring
API key for the chosen model provider. Not required when using the default Groq model.
identify_caller
Returns what the server knows about the current MCP client: clientInfo captured during initialize, User-Agent, and any _meta fields sent with this request. Useful for debugging caller identification.
Parameters (1)
- _metaobject
Optional self-identification. Keys: agent (string), model (string), version (string).
yaml_to_json
Parse a YAML string and return the equivalent JSON value. The reverse of json_to_yaml. Supports nested objects, arrays, anchors, aliases, multi-document streams, and all scalar types. Use when processing config files, CI/CD pipeline definitions, or OpenAPI specs authored in YAML.
Parameters (2)
- inputstringrequired
YAML string to parse
- multiboolean
If true, parse all documents in a multi-document stream and return an array (default: false)
env_parse
Parse a .env file content into a JSON object. Handles quoted values (single and double), inline comments, export prefix, and escaped sequences (\n, \t inside double quotes). Returns all key-value pairs. Use in CI/CD pipelines, agent config loaders, or when processing dotenv files programmatically.
Parameters (1)
- inputstringrequired
.env file content to parse (e.g. the output of `cat .env`)
json_schema_generate
Infer a JSON Schema (draft-07) from a sample JSON value. Detects types, required fields, array item shapes, nested objects, and common string formats (email, uri, date, date-time, uuid). Returns a ready-to-use schema compatible with json_schema_validate. Use when you have a sample API response or LLM output and want to auto-generate a validation schema for CI/CD testing.
Parameters (2)
- inputstringrequired
Sample JSON value (object, array, or scalar) to infer the schema from
- required_allboolean
Mark all detected object properties as required (default: true)
format_table
Convert a JSON array of objects into a Markdown table. Automatically detects columns, aligns headers, and fills missing keys with empty cells. Use when an agent needs to present structured data — tool results, model comparisons, test reports — as a readable table in a response or document.
Parameters (2)
- inputstringrequired
JSON array of objects to convert to a Markdown table
- columnsarray
Column names and order (default: all keys from first row)
openapi_validate
Validate the structure of an OpenAPI 3.x specification (JSON or YAML). Checks required top-level fields (openapi, info.title, info.version, paths), validates each operation (responses, operationId uniqueness), detects undeclared $ref components, and flags missing 2xx responses. Returns a PASS/FAIL verdict, a 0–100 compliance score, and a list of errors and warnings with JSON-pointer locations. Use before publishing an API spec or generating SDK code.
Parameters (1)
- inputstringrequired
OpenAPI 3.x specification as a JSON or YAML string
post_jira_comment
Post the output of jira_to_test_suite as a formatted comment on the source Jira ticket. Converts Gherkin, E2E steps, API tests, and ambiguities into Atlassian Document Format (ADF). STATEFUL — creates a comment on the issue.
Parameters (5)
- issue_keystringrequired
Jira issue key, e.g. "PROJ-123"
- jira_base_urlstringrequired
Atlassian base URL
- jira_emailstringrequired
Atlassian account email
- jira_tokenstringrequired
Atlassian API token
- test_suiteobjectrequired
The test_suite object from jira_to_test_suite result
create_confluence_page
Create a new Confluence page from the output of jira_to_test_suite. Formats Gherkin, E2E steps, API tests, and test data as a properly structured Confluence page with code blocks and tables. STATEFUL — creates a new page in the specified space.
Parameters (9)
- confluence_base_urlstringrequired
Atlassian base URL
- confluence_emailstringrequired
Atlassian account email
- confluence_tokenstringrequired
Atlassian API token
- space_keystringrequired
Confluence space key where the page will be created, e.g. "QA", "ENG"
- titlestring
Page title. Defaults to "Test Plan: {issue_key}"
- parent_page_idstring
Optional parent page ID — page will be created as a child of this page
- test_suiteobjectrequired
The test_suite object from jira_to_test_suite result
- issue_keystring
Source Jira issue key (for the page title and source link)
- issue_urlstring
Source Jira issue URL (added as a link in the page)
fetch_jira_issue
Fetch a complete Jira issue: summary, description converted to Markdown, status, assignee, priority, labels, custom fields, and optionally comments and attachment metadata. BYOK — credentials transit in-memory only, never stored on ia-qa.com.
Parameters (7)
- issue_keystringrequired
Jira issue key, e.g. "PROJ-123"
- jira_base_urlstringrequired
Atlassian base URL, e.g. "https://mycompany.atlassian.net"
- jira_emailstringrequired
Atlassian account email
- jira_tokenstringrequired
Atlassian API token (from id.atlassian.com > Security > API tokens)
- fieldsarray
Specific Jira field names to return. Omit for all standard fields.
- include_commentsboolean
Include issue comments, up to 20 (default: true)
- include_attachmentsboolean
Include attachment metadata list (default: false)
search_jira_issues
Search Jira using JQL (Jira Query Language). Returns matching issues with key fields. Ideal for finding open bugs, sprint tickets, or issues by label/assignee/component. BYOK — credentials transit in-memory only, never stored.
Parameters (6)
- jqlstringrequired
JQL query string, e.g. "project = PROJ AND status = Open AND assignee = currentUser() ORDER BY priority DESC"
- jira_base_urlstringrequired
Atlassian base URL, e.g. "https://mycompany.atlassian.net"
- jira_emailstringrequired
Atlassian account email
- jira_tokenstringrequired
Atlassian API token
- max_resultsnumber
Max issues to return (default: 10, max: 50)
- fieldsarray
Fields per issue. Default: summary, status, assignee, priority, issuetype, labels, created, updated
jira_to_test_suite
Transform a Jira ticket into a complete test suite: Gherkin scenarios, E2E steps, API test cases, test data matrix, and ambiguity detection. Accepts either Jira credentials (auto-fetch) or a pre-fetched issue object. The returned test_suite includes _gherkin_warnings (deterministic syntax validation — empty if clean). Requires BYOK LLM key (OpenAI, Anthropic, etc.).
Parameters (9)
- issue_keystring
Jira issue key to fetch automatically, e.g. "PROJ-123". Required if issue is not provided.
- jira_base_urlstring
Atlassian base URL. Required for auto-fetch mode.
- jira_emailstring
Atlassian account email. Required for auto-fetch mode.
- jira_tokenstring
Atlassian API token. Required for auto-fetch mode.
- issueobject
Pre-fetched issue object from fetch_jira_issue, OR a mock object with fields: key, summary, description (plain text or Markdown), status, issue_type, priority, labels, comments. Use this for offline/CI testing without Jira credentials.
- confluence_pagesarray
Optional array of pre-fetched Confluence page objects from fetch_confluence_page, used as documentation context.
- api_keystringrequired
Your LLM provider API key (OpenAI sk-, Anthropic sk-ant-, Google AIzaSy-, etc.).
- modelstringrequired
LLM model to use, e.g. "gpt-4o-mini", "claude-3-5-haiku-20241022", "gemini-2.0-flash".
- max_tokensinteger
Maximum tokens for the LLM response. Default: 8192. Increase for large tickets with many ACs; decrease to reduce cost on simple tickets.
fix_gherkin
Fix Gherkin syntax warnings from a jira_to_test_suite result. Takes the current gherkin text and the _gherkin_warnings array, calls your LLM to fix ONLY the flagged issues (adds missing Given/When/Then steps, etc.), and returns the corrected Gherkin. Lightweight — uses ~300-500 tokens vs ~5k for a full regeneration. Requires BYOK LLM key.
Parameters (4)
- gherkinstringrequired
The current Gherkin text from the jira_to_test_suite result (test_suite.gherkin).
- warningsarrayrequired
The _gherkin_warnings array from the jira_to_test_suite result.
- api_keystringrequired
Your LLM provider API key.
- modelstringrequired
LLM model to use for the fix, e.g. "gpt-4o-mini".
fetch_confluence_page
Fetch a Confluence page and return its content as clean Markdown. Accepts a numeric page_id or a full page URL. Optionally lists direct child pages. BYOK — credentials transit in-memory only, never stored.
Parameters (6)
- page_idstring
Confluence page ID (numeric string), e.g. "123456789"
- page_urlstring
Full Confluence page URL (alternative to page_id), e.g. "https://mycompany.atlassian.net/wiki/spaces/ENG/pages/123456789"
- confluence_base_urlstringrequired
Atlassian base URL, e.g. "https://mycompany.atlassian.net"
- confluence_emailstringrequired
Atlassian account email (same credentials as Jira)
- confluence_tokenstringrequired
Atlassian API token
- include_childrenboolean
List direct child pages (id + title) (default: false)
rate_tool
Give honest usage feedback on an IA-QA MCP tool. Provide a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Ratings are aggregated on a public dashboard at /devtools/mcp-ratings. Skip rating routine successes — we want signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })
Parameters (3)
- tool_namestringrequired
Name of the MCP tool to rate (e.g. "format_json", "shield_analyze")
- scorenumberrequired
Rating from 1 (poor) to 5 (excellent)
- commentstring
Strongly encouraged — explain what you were trying to do and whether the tool got you there. Be specific about what was missing, wrong, or a poor fit. This is the most valuable part of the rating (max 500 chars).
README not available yet.
Install
claude_desktop_config.json
{
"mcpServers": {
"ia-qa-toolbox": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://www.ia-qa.com/mcp"
]
}
}
}Desktop config is stdio-only; this bridges via mcp-remote. Native remote: Settings > Connectors.