BD Axiom — LLM Evaluation for the Enterprise

Required inputs

LLM Prompt required The user's exact input. Used to check relevance, detect injection attacks and PII.

LLM Response required The model's raw output. All 17 evaluators run against this text.

Optional inputs — each unlocks additional evaluators

Context / Retrieved Docs unlocks groundedness Without this, groundedness evaluator is skipped.

Ground Truth unlocks similarity score Used for BLEU-based answer similarity scoring.

System Prompt enables leakage detection Checks if the response accidentally reveals system instructions.

Model Being Tested metadata only This does not switch the evaluation engine. It records which LLM you tested — for your reports and comparisons. The evaluation judge is always gpt-5.4 (Azure). To compare two models, run two separate evaluations and compare their scores side by side.

Latency (ms) unlocks performance score Scored against the 5000ms threshold.

Evaluation Thresholds 17 metrics · defaults

Customise pass/fail thresholds per metric ▼

Set the minimum score a metric must reach to Pass. Below this threshold → Warning/Fail. Use presets for regulated industries or adjust individually.

Quick presets — click to apply

Quality

Safety

Governance

Security

Performance

Thresholds are sent as part of the API request. Retrieve defaults anytime: GET /thresholds/defaults

Show judge reasoning QUALITY ONLY

!

Synchronous mode — limit 500 samples per run. Async batch processing for 500+ samples is in development. Results are saved to disk for 10 minutes then auto-purged.

Project settings

Project Name

Default Model

Input Method

Paste samples — one JSON object per line (JSONL format) MAX 500 ROWS

Supported fields: id, prompt, response, context, ground_truth, system_prompt, model_name, latency_ms, prompt_tokens, completion_tokens

Fleet Thresholds Applied to all 500 samples

Set pass/fail thresholds for this bulk run ▼

These thresholds apply to all samples in this bulk run. Use presets or customise individually.

Quality

Safety

Governance

Security

Performance

Show judge reasoning QUALITY ONLY

Input Field Reference — What to Enter

REQ

Prompt — The exact message you sent to the LLM (the user query). Used to check relevance (does the response answer this?), detect injection attacks (is the user trying to manipulate the model?), and scan for PII exposure in the input.

REQ

Response — The exact text the LLM returned. This is what all 17 evaluators analyse. Do not summarise or paraphrase — paste the raw model output.

OPT

Context / Retrieved Docs — For RAG pipelines: paste the document chunks retrieved and injected into the prompt. Unlocks the Groundedness evaluator, which checks whether the response stays within the source material or invents facts not present in the documents.

OPT

Ground Truth — The correct expected answer, e.g. from a test set or SME review. Unlocks the Answer Similarity evaluator, which measures how close the response is to the reference using BLEU n-gram overlap.

OPT

System Prompt — Your model's system-level instructions. Enables the System Prompt Leakage detector, which flags if the model accidentally reproduces or hints at its own configuration in the response.

OPT

Latency (ms) — Time the API call took in milliseconds. Enables the Latency performance score. Grades: Good <1000ms · Acceptable <3000ms · Slow >3000ms · Critical >5000ms.

OPT

Prompt / Completion Tokens — From the API response usage field. Enables cost estimation (USD) and token efficiency scoring. Token efficiency penalises responses approaching the configured 4096-token maximum.

Evaluation KPI Explanation

Click any metric below to see what it measures, how the score is calculated, and what to do when it's low.

Universal Score Scale — applies to all 17 metrics

Pass/Fail is ultimately decided by the per-metric threshold sliders. The table below explains the default interpretation

Score	Status	Risk	Recommended action
80–100%	Pass	LOW	No action required. Meets quality standards.
60–79%	Warning	MEDIUM	Review the response. Consider prompt engineering improvements.
Below 60%	Fail	HIGH	Block or filter the response. Alert the team. Investigate root cause.
Multiple fails	Critical	CRITICAL	Immediate action required. Do not deploy. Remediate before next run.

Quality Evaluators (5 metrics)

API Reference

POST /evaluate # single evaluation POST /evaluate/bulk # bulk — max 500 samples POST /bulk-evaluate # async bulk kickoff (202 + job_id) GET /bulk-evaluate/{job_id} # async bulk status/result POST /evaluate/upload # CSV/JSONL file upload GET /evaluate/template/csv # download CSV template GET /evaluate/{run_id} # retrieve past run (10-min TTL) GET /docs # BD Axiom Swagger UI (interactive) GET /health # service health check

What is OAS 3.1? — OpenAPI Specification 3.1. The industry standard format for describing REST APIs. FastAPI generates this automatically and renders it as the interactive Swagger UI at /docs. It means any tool (Postman, Insomnia, API gateways) can auto-import the BD Axiom API schema without manual configuration.

BD Axiom is configured with gpt-5.4 on Azure OpenAI (bdtgscopilotazureopenaiturbopoc.openai.azure.com) as the LLM-as-judge for all evaluations.

Models for LLM-as-judge

BEST ACCURACY · BD AXIOM USES THIS

gpt-5.4 / Claude Opus 4.5

Highest reasoning quality across all 5 evaluation pillars. Best at detecting nuanced bias, hallucination chains, policy violations, and complex injection patterns. Produces richest per-metric justifications. Ideal for regulated industries.

Latency 2–6s per eval

Best for Compliance, accuracy-critical workloads

BALANCED

GPT-4o / Claude Sonnet 4.5

Best cost-quality tradeoff. Scores within 5% of gpt-5.4 on all evaluation pillars. Strong structured JSON output for Pydantic AI integration. Recommended for daily production runs.

Latency 1–3s

Best for Daily production runs, most use cases

HIGH VOLUME

GPT-4o-mini / Claude Haiku 3.5

10× faster, 90% cheaper. Use for pre-screening large batches before deep evaluation with gpt-5.4. Misses ~15% of subtle governance and bias issues.

Latency 0.3–1s

Best for Pre-screening 500+ samples

Pydantic AI Where it's used

All data models — LLMInput, LLMOutput, EvaluationSample, FrameworkConfig are Pydantic v2 BaseModels with automatic validation, type coercion, and JSON serialisation.
LLM-as-judge structured output — Each evaluator creates a pydantic_ai.Agent with a typed output schema (CoherenceJudge, BiasJudge, etc.). gpt-5.4 must return valid JSON matching that schema — invalid output fails cleanly rather than silently corrupting results.
API models — FastAPI uses Pydantic for EvalRequest, BulkEvalRequest, SingleEvalResponse — auto-generates the Swagger docs at /docs.
Config — FrameworkConfig validates default.yaml + .env with type coercion and field validation.

Azure AI Evaluation Where it's used

Quality suite — CoherenceEvaluator, RelevanceEvaluator, GroundednessEvaluator, FluencyEvaluator, SimilarityEvaluator from azure-ai-evaluation SDK.
Safety suite — ViolenceEvaluator, HateUnfairnessEvaluator, SexualEvaluator, SelfHarmEvaluator, IndirectAttackEvaluator, ProtectedMaterialEvaluator.
Judge backend — Azure OpenAI deployment gpt-5.4 on bdtgscopilotazureopenaiturbopoc used for all Pydantic AI agents.
Audit trail — AuditLogger writes JSONL to ./eval_outputs/audit/. Each run auto-purges after 10 minutes. Extendable to Azure Monitor.

17 Evaluators Full breakdown

Quality (5) — Coherence, Relevance, Groundedness, Fluency, Answer Similarity. Uses gpt-5.4 as LLM-as-judge via Pydantic AI. Falls back to BLEU/heuristic if API unavailable.
Safety (3) — Harmful Content (keyword regex + LLM judge), Hate Speech (LLM judge), Hallucination (reference comparison). Severity scored 0.0–1.0.
Governance (4) — Bias (8 categories: gender/racial/ethnic/socioeconomic/age/religious/political/confirmation), Fairness, Transparency, Policy Compliance. Aligned to EU AI Act + NIST AI RMF + Microsoft Responsible AI.
Security (4) — PII Detection (8 types: SSN/CC/Email/Phone/IP/DOB/Passport/IBAN), Prompt Injection (7 attack patterns), Adversarial Robustness (unicode/encoding attacks), System Prompt Leakage. OWASP LLM Top 10.
Performance (2) — Latency vs threshold, Verbosity ratio. Pure computation — no API call required, always instant.

Stack Technologies

Frontend — Vanilla HTML5 + CSS3 + JavaScript ES6+. Zero dependencies, zero build step. Fully responsive via CSS Grid + clamp(). Open index.html directly.
Backend — Python 3.9+ · FastAPI · Uvicorn · ASGI. Auto-generates Swagger (BD Axiom) at /docs.
Validation — Pydantic v2 throughout. Requests, responses, configs, evaluator output schemas all typed.
Output storage — JSON files written to ./eval_outputs/. Auto-purged after 10 minutes via background thread. Retrieve any run by ID within the TTL window.
Bulk limit — 500 samples synchronous. Async batch mode in development for unlimited scale.