Skip to content

opencomplai eval

Run the safety, bias, data-leakage, adversarial, and calibration pipeline evaluators against an EvalSampleSet directly, without going through opencomplai check.

Usage

Bash
opencomplai eval --manifest system-manifest.json --sample-set eval-set.json
PowerShell
opencomplai eval --manifest system-manifest.json --sample-set eval-set.json

Options

Flag Default Description
--manifest / -m system-manifest.json System manifest path
--sample-set (required) Path to an EvalSampleSet JSON
--commit-ref HEAD Commit reference for provenance
--output / -o human human or json
--provider (none) Opt-in: call a live model provider for each prompt (see below)
--model (none) Model name to request from --provider — required when --provider is set
--provider-api-key-env OPENCOMPLAI_PROVIDER_API_KEY Environment variable holding the provider API key
--suite (none) Opt-in: run an external benchmark suite bridge (compl-ai — see below; not yet functional)

See Evaluators for what each local evaluator measures and which are opt-in vs. always-on.

Multi-provider eval (--provider)

--provider adds an optional live-model evaluation pass alongside the local, deterministic evaluators — useful when you want a real model's completions scored, not just pre-recorded outputs in your sample set.

Bash
export OPENCOMPLAI_PROVIDER_API_KEY=sk-...
opencomplai eval \
  --manifest system-manifest.json \
  --sample-set eval-set.json \
  --provider openai \
  --model gpt-4o-mini
PowerShell
$env:OPENCOMPLAI_PROVIDER_API_KEY = "sk-..."
opencomplai eval --manifest system-manifest.json --sample-set eval-set.json --provider openai --model gpt-4o-mini

Opt-in and non-deterministic — read this before using it

The local evaluators (safety, bias, leakage, adversarial, calibration) always run first and are fully deterministic. --provider is a separate, explicitly opt-in pass that calls a real model over the network — its output is inherently non-deterministic and never affects opencomplai check's exit code contract. This code path is not imported by check at all; it is only reachable through opencomplai eval --provider.

Every provider-backed result is tagged "deterministic": false in its output so it can never be mistaken for the local evaluator suite's results:

JSON
{
  "provider": "openai",
  "model": "gpt-4o-mini",
  "deterministic": false,
  "note": "Live model-provider call — non-deterministic, network-dependent, opt-in only.",
  "completions": [
    { "prompt": "...", "completion": "..." }
  ]
}

Required environment variable

The API key is read from the environment variable named by --provider-api-key-env (default OPENCOMPLAI_PROVIDER_API_KEY) — never passed as a CLI argument. Use a different variable name per provider/project if you need to keep credentials separate:

Bash
opencomplai eval --sample-set eval-set.json --provider openai --model gpt-4o-mini \
  --provider-api-key-env MY_OPENAI_KEY

Supported providers

--provider value Endpoint shape
openai OpenAI-compatible /v1/chat/completions
openai_compatible Any self-hosted/vLLM/compatible deployment implementing the same /v1/chat/completions shape

COMPL-AI benchmark suite (--suite compl-ai) — integration scaffolding, not yet functional

Do not use --suite compl-ai expecting a working benchmark run today

This flag currently fails cleanly with an install hint, and even with the optional dependency installed, it raises NotImplementedError before running any real evaluation. It defines the mapping contract for a future integration — it does not vendor the actual COMPL-AI Inspect task definitions.

What exists today:

  • An optional compl-ai-bridge extra: pip install 'opencomplai-core[compl-ai-bridge]' (installs inspect-ai).
  • --suite compl-ai on eval, which checks whether inspect-ai is importable and, if so, calls into the bridge module.
  • A fully-defined mapping function (eval_log_to_evaluator_result) that would convert a COMPL-AI Inspect task result into the same EvaluatorResult shape as every other evaluator on this page — this part works and is tested.
  • run_compl_ai_suite(), which intentionally raises NotImplementedError — the actual COMPL-AI Inspect task definitions are not vendored in this package, and running them live requires sourcing that task suite from the COMPL-AI project itself.
Text Only
$ opencomplai eval --sample-set eval-set.json --suite compl-ai
Error: --suite compl-ai requires the optional 'compl-ai-bridge' extra.
  Install with: pip install 'opencomplai-core[compl-ai-bridge]'

Even after installing the extra, invoking the suite raises NotImplementedError with a message explaining that the mapping contract is ready but the real task definitions are not yet wired in. Treat this as a roadmap item, not a working feature — see logs/autonomous-exec/BLOCKERS.md in the repository for what a maintainer needs to do to complete this integration.

This bridge is never imported by opencomplai check's default path — same deterministic/air-gapped guarantee as the rest of the local evaluator suite.