Data and MLOpen source
orya-eval
A lightweight Python evaluation layer designed to compare runs, surface regressions, and keep ML or AI systems measurable through YAML-first configuration and clearer threshold control.
The brief
ML and AI systems keep changing through new data, prompts, models, and code, and any of those changes can quietly lower quality. ML engineers need a dependable way to compare runs and catch regressions without relying on ad-hoc notebooks or memory of past results.
orya-eval had to provide a lightweight Python evaluation layer with repeatable local runs, comparisons against baseline runs, and failure thresholds that a CI pipeline can act on, all defined in configuration that stays readable in Git.
How we approached it
- 1
YAML-first configuration
Evaluations are defined in YAML, so they stay readable in Git and can be reviewed like any other change to the codebase.
- 2
Baselines as the reference
Each run is compared against a baseline run, which turns evaluation into regression checking rather than a set of isolated scores.
- 3
Thresholds CI can enforce
Metric failures and quality drops are expressed as explicit thresholds, so a CI pipeline can stop a release when quality slips.
What we delivered
- YAML-first evaluation workflow that remains readable in Git
- Regression-safe comparisons against baseline runs
- CI-ready thresholds for metric failures and quality drops
- Practical evaluation tooling for ML and AI systems
What it had to achieve
- Make run-to-run comparisons clearer and more dependable
- Catch quality regressions before release
- Support a more disciplined evaluation workflow for ML teams
