Measure the workload that matters.
Evaluate inference, deployment footprint, and operating cost against the same workload and environment. Use source-level benchmarks to build a reproducible baseline.
Compare like for like
Performance
Record hardware, operating system, runtime revision, model, input size, concurrency, and warm-up. Report latency distributions, throughput, and task quality together.
Footprint
Measure executable size, model files, peak memory, and storage separately. A runtime binary and a model download describe different costs.
Operating cost
Include hardware, power, hosting, support, and operations alongside licence or usage fees. CPU and GPU requirements depend on the model and workload.
Inspect the implementation
Use a pinned source revision and retain the raw output with your environment details. Repeat the same inputs for each configuration before comparing results.