Benchmarks: What and Why
Domain specific
- No single metric possible
- The more general the benchmark, the less useful it is for anything in particular.
- A benchmark is a distillation of the essential attributes of a workload
Desirable attributes
- Relevant è meaningful within the target domain
- Understandable
- Good metric(s) è linear, orthogonal, monotonic
- Scaleable è applicable to a broad spectrum of hardware/architecture
- Coverage è does not oversimplify the typical environment
- Acceptance è Vendors and Users embrace it