Benchmarks and evaluation¶
Throughput¶
spum bench model.gguf --gpu --n-prompt 256 -n 128 --reps 5
spum bench model.gguf --cpu -t 8 --n-prompt 256 -n 128 --reps 5
Prefill and decode are measured separately. The command reports median and best rates after warm-up. Use identical model files, power state, prompt/output lengths and trial counts when comparing engines. Record hardware, versions and precision options.
Precision comparison¶
spum bench model.gguf --gpu --compact off --reps 5
spum bench model.gguf --gpu --compact auto --reps 5
Check output quality separately; a faster precision policy is not necessarily equivalent to the baseline. Do not compare a cached image timing to a first-use image request.
Perplexity¶
The corpus needs enough tokens for complete context chunks. Use the same corpus, tokenizer, context size and scoring protocol across runs. Perplexity is a language-model measure, not a clinical accuracy score.
Inspect tokenization and next-token candidates¶
spum tokenize model.gguf -p "Describe the observation."
spum tokenize model.gguf --raw --parse-special -p "Hello"
spum logits model.gguf --gpu -p "The answer is"
Token IDs depend on the model and formatting. Logits output helps compare an exact prompt across compatible execution configurations; it does not measure the quality of an entire generated report.
Biomedical evaluation¶
Use patient-disjoint data, expert labels and a frozen test set. Keep model development separate from external validation. Localization needs spatial annotations, not merely a plausible textual explanation. Preserve failures and negative cases in the evaluation, and distinguish runtime performance from model accuracy.