Polish SLM Benchmark
A leaderboard for small Polish language models, with raw task scores and parameter-based groups.
Models
| # | Model ↕ | Author ↕ | Grp ↕ | Params ↕ | Type ↕ | AVG acc_norm ↓ |
Polemo2 in ↕ | Polemo2 out ↕ | 8tags ↕ | Belebele ↕ | CBD ↕ | DYK ↕ | KLEJ NER ↕ | PolQA ↕ | PPC ↕ | PSC ↕ |
|---|
How scores are calculated
acc_norm × 100; raw ACC is shown below.acc_norm scores.SpeakLeash lm-evaluation-harness
OpenPL results are produced directly with the SpeakLeash lm-evaluation-harness fork
(polish4), using the Hugging Face backend (--model hf) and 0-shot evaluation.
About the benchmark
Polish SLM Benchmark compares Polish-capable language models below 1B parameters under one reproducible evaluation protocol. Parameter groups organize scale; they are not quality labels.
Principles
- Raw scores
- Task results remain visible directly; the benchmark does not express quality as “+X versus model Y”.
- Comparable groups
- Models are grouped by parameter count so small systems can be compared without disappearing under much larger LLMs.
- Evaluation
- OpenPL results are produced with the SpeakLeash lm-evaluation-harness fork (polish4), using the Hugging Face backend and 0-shot evaluation.
- Base and instruct models
- Likelihood-based tasks can be shared; generation-only tasks remain separate where model type makes direct comparison inappropriate.
Model groups
Evaluation protocol
OpenPL results are produced directly with the SpeakLeash
lm-evaluation-harness fork (polish4), using the Hugging Face backend.
OpenPL command
The published runs use the upstream SpeakLeash tasks directly:
python -m lm_eval \ --model hf \ --model_args pretrained=<MODEL> \ --tasks polemo2_in_multiple_choice,polemo2_out_multiple_choice,polish_8tags_multiple_choice,polish_belebele_mc,polish_cbd_multiple_choice,polish_dyk_multiple_choice,polish_klej_ner_multiple_choice,polish_polqa_reranking_multiple_choice,polish_ppc_multiple_choice,polish_psc_multiple_choice \ --num_fewshot 0 \ --batch_size <BATCH> \ --device cuda:0 \ --output_path results/<MODEL>/ \ --log_samples
What a published run records
Score cells
The large number is the primary acc_norm × 100 score.
The smaller text shows raw acc, and where the upstream task exposes it, F1.
Average is computed only from the ten primary acc_norm values.