rLLM ships with a built-in catalog of 60+ benchmark datasets spanning math, code, question answering, instruction following, search, vision-language, translation, and agentic tasks. All datasets are auto-pulled from HuggingFace on first use.
To add your own dataset (catalog entry, ad-hoc directory, or Harbor package), see Bring your own dataset.
Math
Code
Multiple choice (MCQ)
Question answering
hle and hle_search are gated datasets on HuggingFace. Run huggingface-cli login before pulling them.
Instruction following
Search
Datasets in this category use the search agent, which requires a search backend. Set one with --search-backend serper or --search-backend brave.
Agentic
Most agentic datasets are sandboxed — they ship per-task Dockerfiles and shell verifiers. See Sandboxes for how to pick --sandbox-backend and Tasks (Harbor-compatible) for the on-disk format.
Harbor packages (no registry entry)
Terminal-Bench 2.0 and other Harbor task packages are not in the catalog — they’re resolved from Harbor at run time:
See the Terminal-Bench cookbook for the full walkthrough.
Translation
Vision-language (VLM)
These datasets contain images and require a vision-capable model.
Using custom datasets
Three paths for adding your own dataset:
- Ad-hoc directory — point
rllm eval at a folder on disk; the loader autodetects the shape.
- Catalog entry + builder — register a builder in
rllm/registry/datasets.json so collaborators can rllm dataset pull <name>.
- Harbor package — drop in any Harbor task package with the
harbor: prefix.
See Bring your own dataset for the full walkthrough. 
