---
title: "RLHF Datasets: What They Contain and How They Work"
url: https://www.brightinfo.com/blog/rlhf-datasets/
site: https://www.brightinfo.com/
description: "What RLHF datasets contain, how they differ from fine-tuning data, which public sets people use, and how to check data before training or licensing it."
last_updated: 2026-11-09
---

# RLHF Datasets: What They Contain and How They Work

Published 2026-11-09 · Bright Info

An RLHF dataset is a collection of human (or AI) preference judgments about model outputs. At minimum it holds a prompt, two responses and a label saying which response is better. It is used to train a reward model, and that reward model then guides the optimization of a language model. It is not the same thing as an instruction-tuning set, which only shows the model good answers.

This guide explains the structure of these datasets, names the public ones people use most, shows what training examples look like, and lists what to check before you train on any dataset. It also covers what changes when the data comes from inside a company, such as email, CRM records and support tickets.

## TL;DR

- RLHF datasets store preference relationships (a chosen and a rejected response to the same prompt). Supervised fine-tuning sets store demonstrations (a prompt and one target answer).
- The public sets most often cited are Anthropic HH-RLHF, UltraFeedback, LMSYS Arena Preferences and PKU-SafeRLHF.
- Before you use any set, check its license, its intended use, how it was collected, and whether it contains personal or offensive content.
- Company data such as Slack, email and CRM history follows different rules. It needs de-identification and a legal review before anyone outside the company sees it.
- If you own data that AI buyers may want, a sell-side process with several competing bidders helps you avoid underselling it.

## Table of Contents

- [What RLHF datasets are](#what-rlhf-datasets-are)
- [How RLHF data differs from other fine-tuning data](#how-rlhf-data-differs-from-other-fine-tuning-data)
- [Public RLHF datasets people use](#public-rlhf-datasets-people-use)
- [What a training data example looks like](#what-a-training-data-example-looks-like)
- [How preference data is used in training](#how-preference-data-is-used-in-training)
- [What to check before you train on a dataset](#what-to-check-before-you-train-on-a-dataset)
- [Our perspective](#our-perspective)
- [When company data enters the picture](#when-company-data-enters-the-picture)
- [Get a free valuation of your company data](#get-a-free-valuation-of-your-company-data)
- [FAQ](#faq)
- [Sources](#sources)

## What RLHF datasets are

Reinforcement learning from human feedback (RLHF) starts from a simple problem. People cannot write down a precise formula for what makes an answer helpful, safe or well written. They can, however, look at two answers and say which one they prefer. [Preference data is the proxy signal](https://github.com/natolambert/rlhf-book/blob/main/book/chapters/11-preference-data.md) that fills this gap, since human preferences cannot be modeled as a clean loss function and optimized directly.

An RLHF dataset is therefore a record of those comparisons. According to [Argilla's guide to collecting RLHF data](https://docs.v1.argilla.io/en/v1.17.0/conceptual_guides/llm/rlhf.html), this data trains a reward model. The reward model scores prompt and response pairs so that preferred responses get higher scores. An optimization step, often PPO-style, then updates the language model to produce answers the reward model rates highly.

A [data-centric study of RLHF](https://arxiv.org/html/2409.09603) puts the minimum structure this way: a prompt, two responses, and an annotation marking the preferred one. Beyond that minimum, datasets vary a lot in size, domain, and collection method. Some use human annotators for everything. Others use synthetic responses, synthetic rankings, or both.

![Four fields in an RLHF preference record](https://www.brightinfo.com/assets/blog/rlhf-datasets/image-1.jpg)

### Rankings and ratings

Preference labels come in two forms. A ranking gives the relative order of completions. A rating gives each completion its own score. Common practice, as described in the [RLHF book's chapter on preference data](https://github.com/natolambert/rlhf-book/blob/main/book/chapters/11-preference-data.md), is to train on rankings and to keep ratings as metadata. The choice shapes how you design the annotation interface and how many labelers you need per prompt.

## How RLHF data differs from other fine-tuning data

Most people searching for LLM fine-tuning datasets are looking for supervised fine-tuning (SFT) data. The two types serve different stages of training and look different on disk.

|  | SFT (instruction) data | RLHF preference data |
| --- | --- | --- |
| What each record holds | Prompt and one target response | Prompt, a chosen response and a rejected response |
| What it teaches | Imitate this answer | Prefer this kind of answer over that kind |
| Typical use | Instruction following, chat format, domain tone | Training a reward model, or preference methods such as DPO |
| Label source | Human demonstrations or curated answers | Human rankings, or AI-generated feedback |
| Risk if misused | Model copies weak examples | Model optimizes against a flawed reward signal |

OpenAI's [InstructGPT paper](https://cdn.openai.com/papers/Training%5Flanguage%5Fmodels%5Fto%5Ffollow%5Finstructions%5Fwith%5Fhuman%5Ffeedback.pdf) shows how the pieces fit together. Labelers produced three datasets: an SFT set of demonstrations, a reward-model set of rankings of model outputs, and a PPO set of prompts with no human labels. The paper reports roughly 13k training prompts for SFT, 33k for the reward model, and 31k for PPO. The reward-model set was the one built from rankings.

That split matters when you read a dataset card. A set built for reward modeling is not automatically good SFT data, and the reverse is also true.

![How RLHF data differs from other fine-tuning data](https://www.brightinfo.com/assets/blog/rlhf-datasets/image-2.jpg)

## Public RLHF datasets people use

A handful of open preference datasets show up in most papers and tutorials. The [Towards Data-Centric RLHF paper](https://arxiv.org/html/2409.09603) names four commonly used ones: Anthropic HH-RLHF, UltraFeedback, LMSYS Arena Preferences, and PKU-SafeRLHF.

### Anthropic HH-RLHF

This is the set that most results for this topic point to. The [Hugging Face dataset page](https://huggingface.co/datasets/Anthropic/hh-rlhf) shows 161k training rows, each with a chosen and a rejected text field. The helpfulness data comes in three tranches:

1. A base set sampled from context-distilled 52B language models.
2. A rejection-sampling set, mostly best-of-16 sampling against an early preference model.
3. An iterated online set sampled during weekly RLHF updates.

Harmlessness data was collected only for the base models. The accompanying paper, [Training a helpful and harmless assistant with RLHF](https://arxiv.org/abs/2204.05862), was published on April 12, 2022.

The dataset card is explicit about intended use. The set is released under the MIT license. It is meant for training preference or reward models and for studying red-teaming attacks. It is not meant for supervised training of dialogue agents, and the authors warn that doing so is likely to produce harmful models. It also contains content that may be offensive or upsetting, including discriminatory language and discussions of abuse, violence and self-harm.

### Others worth knowing

UltraFeedback, LMSYS Arena Preferences and PKU-SafeRLHF differ in how they were built. UltraFeedback relies heavily on synthetic feedback. The LMSYS set comes from user votes in head-to-head model comparisons. PKU-SafeRLHF focuses on safety preferences. Check each dataset card for the current size and collection details, since these change between versions.

One caution about older advice: the Hugging Face explainer [Illustrating RLHF](https://huggingface.co/blog/rlhf) said in 2022 that there was only one large-scale general RLHF dataset, plus a few smaller task-specific ones such as summarization data from OpenAI. The landscape has widened since then, which is why reading the current cards matters more than relying on a list.

## What a training data example looks like

Seeing the two formats side by side removes most of the confusion.

### A supervised fine-tuning record

Chat-style fine-tuning data is usually JSONL, one example per line. OpenAI's [sample fine-tuning file](https://github.com/openai/openai-node/blob/e228aaad/examples/fine-tuning-data.jsonl) uses a messages array of role and content pairs, such as system, user and assistant. The assistant message is the target completion. A simplified record reads:

`{"messages": [{"role": "system", "content": "You are a support agent."}, {"role": "user", "content": "How do I reset my password?"}, {"role": "assistant", "content": "Open Settings, choose Security, then select Reset password."}]}
`

The [Hugging Face Optimum Neuron tutorial](https://huggingface.co/docs/optimum-neuron/en/training_tutorials/finetune_qwen3) describes the same idea: you convert structured examples into prompt and expected-output pairs, using the model tokenizer's chat template. Preprocessing has to match that template, including how the end of a sequence is handled, or training quality suffers.

### A preference record

A preference record keeps the same prompt but pairs two answers:

`{"prompt": "How do I reset my password?", "chosen": "Open Settings, choose Security, then select Reset password.", "rejected": "Passwords are important. Please contact someone."}
`

HH-RLHF follows this shape: each line is a pair of texts, one chosen and one rejected, with the conversation history included in both.

## How preference data is used in training

The pipeline has three broad steps, as laid out in the [Argilla guide](https://docs.v1.argilla.io/en/v1.17.0/conceptual_guides/llm/rlhf.html) and the InstructGPT paper.

1. **Start from a supervised baseline.** Fine-tune a pretrained model on demonstrations so it follows instructions at all.
2. **Train a reward model.** Sample prompts, generate several responses with the current model, and have people rank them. The [Illustrating RLHF explainer](https://huggingface.co/blog/rlhf) notes that for most applications the reward-model set is around 50k labeled preference samples.
3. **Optimize against the reward model.** Use reinforcement learning to push the language model toward responses the reward model scores highly.

Newer recipes skip the explicit reward model. The [Alignment Handbook](https://github.com/huggingface/alignment-handbook?tab=readme-ov-file) from Hugging Face covers reward modeling and preference alignment with methods such as DPO and ORPO, alongside SFT for chat, with a YAML configuration per recipe. Those methods still consume the same chosen and rejected pairs, so the dataset format carries over.

Cost and quality depend on collection. The Hugging Face explainer points out that disagreement among human annotators introduces substantial variance into the reward-model training data. Clear guidelines, overlap between labelers and spot audits are the standard answers.

## What to check before you train on a dataset

A dataset that looks right in a viewer can still be wrong for your project. Run through these checks first.

- **Intended use.** HH-RLHF is built for preference models, not for imitation. Using a set outside its stated purpose is the most common avoidable mistake.
- **License.** HH-RLHF is MIT licensed, but blends are different. The [NVIDIA Nemotron RL blends card](https://huggingface.co/datasets/nvidia/Nemotron-RL-Super-Training-Blends) says users who download original data through placeholders are responsible for checking that each dataset's license fits their purpose.
- **Collection method.** Human versus synthetic prompts, responses and rankings change how a reward model behaves. The data-centric RLHF paper finds that dataset properties such as domain, synthetic versus human generation, annotation procedure and size can substantially change reward-model training outcomes.
- **Provenance and documentation.** A draft paper on [best practices for open LLM datasets](https://www.arxiv.org/pdf/2501.08365) recommends recording provenance, the tools used in sourcing, and the metadata needed to trace each item, along with privacy-preserving methods such as anonymization for PII removal.
- **Privacy.** One governance guide, [Avichala's note on data governance for LLM training](https://www.avichala.com/blog/data-governance-for-llm-training), argues that de-identification belongs inside the processing pipeline, including redaction of PII and use of differential privacy where appropriate.
- **Content.** Safety sets often contain disturbing text by design. Decide who on your team will read samples and how.

## Our perspective

> Most of the public conversation about RLHF datasets is about the handful of open sets that everyone already trains on. The more interesting question for a business is a different one: what do you hold that nobody else has?
>
> Years of support tickets, sales email, CRM history, call recordings and internal documents show how real work gets done. That is the kind of material AI builders look for when they want models and agents to handle real workflows. Whether a given buyer wants it for fine-tuning, evaluation or agent-training environments is their call, and each buyer sets its own requirements.
>
> What we see as the main risk for a data owner is not a lack of interest. It is selling to the first buyer who asks. A single offer tells you what one buyer will pay, not what your data is worth. That is why we work as a sell-side broker: we invite every qualified buyer to bid against the others in sealed bids, and we represent the data owner, not the buyers.
>
> Privacy has to be settled before value is discussed. Our approach is to use an independent, owner-approved de-identification partner, and we do not move any company data until a deal is signed. Sellers pay nothing to us, and we are paid only if a deal closes, by the winning buyer.
>
> *Erik*

## When company data enters the picture

RLHF-style preference sets are usually built on purpose, with paid labelers comparing outputs. Company records are different. Emails, CRM entries, Slack threads and support tickets are a record of real work, not a set of ranked answers. Some of that material can be turned into prompt and response examples. Support tickets with agent replies, for example, resemble the SFT record shown above.

Rights and safeguards come first. A 2026 [guide for general counsel on US AI deployment](https://www.squirepattonboggs.com/media/ey3poizc/what-gcs-should-consider-for-us-ai-deployment-in-2026.pdf) says AI use cases need diligence beyond traditional privacy and security review, because of risks such as model memorization, prompt injection and data leakage through outputs. It recommends refreshing AI-specific rights and restrictions in contract templates.

In practice, before any license conversation, a company should confirm:

- who owns the records, and whether customer contracts or employee agreements limit reuse;
- which personal or confidential details must be removed, and by whom;
- what the buyer may do with the data, for how long, and whether the license is exclusive;
- how the license terms are written, so that permitted uses are explicit.

We wrote more about how data types are assessed in our guide to [financial datasets for AI](https://www.brightinfo.com/blog/financial-dataset-ai/), and about the economics in [AI training costs and what data is worth](https://www.brightinfo.com/blog/ai-training-costs/).

### Estimating value without a universal price

There is no standard price per record. One research write-up on [valuing proprietary AI data](https://tracerml.ai/research/pricing-capability/) suggests running matched evaluations on a real task with the dataset included and excluded, holding the model, prompts, tools and scoring fixed. The difference estimates the capability uplift, which you then translate into business terms: how many decisions or workflows are affected, the value per unit, how long the benefit lasts, permitted uses and integration costs. Treat this as one method, and note that it comes from a company blog, so use it as a framework rather than a benchmark.

For a market view, competing bids are the most direct signal. Our own published range for company data deals is $100K to $4M, and where a company lands inside it depends on the data. You can see how different buyer types operate in our [profiles of data marketplaces](https://www.brightinfo.com/buyers/data-marketplaces/) and our [comparison of data monetization companies](https://www.brightinfo.com/data-monetization-companies/).

## Get a free valuation of your company data

If your company has more than three years of history in systems such as email, Slack, CRM, support tickets, codebases, documents, call recordings or financial exports, we can show you what it may be worth to AI buyers. Intake takes about two minutes. Buyers bid against each other in sealed bids, license terms are reviewed and transparent, and you pay nothing unless a deal closes.

Start with the [free valuation](https://www.brightinfo.com/), and read how we handle privacy on our [security and data handling page](https://www.brightinfo.com/security/).

## FAQ

### What is the difference between RLHF data and fine-tuning data?

Fine-tuning (SFT) data pairs a prompt with one target response, and the model learns to imitate it. RLHF data pairs a prompt with a chosen and a rejected response, and it trains a reward model or a preference-based method. Many projects use both: SFT first, then preference data to refine behavior.

### How many examples do I need for a reward model?

There is no fixed number. The Hugging Face RLHF explainer cites about 50k labeled preference samples for most applications, and InstructGPT used roughly 33k training prompts for its reward-model set. Quality, annotator agreement and domain fit often matter more than raw count.

### Can I train a chat model directly on HH-RLHF?

The dataset card advises against it. The data is intended for training preference or reward models, and the authors warn that using it for supervised training of dialogue agents is likely to produce harmful models. It also includes offensive and upsetting content.

### Can a company license its emails and CRM records as AI training data?

Some companies do, but the data needs a rights review and de-identification first. We handle this as a sell-side broker: personal details are removed through an owner-approved partner, and no data moves until a deal is signed. Your own counsel should confirm what your contracts allow.

## Sources

- [Collecting RLHF data (Argilla docs)](https://docs.v1.argilla.io/en/v1.17.0/conceptual_guides/llm/rlhf.html)
- [Anthropic/hh-rlhf dataset card (Hugging Face)](https://huggingface.co/datasets/Anthropic/hh-rlhf)
- [Training a helpful and harmless assistant with RLHF (arXiv)](https://arxiv.org/abs/2204.05862)
- [Towards Data-Centric RLHF (arXiv)](https://arxiv.org/html/2409.09603)
- [Illustrating Reinforcement Learning from Human Feedback (Hugging Face)](https://huggingface.co/blog/rlhf)
- [Training language models to follow instructions with human feedback (OpenAI)](https://cdn.openai.com/papers/Training%5Flanguage%5Fmodels%5Fto%5Ffollow%5Finstructions%5Fwith%5Fhuman%5Ffeedback.pdf)
- [RLHF book, chapter on preference data](https://github.com/natolambert/rlhf-book/blob/main/book/chapters/11-preference-data.md)
- [OpenAI sample fine-tuning data](https://github.com/openai/openai-node/blob/e228aaad/examples/fine-tuning-data.jsonl)
- [Fine-tune Qwen3 8B with LoRA (Hugging Face Optimum Neuron)](https://huggingface.co/docs/optimum-neuron/en/training_tutorials/finetune_qwen3)
- [Alignment Handbook (GitHub)](https://github.com/huggingface/alignment-handbook?tab=readme-ov-file)
- [NVIDIA Nemotron RL Super-Training Blends (Hugging Face)](https://huggingface.co/datasets/nvidia/Nemotron-RL-Super-Training-Blends)
- [Best Practices for Open Datasets for LLM Training (arXiv)](https://www.arxiv.org/pdf/2501.08365)
- [Data Governance for LLM Training (Avichala)](https://www.avichala.com/blog/data-governance-for-llm-training)
- [What GCs should consider for US AI deployment in 2026](https://www.squirepattonboggs.com/media/ey3poizc/what-gcs-should-consider-for-us-ai-deployment-in-2026.pdf)
- [Pricing Capability: Valuing Proprietary AI Data (Tracer)](https://tracerml.ai/research/pricing-capability/)

## Recommended

- [Predictive Maintenance Datasets: Where to Find Them and How to Choose](https://www.brightinfo.com/blog/predictive-maintenance-datasets/)
- [Invoice Datasets: What They Contain and What They're Worth](https://www.brightinfo.com/blog/invoice-datasets/)
- [Supply Chain Datasets: Where to Find Them and What to Check](https://www.brightinfo.com/blog/supply-chain-datasets/)
- [Financial Dataset AI: Types, Quality Checks and Value](https://www.brightinfo.com/blog/financial-dataset-ai/)

---

More posts: https://www.brightinfo.com/blog/