RL (GRPO)

RL (GRPO) applies when you can score with reward / verifier / environment feedback — for example, math reasoning (answers verifiable programmatically), code (sandbox execution), or chat quality (judge-model scoring). MinT wraps GRPO, PPO, and other RL algorithms in APIs that look just like SFT, cutting the cost of building your own.

The RL training loop

The GRPO loop has four stages:

Sampling. The current policy generates multiple responses for each prompt.
Scoring. A reward function (verifier / judge / environment) scores each response.
Advantage computation. Center the rewards within the group of samples for the same prompt: advantage[i] = reward[i] - mean_r.
Training. Train with loss_fn="importance_sampling"; samples with reward above mean_r get positive gradients, those below get negative gradients.

Building a Datum

The prompt positions get a weight of 0, and the response positions get the advantage value; you also need to pass the per-token logprobs (obtained from the sampling step) and advantages (zero-padded over the prompt prefix).

mean_r = sum(rewards) / num_samples
for i, (seq, reward) in enumerate(zip(sequences, rewards)):
    advantage = reward - mean_r
    response_len = len(seq.tokens) - prompt_len
    advantages   = [0.0] * (prompt_len - 1) + [advantage] * response_len
    datums.append(types.Datum(
        model_input=types.ModelInput.from_ints(tokens=seq.tokens[:-1]),
        loss_fn_inputs={
            "target_tokens": seq.tokens[1:],
            "weights":       [0.0] * (prompt_len - 1) + [1.0] * response_len,
            "logprobs":      seq.logprobs[1:],
            "advantages":   advantages,
        },
    ))

Parameters

Parameter	Type	Default	Meaning
`loss_fn`	str	`"importance_sampling"`	GRPO default; the server also supports `"ppo"`, `"cispo"`, `"dro"`
`group_size`	int	`4`	Samples per prompt; larger means lower variance but more memory, typically 4–16
`groups_per_batch`	int	`8`	Number of prompts per training step
`max_tokens`	int	`16`	Max generation length; math ≈ 16, chat ≈ 128, code ≈ 256
`learning_rate`	float	`2e-5`	RL is usually lower than SFT, typically 1e-5 to 4e-5
`temperature`	float	`0.8`	Sampling temperature, typically 0.7–1.0 for RL
`kl_penalty_coef`	float	`0.0`	When > 0, applies a KL penalty on policy deviation from the reference to prevent collapse
`base_model`	str	`"Qwen/Qwen3-0.6B"`	Base model ID
`rank`	int	`16`	LoRA rank

SFT + RL two-stage

For scenarios with both supervised data and a reward signal, we recommend a warm start with SFT first, then switching to GRPO. For a reference path, see the Community quickstart.py.

Next steps

Math RL: exact-match scoring with a programmatic verifier
Chat RL: judge-model preference reward
Code RL: scoring from sandbox execution results
dapo-aime case study: an end-to-end GRPO example for math reasoning