This guide is Part 2 of our series on GRPO-based fine-tuning. It assumes you are already familiar with the core concepts of reasoning models and GRPO (see Part 1 Understanding Reasoning Models with GRPO: A Conceptual Introduction for Building your own Medical Reasoning Model ), and you are ready to apply them in practice using Python. All code is provided in a reproducible notebook, with detailed explanations and references to help you get started. Abstract: In this guide, we’ll walk step by step through fine tuning a large language model on a medical reasoning dataset from Hugging Face, using Group Relative Policy Optimization (GRPO). In the following sections, you will see exactly how the technical setup of GRPO works, how the data is structured, how to create custom reward functions, and how to test, evaluate, and save the medical reasoning model. By the end, you will have a working GRPO pipeline for building your own medical reasoning model that produces <reasoning>..</reasoning> as well as <answer>..</answer> outputs tailored for medical questions - complete with code snippets, narrative explanations, and links to key resources. Below is the guide outline. Each section builds on the previous one and by the end, you will not only understand the theory behind GRPO but also have a fully reproducible medical‐reasoning reinforcement fine-tuning pipeline. Fundamentals of GRPO Discover what Group Relative Policy Optimization truly entails, understand why it’s a powerful Reinforcement Learning (RL) method for LLMs, and explore a high-level pseudocode sketch to solidify the details. Adapting GRPO to Your Dataset Learn how to prepare any dataset—specifically, a medical-reasoning SFT corpus— for RL. This includes steps for loading data via Hugging Face to enforcing <reasoning>–<answer> formatting. Defining Reward Functions Dive into the three core reward signals:semantic correctness, fluency measured through perplexity, and tag presence. Understand  how they are combined, normalized, and clamped to guide your model. Configuring Training Get a detailed walkthrough of the GRPOConfig and GRPOTrainer settings, including learning rates, batch sizes, generation counts, and hardware requirements, to ensure your training process is both efficient and stable. Executing the Training Loop This is where you initiate  trainer.train(), monitor the increasing rewards, and analyze the step-by-step logs to ensure your model is learning effectively. Testing the Fine-Tuned Model Save and reload your LoRA weights, then compare “before” and “after” outputs on sample medical questions to validate the structured <reasoning> → <answer> behavior. Evaluating with an LLM Judge Scale your assessment by using an LLM (e.g. GPT-4o-mini) as a judge: Test  100 sample cases, score outputs on medical accuracy, reasoning clarity, format adherence, fluency, and overall usefulness, and then aggregate the results. Saving Your Model Learn how to merge 4-bit + LoRA to 16-bit, upload the model to the Hugging Face Hub (including GGUF formats), and integrate it within applications like Cloudera AI Workbench. Impact & Takeaways Reflect on the VRAM savings, improvements in structured reasoning, and real-world scenarios. Additionally, explore tips for experimenting with group sizes, reward adjustments, and KL penalties. Conclusion Summarize what you’ve accomplished, understand why GRPO works so well for structured generation tasks, and explore  next steps to adapt and extend this workflow to new domains. 1. Fundamentals of GRPO Group Relative Policy Optimization (GRPO) is a reinforcement-learning method designed to steer language models toward desired behaviors by leveraging groupwise feedback instead of relying on individual examples. During each training iteration, the model: Generates a set of candidate outputs for each prompt. Scores each candidate using one or more custom reward functions. Calculates advantages relative to the group (i.e. how much better or worse each output is compared to the batch mean). Performs a clipped policy update that reinforces above-average outputs while applying a Kullback-Leibler (KL) divergence penalty to maintain stability. By comparing candidates within each group, GRPO amplifies useful behaviors (e.g. clear reasoning, correct answers, proper formatting) without requiring manually crafted labels for every training example. GRPO in a Nutshell What: A reinforcement learning approach that fine-tunes models by rewarding entire generations using custom reward signals. How: It rewards desired output features, similar to how a student learns from feedback. Why: The model refines its responses based on these tailored rewards. Reward Functions Definition: Functions that score the model’s outputs. Purpose: Evaluate outputs for correctness, format, and additional criteria (e.g. numeric accuracy). Examples: Check if the answer is correct. Verify that the response adheres to an XML-like format. 1.1 GRPO Algorithm in Pseudocode To implement GRPO, generate multiple responses, score them using reward functions, compare them within a batch, and update the LLM based on the best responses.  Step 1: Generate Multiple Responses: The LLM outputs several different answers for the same prompt. Step 2: Assign Rewards: Each response is evaluated and scored with a reward based on reasoning depth, formatting, and clinical accuracy. Step 3: Compare Within the Group: Responses are compared to the group's average, and those that perform above average are reinforced. Step 4: Optimize the Model: The LLM is fine-tuned to prioritize better responses using policy optimization. Step 5: Ensure Stability: Kullback-Leibler (KL) Divergence Regularization is applied to prevent the model from undergoing drastic changes while still improving its performance. This safeguards against over-optimization, which may lead to reward hacking, where the model exploits the reward function instead of genuinely improving. A visual representation of the RL pipeline used in DeepSeek-R1-Zero is illustrated below. This was the first open-source model to demonstrate that advanced reasoning capabilities can be developed purely through reinforcement learning. Without pre-labeled datasets, the model learns through trial and error, refining its behavior, parameters, and weights based solely on feedback from the solutions it generates. Fig 1: RL pipeline used in Deepseek-R1-Zero, Reference: A Visual Guide to Reasoning LLMs Now that we have outlined the key components of GRPO, let’s look at the algorithm in pseudocode. While simplified,  this version highlights the core concepts.. Input:  - initial_policy: Starting model to be trained - reward_function: Function that evaluates the quality of outputs - training_prompts: Set of training examples - group_size: Number of outputs generated per prompt (typically 4-16) Algorithm for GRPO: For each training iteration: Set reference_policy = initial_policy (snapshot the current policy) For each prompt in batch: Generate group_size different outputs using initial_policy Compute rewards for each output using reward_function       iii. Normalize rewards within group:            normalized_advantage = (reward - mean(rewards)) / std(rewards) Update the policy by maximizing the clipped ratio:           min(prob_ratio * normalized_advantage,                clip(prob_ratio, 1-epsilon, 1+epsilon) * normalized_advantage)           - kl_weight * KL(initial_policy || reference_policy)           where prob_ratio is current_prob / reference_prob Output: Optimized policy model Reference: https://huggingface.co/learn/llm-course/chapter12/3 GRPO’s key innovations are: Learning directly from any function or model. GPRO eliminates the reliance for  a separate reward, unlike methods such as PPO. Group-based learning, which is more stable and efficient than traditional methods like pairwise comparisons This shows how GRPO combines group-based advantage estimation with policy optimization while maintaining stability through clipping and KL divergence constraints. By comparing each candidate with its peers and carefully regularizing updates, GRPO provides a stable yet adaptive reinforcement learning fine-tuning algorithm, well-suited for structured generation tasks, like medical reasoning. 2. Adapting GRPO to Your Dataset This guide walks you through the process of adapting GRPO to your specific dataset, using a combination of tutorials, examples, and practical steps.  2.1 Quickstart Tutorial If you're eager to start implementing GRPO, our example notebook is an excellent resource to get you up and running quickly. The notebook demonstrates GRPO training using the Llama 3.1 (8B Instruct) base model and includes step-by-step instructions for:  Installing Unsloth and its dependencies Setup Unsloth for accelerated fine tuning while minimizing costs. Configuring GRPO settings Use our pre-selected optimal parameters, or customize the settings for your specific dataset, model, and hardware constraints. Selecting your dataset We are using an open medical reasoning dataset in this example, which is ideal for medical reasoning tasks. However, you can substitute your own dataset provided it contains at least two columns—one for questions and one for answers (with the answer omitting their reasoning process). Reward Functions The example notebook uses rewards for semantic correctness, perplexity, and tag presence. Other common options include correctness, format, and additional criteria (e.g. numeric responses). 2.2 Prepare Your Own Dataset Data Collection: If you have a custom dataset for your application, such as  a medical dataset, you can adapt GRPO to fine-tune a model to specialize in that domain. Below, we outline key considerations for preparing a dataset tailored to medical reasoning tasks. For our medical reasoning model, we want a dataset that has the following characteristics: Covers a breadth of medical reasoning scenarios Contains columns for question and answer pairs, with the answer column not revealing the reasoning behind its derivation A system prompt for reasoning to be included so that each row of the dataset is a dictionary with a system prompt enforcing structured reasoning, and question-answer pairs Given that medical applications have their own terminology, we select an open source medical dataset to gather expert-level content about the problem domain. For our medical use case demonstration, a dataset focusing on complex medical reasoning tasks is used for GRPO training from Hugging Face: 'FreedomIntelligence/medical-o1-reasoning-SFT' Format the Data: Structure your dataset into clear question-and-answer pairs. For example: Question: "Is Aspirin good for cardio vascular function?" Answer: "Aspirin can be beneficial for cardiovascular function, especially for secondary prevention (after a heart attack or stroke), but its use for primary prevention (to prevent a first heart attack or stroke) is now more carefully considered due to potential risks like bleeding." Update the Data Loader: Write a loader that reads your custom data file. For example, if your data is a Hugging Face dataset, you can load the raw dataset from the hub as shown in Option 1. Alternatively if the data is in CSV format, you can load as shown in Option 2 # Option 1: Load directly from Hugging Face from datasets import load_dataset data = load_dataset( 'FreedomIntelligence/medical-o1-reasoning-SFT', 'en' )[split] # Option 2: Load from your own CSV file from datasets import load_dataset dataset = load_dataset( 'csv', data_files={'train': 'your_data_set.csv'} )   Create a System Prompt: Use the system prompt to produce the desired output format: SYSTEM_PROMPT = """ Respond in the following format: <reasoning> ... </reasoning> <answer> ... </answer> """   3. Defining Reward Functions  At the heart of GRPO lies the reward signal: it tells the model what good behavior looks like and how to improve. In a medical reasoning context, “good behavior” has three dimensions—accuracy of the answer, readability of the explanation, and adherence to our prescribed <reasoning>/<answer> format. By carefully designing each reward function to target one of these dimensions, we can guide the model toward responses that are not only clinically correct but also clear and properly structured. In this section, we’ll introduce three complementary reward signals: Semantic Correctness Measures how closely the model’s answer matches the ground-truth reference, ensuring medical accuracy. Perplexity (Fluency) Assesses the naturalness and readability of the generated text using a language model. It penalizes outputs that are less useful or harder to understand.. Tag Presence Verifies that each completion wraps its chain of thought in <reasoning> tags and its final conclusion in <answer> tags, guaranteeing consistent formatting. Together, these rewards form a balanced scorecard—accuracy, fluency, and format—so your fine-tuned model learns not just to say the right thing, but to say it in exactly the way you’ve specified. Now, we explain the reward functions in more detail: 3.1. Device Configuration   Sets whether to run models on GPU (CUDA) or CPU based on availability. main_device = "cuda" if torch.cuda.is_available() else "cpu" reward_device = "cuda" if torch.cuda.is_available() else "cpu" 3.2. Semantic Correctness Reward Uses the cross-encoder/stsb-roberta-base model to score semantic similarity between each generated response and a ground truth answer.  CrossEncoder takes pairs (response, answer) and returns similarity scores between 0 and 1. If response is empty, the reward is -1 (penalizing no response)   def semantic_correctness(responses: List[str], answers: List[str]) -> List[float]: ...   3.3. Perplexity Calculation Reward Uses the cross-encoder/stsb-roberta-base model to score semantic similarity between each generated response and a ground truth answer.  CrossEncoder takes pairs (response, answer) and returns similarity scores between 0 and 1. If the response is empty, the reward is -1 (penalizing no response)   class PerplexityCalculator: ... Measures how fluent or "natural" a sentence is using a pretrained language model, such as microsoft/biogpt.     Tokenizes and feeds batches of texts into the model.     Calculates the loss of the model (how "surprised" it is by the input).     Converts the loss to perplexity using the formula: exp(loss)     High perplexity indicates lower reward (unreadable/unnatural), so later we invert this. 3.4. Tag Presence Reward This encourages generated outputs to include:     <reasoning> ... </reasoning>     <answer> ... </answer>   def tag_presence_reward(completions: List[dict]) -> List[float]: …       Uses regular expressions to check for the presence of each tag.     Gives a reward of 0.5 for each present tag. 3.5. Combined Reward Function This is the main reward function used during the RL training loop.  def combined_reward_func(prompts, completions, answer, **kwargs): … It does the following: Step 1: Parse generated <answer> content Extracts just the text inside <answer>...</answer> tags. Removes empty responses or those that copy the prompt. Step 2: Calculate rewards Semantic similarity with reference answers. Perplexity using BioGPT. Tag presence reward. Step 3: Normalize and combine scores A weighted sum is calculated:  combined = [ 0.5 * sim + 0.4 * perplex + 0.1 * tag for sim, perplex, tag in zip(...) ] Weighted sum of: 0.5 × similarity 0.4 × perplexity reward 0.1 × tag reward Step 4: Clamp reward Output values are clamped to the range [-1.0, 1.0]. Perplexity Normalization Logic: perplex_rewards = 1 / (perplex_scores / (perplex_scores.mean() + 1e-9))   Lower perplexity → higher reward. Then it's normalized to a [0, 1] scale:  (x - min) / (max - min) Outputs: A List[float] of rewards, one per completion, which are aligned with the input batch. Any invalid or empty completions automatically receive a score of -1.0. 4. Configuring Training  In this section, we define the GRPOConfig settings for fine-tuning the model. Here we break down the core GRPOConfig options - learning rates, optimizer choices, batch sizes, and number of generations per prompt to show you how to balance iteration speed against memory constraints. Here’s the code to configure hyperparameters for GRPO training: from trl import GRPOConfig, GRPOTrainer training_args = GRPOConfig( use_vllm=True, # use vLLM for fast inference! learning_rate=5e-6, adam_beta1=0.9, adam_beta2=0.99, weight_decay=0.1, warmup_ratio=0.1, lr_scheduler_type="cosine", optim="adamw_8bit", logging_steps=1, bf16=is_bfloat16_supported(), fp16=not is_bfloat16_supported(), per_device_train_batch_size=per_device_train_batch_size, gradient_accumulation_steps=gradient_accumulation_steps, num_generations=5, # Decrease if out of memory max_prompt_length=128, # 128 to balance longer input prompts with training time requirements max_completion_length=128, max_steps=total_steps, save_steps=int(total_steps // num_checkpoints), max_grad_norm=0.1, report_to="none", # Can use Weights & Biases output_dir="grpo_outputs", save_strategy="steps",) 4.1 Key Points about Training Configuration: The GRPOConfig sets various hyperparameters for training: use_vllm: Enables fast inference with vLLM Hyperparameters: Define learning rate, batch size, epochs, and so on. learning_rate: Controls how quickly the model learns.  A low initial LR (5e-6) with a cosine decay and 10% warmup helps the model adapt gradually without overshooting Generation Settings: Specify maximum lengths for prompts and completions. num_generations: Number of completions to generate for each prompt.  GRPO relies on comparing a group of outputs for each prompt. More generations → better signal, but higher latency and memory use. optim="adamw_8bit": 8-bit Adam lets you train larger models on limited GPU memory with minimal quality loss max_steps: Total number of training steps to perform Precision (bf16 vs fp16): Uses bfloat16 for enhanced performance on supported hardware. If your GPU supports bfloat16, you’ll get extra stability; otherwise, fall back to fp16. Max Prompt vs. completion lengths: Balancing input content and generation length keeps VRAM usage predictable. If you have longer medical cases, increase both values and reduce num_generations accordingly. 5.  Executing the Training Loop With data, rewards, and training configuration in place, the next step is to launch trainer.train(). In this section, we execute the training loop to verify that the model is learning. The call to trainer.train() will run for training_args.max_steps iterations (200 in this example), generating and scoring groups of outputs, then updating the model each step. Here’s the code to execute GRPO training: from unsloth import is_bfloat16_supported import torch max_seq_length = 1024 # Can increase for longer reasoning traces lora_rank = 64 # Larger rank = smarter, but slower model, tokenizer = FastLanguageModel.from_pretrained( model_name = "meta-llama/meta-Llama-3.1-8B-Instruct", max_seq_length = max_seq_length, load_in_4bit = True, # False for LoRA 16bit fast_inference = True, # Enable vLLM fast inference max_lora_rank = lora_rank, gpu_memory_utilization = 0.5, # Reduce if out of memory ) model = FastLanguageModel.get_peft_model( model, r = lora_rank, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 target_modules = [ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], # Remove QKVO if out of memory lora_alpha = lora_rank, use_gradient_checkpointing = "unsloth", # Enable long context finetuning random_state = 3407, ) # Set up the GRPO trainer with reward functions and the dataset. trainer = GRPOTrainer( model = model, processing_class = tokenizer, reward_funcs = [ combined_reward_func ], args = training_args, train_dataset = train_dataset,) # Begin training. trainer.train()   5.1 Key Points about Training Execution: Model Loading: Prepares the language model by loading a pre-trained language model with the appropriate precision. All training and evaluation logic works across supported architectures with minimal changes To switch models, update this line in the notebook: model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct" # Swap with another base model # model_name = "Qwen/Qwen2.5-7B" # model_name = "microsoft/Phi-4" # model_name = "google/gemma-3-1b-it" Tokenizer: Converts text to tokens and also converts tokens back into readable text for evaluation. Trainer Setup: Utilizes the model, reward functions, dataset, and training configuration to fine-tune the model via reinforcement learning Optional PEFT: Can be enabled for parameter-efficient fine-tuning. 5.2 Results of Training Performance Initial reward: -0.30, this is a sign of random / untrained behavior Mid-training: Rewards fluctuate as the model explores (eg, from -0.2 to +0.2) End of training: By the end of the 200-step training regime, rewards steadily climb from negative to strongly positive values and plateau around +0.55-0.65, indicating improved semantic accuracy, fluency, and format adherence 6. Testing the Fine-Tuned Model   After completing the training process,, we’ll save LoRA weights and merge them back into the base model. Testing begins with running a set of representative  medical questions-ideally ones not seen during training through both the base and fine-tuned model.  This allows us to qualitatively access  improvements in areas, such as structured reasoning tags, more concise answers, and domain-specific correctness. Let’s test our model to see how it performs. For this, let’s first save the LoRA weights: model.save_lora("grpo_saved_lora") Now, let’s test the model with a new question: from vllm import SamplingParams text = tokenizer.apply_chat_template( [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": "Is Aspirin good for cardiovascular function?"}, ], tokenize=False, add_generation_prompt=True, ) sampling_params = SamplingParams( temperature=0.8, top_p=0.95, max_tokens=1024, ) output = ( model.fast_generate( Text, sampling_params=sampling_params, lora_request=model.load_lora("grpo_saved_lora"), )[0] .outputs[0] .text ) print(output) You should observe that the model now adheres to the specified format, presenting its reasoning before providing an answer. 6.1 Before vs After GRPO Before GRPO Fine-Tuning: Free form response, no <reasoning>/<answer> tags Responses are detailed, but unstructured output After GRPO Fine-Tuning We notice structured output, with a <reasoning> section, then <answer> section Strict adherence to  the system prompt format Concise, relevant clinical output Input: “Is Aspirin good for cardiovascular function?” Output (Before GRPO): A response from the base LLM, answering the question “Is Aspirin good for cardio vascular function?”   Output (After GRPO): A response with reasoning leading to the final answer, with a more structured output presented within the answer tags. Additional training can lead to more fine-grained reasoning traces within the reasoning tags.   7. Evaluating the Model Evaluation is crucial for assessing the fine-tuned model's performance across multiple dimensions. The core idea is to use an LLM “judge” (e.g. GPT-4o-mini or GPT-4) to score model outputs on multiple axes. 7.1. Evaluation Dataset   Size & Sampling Randomly select 100 examples from your held-out test_dataset. Ensure diversity in question types (e.g., differential diagnosis, drug interactions, physiology.).          2. Data Structure Each example should contain: { "question": str, # e.g. "What are the risk factors for DVT?" "reference_answer": str, # the ground-truth <reasoning>…</reasoning><answer>…</answer> "system_prompt": SYSTEM_PROMPT }   7.2. Evaluation Dimensions   Use the judge to score each output (base vs. fine-tuned) on a 1–5 scale across the following metrics:   Medical Accuracy   Does the answer reflect correct, evidence-based medical knowledge?        2. Reasoning Quality   Is the chain of thought logical, step-wise, and medically coherent?        3. Format Adherence   Are the <reasoning> and <answer> tags correctly used?        4. Fluency & Clarity   Is the language clear, concise, and free of major grammatical issues?        5. Overall Usefulness   Would this output help a clinician or student understand the reasoning and answer? Now we compute an Overall Score as the average of the above. 7.3. LLM-as-Judge Prompt Template     You are a medical expert and evaluator. For each case below, you will see: - QUESTION: {question} - REFERENCE: {reference_answer} - MODEL OUTPUT: {model_output} Please provide scores (1–5 where 1=poor, 5=excellent): MEDICAL_SCORE: REASONING_SCORE: FORMAT_SCORE: FLUENCY_SCORE: USEFULNESS_SCORE: OVERALL_SCORE: [average of above] EXPLANATION: [2–3 sentences explaining your scoring]     7.4. Interpreting Results   We sample 100 test cases for statistical power. For each, generate both base and fine-tuned outputs. Use a deterministic judge call (temperature=0.0) to reduce variance. Parse the judge’s structured response into numeric scores. Aggregate mean & std for each metric and model.  Δ in Mean Scores: An improvement in scores (e.g., +0.8 → +1.2) on MEDICAL_SCORE or REASONING_SCORE indicates better domain reasoning. 8. Saving Your Model Unsloth provides several options for saving your fine-tuned model, but we’ll focus on the most common use case. Saving in 16-bit Precision You can save your fine-tuned  model with 16-bit precision using the following command: # Save to 16-bit precision model.save_pretrained_merged("model", tokenizer, save_method="merged_16bit") Pushing to Hugging Face Hub To share your model with the community or collaborators, you can push it to the Hugging Face Hub using the push_to_hub_merged method. This method allows us to push the model in multiple quantization formats. # Push to Hugging Face Hub (requires a token) model.push_to_hub_merged(     "your-username/model-name", tokenizer, save_method="merged_16bit", token="your-token" ) Saving in GGUF Format for llama.cpp Unsloth also supports saving your model in GGUF format for use with llama.cpp: #Saving in GGUF Format for llama.cpp model.push_to_hub_gguf(     "your-username/model-name",     tokenizer,     quantization_method=["q4_k_m", "q8_0", "q5_k_m"],     token="your-token", ) The GGUF files can be used with llama.cpp or UI-based systems like Jan or Open WebUI. 9. Impact & Takeaways   In this guide, you’ve learned how to: Prepare data for GRPO training Define custom reward functions to guide the model’s learning Train a model using GRPO Test the fine-tuned model Evaluate the fine-tuned model Save the model in various formats Key Benefits: Cost savings upto 90% VRAM savings compared to PPO methods. LLMs trained with GRPO provide structured, explainable reasoning, improving AI trustworthiness Real-World Applications: Beyond medical AI applications such as Clinical decision support systems, Virtual care triage assistant, and  Educational tools for medical residents, GRPO can enhance use cases in other domains including legal analysis, content, and compliance workflows. What’s Next? Run this notebook within your environment.  One-click deploy and experiment with the GRPO AMP within your Cloudera AI environment, extending GRPO by customizing base models, datasets and reward functions, and see how it improves structured reasoning. If you don’t have a Cloudera Notebook, then register for 5-day Cloudera trial and experience this AMP within Cloudera AI: https://www.cloudera.com/products/cloudera-public-cloud-trial.html 10. Conclusion Reward Functions: Serve as the model's scorecard by evaluating correctness and format, guiding the model to produce improved responses. Custom Training Dataset: A curated dataset (e.g. for medical reasoning) transforms a general-purpose language model into a domain-specific expert for your specific use cases. Improved Reasoning Outcomes: In this fine tuning guide to building your own medical reasoning model with GRPO, we notice how the fine tuned model not only learns how to maximize a balanced reward (across accuracy, fluency, structure), but also adopts the prescribed <reasoning> and <answer> format. Quantitatively, the reward score shifts from around -0.3 (pre-training) to +0.6 (post training), and qualitatively, the output becomes neatly structured and domain focused. Overall Benefit: This approach converts a general language model into a domain-specific expert medical reasoning model that can efficiently provide accurate and structured medical answers. These steps can be adapted by organisations with complex datasets for their own domain specific use cases. As you continue exploring GRPO, consider experimenting with different group sizes, datasets, base models, reward functions, and KL penalty coefficients to see how they affect your model’s performance. Additional Resources: Accelerator for ML Projects (AMP) on tuning models with GRPO GitHub repository. References: Unsloth Documentation | Hugging Face Reasoning Models Course | A Visual Guide to Reasoning LLMs    ... View more

1. Introduction Problem: Most readily-available Medical AI models lack structured reasoning, which significantly limits their reliability and trustworthiness in critical clinical decision-making. GRPO offers a robust solution for building reasoning Medical AI models by reinforcing logical step-by-step explanations. Takeaway: This post explains how enterprises can fine-tune Large Language Models (LLMs)  with reinforcement learning (RL) using GRPO for more structured, explainable AI, demonstrating its application with a medical reasoning use case. 2. The Business Case  Why should enterprises care? Reasoning is critical in high-stakes industries like healthcare, finance, and legal sectors.  It provides better transparency, explainability, and tracing on how models generate their outputs, building crucial trust and facilitating accountability.  Why GRPO? GRPO efficiently delivers advanced reasoning capabilities. It significantly reduces  compute requirements, cutting them by nearly half compared to traditional Reinforcement Learning from Human Feedback (RLHF) methods like PPO. Concurrently, GRPO develops models that self-verify answers, provide step-by-step reasoning, explore multiple problem-solving approaches, and even reflect on their own reasoning process. This democratizes access to reasoning models, enabling their deployment on modest hardware (for example, systems with even 16GB VRAM) and making sophisticated AI reasoning accessible  for organizations of all sizes, without requiring enterprise-scale infrastructure. Who benefits? Data science teams, AI engineers, business leaders, and decision makers seeking to optimize LLMs for complex reasoning tasks. 3. GRPO Deep Dive What is GRPO?  How does GRPO improve reasoning?  3.1. What is GRPO? GRPO is a reinforcement learning (RL) algorithm specifically designed to improve the reasoning capabilities  of LLMs. Unlike traditional supervised fine-tuning (SFT), GRPO does not just teach the model to predict the next word, it optimizes the model for specific outcomes, such as  correctness, formatting, and other task-specific rewards. At its core, GRPO: Compares multiple model outputs (candidates) per prompt in a batch. Assigns rewards based on correctness, formatting, and other predefined metrics. Adjusts the model to increase the likelihood of generating better reasoning paths and answers in future iterations. Traditional reinforcement learning methods rely heavily on separate value functions, but GRPO simplifies this, creating a “group-based” learning experience. This is achieved through a clever iterative process: Group Sampling: The model simultaneously generates multiple diverse answers for each question. Reward Scoring: Each generated answer is evaluated for accuracy, format, and consistency. Group Advantage: Answers outperforming the batch average are rewarded; lower performers are discouraged. Policy Update: The model's policy is updated, increasingly favoring the generation of more logical and structurally sound answers.. Iterative Refinement: The entire process repeats, continuously refining the model towards optimal reasoning. GRPO is memory efficient and aligns particularly with tasks where correctness is objectively verifiable but labeled data is scarce. Examples include medical Q&A, legal reasoning, and code generation. In essence, it represents a shift from text-style traditional fine-tuning to outcome-based fine-tuning for LLMs. Tools like Unsloth are making this method more  accessible, which can lead to superior performance on verifiable tasks while improving explainability with reasoning chains. 3.2. How Does GRPO Improve Reasoning? Fig 1: “Regular” vs “Reasoning” LLMs  Medical Reasoning Example Let’s walk through a practical medical reasoning example to illustrate the difference: Traditional Fine-Tuning Model Behavior: Given a patient case, a model trained with traditional Supervised Fine-Tuning (SFT) methods might hallucinate by generating  the most likely next word based on patterns in its training dataset. It can generate a diagnosis but often lacks structured reasoning or explanation for its conclusion. GRPO-Enhanced Model Behavior: In contrast, a GRPO-trained model is rewarded not only for the correct final diagnosis but for providing a detailed, reasoned trace:  <reasoning> The patient presents with chest pain, high blood pressure, and shortness of breath. These symptoms are indicative of cardiovascular issues. Given the elevated blood pressure and the patient's age, the most probable diagnosis is acute coronary syndrome. </reasoning> <answer> Acute Coronary Syndrome </answer> The model is rewarded for: Clinical accuracy (matching ground truth diagnosis) Providing traceable reasoning in a structured format Using medically valid explanations 4. Real-World Outcomes By fine-tuning with tailored datasets, base models, and reward functions, you achieve: Custom Expertise: The model becomes a specialised domain-specific assistant, capable of answering questions with a high degree of accuracy and relevance within its designated field. Consistent Responses: Enforced formatting ensures that each response includes clear reasoning and a final answer in a predictable format. Efficiency: Leveraging Unsloth’s fine-tuning optimizations, the GRPO-trained model can efficiently deliver medical suggestions with a variety of open-source language models. 5. Conclusion GRPO offers a powerful approach to developing highly specialized and reliable LLMs: Domain specific reasoning model: A curated dataset (e.g. for medical reasoning) is instrumental in transforming a general-purpose language model into a domain-specific expert.  Reward Functions: Serve as the model's scorecard by evaluating correctness and format, guiding the model to produce improved responses. Overall Benefit: GRPO is a powerful technique for aligning language models with specific behaviors. It helps convert a general LLM into an expert system that can efficiently provide accurate and detailed answers for organisations working with complex datasets.  6. Resources   GRPO is a powerful technique for aligning language models with specific behaviors. It helps convert a general language model into an expert system that can efficiently provide accurate and detailed answers for organisations with complex datasets. Run this notebook within your environment One-click deploy and experiment with the GRPO AMP within your Cloudera AI environment If you don’t have a Cloudera notebook, then register for 5-day Cloudera trial and experience this AMP within Cloudera AI: https://www.cloudera.com/products/cloudera-public-cloud-trial.html Suggested links: Part II, A Practical Guide to Fine-Tuning Language Models with GRPO ... View more

This guide is Part 2 of our series on GRPO-based fine-tuning. It assumes you are already familiar with the core concepts of reasoning models and GRPO (see Part 1 Understanding Reasoning Models with GRPO: A Conceptual Introduction for Building your own Medical Reasoning Model ), and you are ready to apply them in practice using Python. All code is provided in a reproducible notebook, with detailed explanations and references to help you get started. Abstract: In this guide, we’ll walk step by step through fine tuning a large language model on a medical reasoning dataset from Hugging Face, using Group Relative Policy Optimization (GRPO). In the following sections, you will see exactly how the technical setup of GRPO works, how the data is structured, how to create custom reward functions, and how to test, evaluate, and save the medical reasoning model. By the end, you will have a working GRPO pipeline for building your own medical reasoning model that produces <reasoning>..</reasoning> as well as <answer>..</answer> outputs tailored for medical questions - complete with code snippets, narrative explanations, and links to key resources. Below is the guide outline. Each section builds on the previous one and by the end, you will not only understand the theory behind GRPO but also have a fully reproducible medical‐reasoning reinforcement fine-tuning pipeline. Fundamentals of GRPO Discover what Group Relative Policy Optimization truly entails, understand why it’s a powerful Reinforcement Learning (RL) method for LLMs, and explore a high-level pseudocode sketch to solidify the details. Adapting GRPO to Your Dataset Learn how to prepare any dataset—specifically, a medical-reasoning SFT corpus— for RL. This includes steps for loading data via Hugging Face to enforcing <reasoning>–<answer> formatting. Defining Reward Functions Dive into the three core reward signals:semantic correctness, fluency measured through perplexity, and tag presence. Understand  how they are combined, normalized, and clamped to guide your model. Configuring Training Get a detailed walkthrough of the GRPOConfig and GRPOTrainer settings, including learning rates, batch sizes, generation counts, and hardware requirements, to ensure your training process is both efficient and stable. Executing the Training Loop This is where you initiate  trainer.train(), monitor the increasing rewards, and analyze the step-by-step logs to ensure your model is learning effectively. Testing the Fine-Tuned Model Save and reload your LoRA weights, then compare “before” and “after” outputs on sample medical questions to validate the structured <reasoning> → <answer> behavior. Evaluating with an LLM Judge Scale your assessment by using an LLM (e.g. GPT-4o-mini) as a judge: Test  100 sample cases, score outputs on medical accuracy, reasoning clarity, format adherence, fluency, and overall usefulness, and then aggregate the results. Saving Your Model Learn how to merge 4-bit + LoRA to 16-bit, upload the model to the Hugging Face Hub (including GGUF formats), and integrate it within applications like Cloudera AI Workbench. Impact & Takeaways Reflect on the VRAM savings, improvements in structured reasoning, and real-world scenarios. Additionally, explore tips for experimenting with group sizes, reward adjustments, and KL penalties. Conclusion Summarize what you’ve accomplished, understand why GRPO works so well for structured generation tasks, and explore  next steps to adapt and extend this workflow to new domains. 1. Fundamentals of GRPO Group Relative Policy Optimization (GRPO) is a reinforcement-learning method designed to steer language models toward desired behaviors by leveraging groupwise feedback instead of relying on individual examples. During each training iteration, the model: Generates a set of candidate outputs for each prompt. Scores each candidate using one or more custom reward functions. Calculates advantages relative to the group (i.e. how much better or worse each output is compared to the batch mean). Performs a clipped policy update that reinforces above-average outputs while applying a Kullback-Leibler (KL) divergence penalty to maintain stability. By comparing candidates within each group, GRPO amplifies useful behaviors (e.g. clear reasoning, correct answers, proper formatting) without requiring manually crafted labels for every training example. GRPO in a Nutshell What: A reinforcement learning approach that fine-tunes models by rewarding entire generations using custom reward signals. How: It rewards desired output features, similar to how a student learns from feedback. Why: The model refines its responses based on these tailored rewards. Reward Functions Definition: Functions that score the model’s outputs. Purpose: Evaluate outputs for correctness, format, and additional criteria (e.g. numeric accuracy). Examples: Check if the answer is correct. Verify that the response adheres to an XML-like format. 1.1 GRPO Algorithm in Pseudocode To implement GRPO, generate multiple responses, score them using reward functions, compare them within a batch, and update the LLM based on the best responses.  Step 1: Generate Multiple Responses: The LLM outputs several different answers for the same prompt. Step 2: Assign Rewards: Each response is evaluated and scored with a reward based on reasoning depth, formatting, and clinical accuracy. Step 3: Compare Within the Group: Responses are compared to the group's average, and those that perform above average are reinforced. Step 4: Optimize the Model: The LLM is fine-tuned to prioritize better responses using policy optimization. Step 5: Ensure Stability: Kullback-Leibler (KL) Divergence Regularization is applied to prevent the model from undergoing drastic changes while still improving its performance. This safeguards against over-optimization, which may lead to reward hacking, where the model exploits the reward function instead of genuinely improving. A visual representation of the RL pipeline used in DeepSeek-R1-Zero is illustrated below. This was the first open-source model to demonstrate that advanced reasoning capabilities can be developed purely through reinforcement learning. Without pre-labeled datasets, the model learns through trial and error, refining its behavior, parameters, and weights based solely on feedback from the solutions it generates. Fig 1: RL pipeline used in Deepseek-R1-Zero, Reference: A Visual Guide to Reasoning LLMs Now that we have outlined the key components of GRPO, let’s look at the algorithm in pseudocode. While simplified,  this version highlights the core concepts.. Input:  - initial_policy: Starting model to be trained - reward_function: Function that evaluates the quality of outputs - training_prompts: Set of training examples - group_size: Number of outputs generated per prompt (typically 4-16) Algorithm for GRPO: For each training iteration: Set reference_policy = initial_policy (snapshot the current policy) For each prompt in batch: Generate group_size different outputs using initial_policy Compute rewards for each output using reward_function       iii. Normalize rewards within group:            normalized_advantage = (reward - mean(rewards)) / std(rewards) Update the policy by maximizing the clipped ratio:           min(prob_ratio * normalized_advantage,                clip(prob_ratio, 1-epsilon, 1+epsilon) * normalized_advantage)           - kl_weight * KL(initial_policy || reference_policy)           where prob_ratio is current_prob / reference_prob Output: Optimized policy model Reference: https://huggingface.co/learn/llm-course/chapter12/3 GRPO’s key innovations are: Learning directly from any function or model. GPRO eliminates the reliance for  a separate reward, unlike methods such as PPO. Group-based learning, which is more stable and efficient than traditional methods like pairwise comparisons This shows how GRPO combines group-based advantage estimation with policy optimization while maintaining stability through clipping and KL divergence constraints. By comparing each candidate with its peers and carefully regularizing updates, GRPO provides a stable yet adaptive reinforcement learning fine-tuning algorithm, well-suited for structured generation tasks, like medical reasoning. 2. Adapting GRPO to Your Dataset This guide walks you through the process of adapting GRPO to your specific dataset, using a combination of tutorials, examples, and practical steps.  2.1 Quickstart Tutorial If you're eager to start implementing GRPO, our example notebook is an excellent resource to get you up and running quickly. The notebook demonstrates GRPO training using the Llama 3.1 (8B Instruct) base model and includes step-by-step instructions for:  Installing Unsloth and its dependencies Setup Unsloth for accelerated fine tuning while minimizing costs. Configuring GRPO settings Use our pre-selected optimal parameters, or customize the settings for your specific dataset, model, and hardware constraints. Selecting your dataset We are using an open medical reasoning dataset in this example, which is ideal for medical reasoning tasks. However, you can substitute your own dataset provided it contains at least two columns—one for questions and one for answers (with the answer omitting their reasoning process). Reward Functions The example notebook uses rewards for semantic correctness, perplexity, and tag presence. Other common options include correctness, format, and additional criteria (e.g. numeric responses). 2.2 Prepare Your Own Dataset Data Collection: If you have a custom dataset for your application, such as  a medical dataset, you can adapt GRPO to fine-tune a model to specialize in that domain. Below, we outline key considerations for preparing a dataset tailored to medical reasoning tasks. For our medical reasoning model, we want a dataset that has the following characteristics: Covers a breadth of medical reasoning scenarios Contains columns for question and answer pairs, with the answer column not revealing the reasoning behind its derivation A system prompt for reasoning to be included so that each row of the dataset is a dictionary with a system prompt enforcing structured reasoning, and question-answer pairs Given that medical applications have their own terminology, we select an open source medical dataset to gather expert-level content about the problem domain. For our medical use case demonstration, a dataset focusing on complex medical reasoning tasks is used for GRPO training from Hugging Face: 'FreedomIntelligence/medical-o1-reasoning-SFT' Format the Data: Structure your dataset into clear question-and-answer pairs. For example: Question: "Is Aspirin good for cardio vascular function?" Answer: "Aspirin can be beneficial for cardiovascular function, especially for secondary prevention (after a heart attack or stroke), but its use for primary prevention (to prevent a first heart attack or stroke) is now more carefully considered due to potential risks like bleeding." Update the Data Loader: Write a loader that reads your custom data file. For example, if your data is a Hugging Face dataset, you can load the raw dataset from the hub as shown in Option 1. Alternatively if the data is in CSV format, you can load as shown in Option 2 # Option 1: Load directly from Hugging Face from datasets import load_dataset data = load_dataset( 'FreedomIntelligence/medical-o1-reasoning-SFT', 'en' )[split] # Option 2: Load from your own CSV file from datasets import load_dataset dataset = load_dataset( 'csv', data_files={'train': 'your_data_set.csv'} )   Create a System Prompt: Use the system prompt to produce the desired output format: SYSTEM_PROMPT = """ Respond in the following format: <reasoning> ... </reasoning> <answer> ... </answer> """   3. Defining Reward Functions  At the heart of GRPO lies the reward signal: it tells the model what good behavior looks like and how to improve. In a medical reasoning context, “good behavior” has three dimensions—accuracy of the answer, readability of the explanation, and adherence to our prescribed <reasoning>/<answer> format. By carefully designing each reward function to target one of these dimensions, we can guide the model toward responses that are not only clinically correct but also clear and properly structured. In this section, we’ll introduce three complementary reward signals: Semantic Correctness Measures how closely the model’s answer matches the ground-truth reference, ensuring medical accuracy. Perplexity (Fluency) Assesses the naturalness and readability of the generated text using a language model. It penalizes outputs that are less useful or harder to understand.. Tag Presence Verifies that each completion wraps its chain of thought in <reasoning> tags and its final conclusion in <answer> tags, guaranteeing consistent formatting. Together, these rewards form a balanced scorecard—accuracy, fluency, and format—so your fine-tuned model learns not just to say the right thing, but to say it in exactly the way you’ve specified. Now, we explain the reward functions in more detail: 3.1. Device Configuration   Sets whether to run models on GPU (CUDA) or CPU based on availability. main_device = "cuda" if torch.cuda.is_available() else "cpu" reward_device = "cuda" if torch.cuda.is_available() else "cpu" 3.2. Semantic Correctness Reward Uses the cross-encoder/stsb-roberta-base model to score semantic similarity between each generated response and a ground truth answer.  CrossEncoder takes pairs (response, answer) and returns similarity scores between 0 and 1. If response is empty, the reward is -1 (penalizing no response)   def semantic_correctness(responses: List[str], answers: List[str]) -> List[float]: ...   3.3. Perplexity Calculation Reward Uses the cross-encoder/stsb-roberta-base model to score semantic similarity between each generated response and a ground truth answer.  CrossEncoder takes pairs (response, answer) and returns similarity scores between 0 and 1. If the response is empty, the reward is -1 (penalizing no response)   class PerplexityCalculator: ... Measures how fluent or "natural" a sentence is using a pretrained language model, such as microsoft/biogpt.     Tokenizes and feeds batches of texts into the model.     Calculates the loss of the model (how "surprised" it is by the input).     Converts the loss to perplexity using the formula: exp(loss)     High perplexity indicates lower reward (unreadable/unnatural), so later we invert this. 3.4. Tag Presence Reward This encourages generated outputs to include:     <reasoning> ... </reasoning>     <answer> ... </answer>   def tag_presence_reward(completions: List[dict]) -> List[float]: …       Uses regular expressions to check for the presence of each tag.     Gives a reward of 0.5 for each present tag. 3.5. Combined Reward Function This is the main reward function used during the RL training loop.  def combined_reward_func(prompts, completions, answer, **kwargs): … It does the following: Step 1: Parse generated <answer> content Extracts just the text inside <answer>...</answer> tags. Removes empty responses or those that copy the prompt. Step 2: Calculate rewards Semantic similarity with reference answers. Perplexity using BioGPT. Tag presence reward. Step 3: Normalize and combine scores A weighted sum is calculated:  combined = [ 0.5 * sim + 0.4 * perplex + 0.1 * tag for sim, perplex, tag in zip(...) ] Weighted sum of: 0.5 × similarity 0.4 × perplexity reward 0.1 × tag reward Step 4: Clamp reward Output values are clamped to the range [-1.0, 1.0]. Perplexity Normalization Logic: perplex_rewards = 1 / (perplex_scores / (perplex_scores.mean() + 1e-9))   Lower perplexity → higher reward. Then it's normalized to a [0, 1] scale:  (x - min) / (max - min) Outputs: A List[float] of rewards, one per completion, which are aligned with the input batch. Any invalid or empty completions automatically receive a score of -1.0. 4. Configuring Training  In this section, we define the GRPOConfig settings for fine-tuning the model. Here we break down the core GRPOConfig options - learning rates, optimizer choices, batch sizes, and number of generations per prompt to show you how to balance iteration speed against memory constraints. Here’s the code to configure hyperparameters for GRPO training: from trl import GRPOConfig, GRPOTrainer training_args = GRPOConfig( use_vllm=True, # use vLLM for fast inference! learning_rate=5e-6, adam_beta1=0.9, adam_beta2=0.99, weight_decay=0.1, warmup_ratio=0.1, lr_scheduler_type="cosine", optim="adamw_8bit", logging_steps=1, bf16=is_bfloat16_supported(), fp16=not is_bfloat16_supported(), per_device_train_batch_size=per_device_train_batch_size, gradient_accumulation_steps=gradient_accumulation_steps, num_generations=5, # Decrease if out of memory max_prompt_length=128, # 128 to balance longer input prompts with training time requirements max_completion_length=128, max_steps=total_steps, save_steps=int(total_steps // num_checkpoints), max_grad_norm=0.1, report_to="none", # Can use Weights & Biases output_dir="grpo_outputs", save_strategy="steps",) 4.1 Key Points about Training Configuration: The GRPOConfig sets various hyperparameters for training: use_vllm: Enables fast inference with vLLM Hyperparameters: Define learning rate, batch size, epochs, and so on. learning_rate: Controls how quickly the model learns.  A low initial LR (5e-6) with a cosine decay and 10% warmup helps the model adapt gradually without overshooting Generation Settings: Specify maximum lengths for prompts and completions. num_generations: Number of completions to generate for each prompt.  GRPO relies on comparing a group of outputs for each prompt. More generations → better signal, but higher latency and memory use. optim="adamw_8bit": 8-bit Adam lets you train larger models on limited GPU memory with minimal quality loss max_steps: Total number of training steps to perform Precision (bf16 vs fp16): Uses bfloat16 for enhanced performance on supported hardware. If your GPU supports bfloat16, you’ll get extra stability; otherwise, fall back to fp16. Max Prompt vs. completion lengths: Balancing input content and generation length keeps VRAM usage predictable. If you have longer medical cases, increase both values and reduce num_generations accordingly. 5.  Executing the Training Loop With data, rewards, and training configuration in place, the next step is to launch trainer.train(). In this section, we execute the training loop to verify that the model is learning. The call to trainer.train() will run for training_args.max_steps iterations (200 in this example), generating and scoring groups of outputs, then updating the model each step. Here’s the code to execute GRPO training: from unsloth import is_bfloat16_supported import torch max_seq_length = 1024 # Can increase for longer reasoning traces lora_rank = 64 # Larger rank = smarter, but slower model, tokenizer = FastLanguageModel.from_pretrained( model_name = "meta-llama/meta-Llama-3.1-8B-Instruct", max_seq_length = max_seq_length, load_in_4bit = True, # False for LoRA 16bit fast_inference = True, # Enable vLLM fast inference max_lora_rank = lora_rank, gpu_memory_utilization = 0.5, # Reduce if out of memory ) model = FastLanguageModel.get_peft_model( model, r = lora_rank, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 target_modules = [ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], # Remove QKVO if out of memory lora_alpha = lora_rank, use_gradient_checkpointing = "unsloth", # Enable long context finetuning random_state = 3407, ) # Set up the GRPO trainer with reward functions and the dataset. trainer = GRPOTrainer( model = model, processing_class = tokenizer, reward_funcs = [ combined_reward_func ], args = training_args, train_dataset = train_dataset,) # Begin training. trainer.train()   5.1 Key Points about Training Execution: Model Loading: Prepares the language model by loading a pre-trained language model with the appropriate precision. All training and evaluation logic works across supported architectures with minimal changes To switch models, update this line in the notebook: model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct" # Swap with another base model # model_name = "Qwen/Qwen2.5-7B" # model_name = "microsoft/Phi-4" # model_name = "google/gemma-3-1b-it" Tokenizer: Converts text to tokens and also converts tokens back into readable text for evaluation. Trainer Setup: Utilizes the model, reward functions, dataset, and training configuration to fine-tune the model via reinforcement learning Optional PEFT: Can be enabled for parameter-efficient fine-tuning. 5.2 Results of Training Performance Initial reward: -0.30, this is a sign of random / untrained behavior Mid-training: Rewards fluctuate as the model explores (eg, from -0.2 to +0.2) End of training: By the end of the 200-step training regime, rewards steadily climb from negative to strongly positive values and plateau around +0.55-0.65, indicating improved semantic accuracy, fluency, and format adherence 6. Testing the Fine-Tuned Model   After completing the training process,, we’ll save LoRA weights and merge them back into the base model. Testing begins with running a set of representative  medical questions-ideally ones not seen during training through both the base and fine-tuned model.  This allows us to qualitatively access  improvements in areas, such as structured reasoning tags, more concise answers, and domain-specific correctness. Let’s test our model to see how it performs. For this, let’s first save the LoRA weights: model.save_lora("grpo_saved_lora") Now, let’s test the model with a new question: from vllm import SamplingParams text = tokenizer.apply_chat_template( [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": "Is Aspirin good for cardiovascular function?"}, ], tokenize=False, add_generation_prompt=True, ) sampling_params = SamplingParams( temperature=0.8, top_p=0.95, max_tokens=1024, ) output = ( model.fast_generate( Text, sampling_params=sampling_params, lora_request=model.load_lora("grpo_saved_lora"), )[0] .outputs[0] .text ) print(output) You should observe that the model now adheres to the specified format, presenting its reasoning before providing an answer. 6.1 Before vs After GRPO Before GRPO Fine-Tuning: Free form response, no <reasoning>/<answer> tags Responses are detailed, but unstructured output After GRPO Fine-Tuning We notice structured output, with a <reasoning> section, then <answer> section Strict adherence to  the system prompt format Concise, relevant clinical output Input: “Is Aspirin good for cardiovascular function?” Output (Before GRPO): A response from the base LLM, answering the question “Is Aspirin good for cardio vascular function?”   Output (After GRPO): A response with reasoning leading to the final answer, with a more structured output presented within the answer tags. Additional training can lead to more fine-grained reasoning traces within the reasoning tags.   7. Evaluating the Model Evaluation is crucial for assessing the fine-tuned model's performance across multiple dimensions. The core idea is to use an LLM “judge” (e.g. GPT-4o-mini or GPT-4) to score model outputs on multiple axes. 7.1. Evaluation Dataset   Size & Sampling Randomly select 100 examples from your held-out test_dataset. Ensure diversity in question types (e.g., differential diagnosis, drug interactions, physiology.).          2. Data Structure Each example should contain: { "question": str, # e.g. "What are the risk factors for DVT?" "reference_answer": str, # the ground-truth <reasoning>…</reasoning><answer>…</answer> "system_prompt": SYSTEM_PROMPT }   7.2. Evaluation Dimensions   Use the judge to score each output (base vs. fine-tuned) on a 1–5 scale across the following metrics:   Medical Accuracy   Does the answer reflect correct, evidence-based medical knowledge?        2. Reasoning Quality   Is the chain of thought logical, step-wise, and medically coherent?        3. Format Adherence   Are the <reasoning> and <answer> tags correctly used?        4. Fluency & Clarity   Is the language clear, concise, and free of major grammatical issues?        5. Overall Usefulness   Would this output help a clinician or student understand the reasoning and answer? Now we compute an Overall Score as the average of the above. 7.3. LLM-as-Judge Prompt Template     You are a medical expert and evaluator. For each case below, you will see: - QUESTION: {question} - REFERENCE: {reference_answer} - MODEL OUTPUT: {model_output} Please provide scores (1–5 where 1=poor, 5=excellent): MEDICAL_SCORE: REASONING_SCORE: FORMAT_SCORE: FLUENCY_SCORE: USEFULNESS_SCORE: OVERALL_SCORE: [average of above] EXPLANATION: [2–3 sentences explaining your scoring]     7.4. Interpreting Results   We sample 100 test cases for statistical power. For each, generate both base and fine-tuned outputs. Use a deterministic judge call (temperature=0.0) to reduce variance. Parse the judge’s structured response into numeric scores. Aggregate mean & std for each metric and model.  Δ in Mean Scores: An improvement in scores (e.g., +0.8 → +1.2) on MEDICAL_SCORE or REASONING_SCORE indicates better domain reasoning. 8. Saving Your Model Unsloth provides several options for saving your fine-tuned model, but we’ll focus on the most common use case. Saving in 16-bit Precision You can save your fine-tuned  model with 16-bit precision using the following command: # Save to 16-bit precision model.save_pretrained_merged("model", tokenizer, save_method="merged_16bit") Pushing to Hugging Face Hub To share your model with the community or collaborators, you can push it to the Hugging Face Hub using the push_to_hub_merged method. This method allows us to push the model in multiple quantization formats. # Push to Hugging Face Hub (requires a token) model.push_to_hub_merged(     "your-username/model-name", tokenizer, save_method="merged_16bit", token="your-token" ) Saving in GGUF Format for llama.cpp Unsloth also supports saving your model in GGUF format for use with llama.cpp: #Saving in GGUF Format for llama.cpp model.push_to_hub_gguf(     "your-username/model-name",     tokenizer,     quantization_method=["q4_k_m", "q8_0", "q5_k_m"],     token="your-token", ) The GGUF files can be used with llama.cpp or UI-based systems like Jan or Open WebUI. 9. Impact & Takeaways   In this guide, you’ve learned how to: Prepare data for GRPO training Define custom reward functions to guide the model’s learning Train a model using GRPO Test the fine-tuned model Evaluate the fine-tuned model Save the model in various formats Key Benefits: Cost savings upto 90% VRAM savings compared to PPO methods. LLMs trained with GRPO provide structured, explainable reasoning, improving AI trustworthiness Real-World Applications: Beyond medical AI applications such as Clinical decision support systems, Virtual care triage assistant, and  Educational tools for medical residents, GRPO can enhance use cases in other domains including legal analysis, content, and compliance workflows. What’s Next? Run this notebook within your environment.  One-click deploy and experiment with the GRPO AMP within your Cloudera AI environment, extending GRPO by customizing base models, datasets and reward functions, and see how it improves structured reasoning. If you don’t have a Cloudera Notebook, then register for 5-day Cloudera trial and experience this AMP within Cloudera AI: https://www.cloudera.com/products/cloudera-public-cloud-trial.html 10. Conclusion Reward Functions: Serve as the model's scorecard by evaluating correctness and format, guiding the model to produce improved responses. Custom Training Dataset: A curated dataset (e.g. for medical reasoning) transforms a general-purpose language model into a domain-specific expert for your specific use cases. Improved Reasoning Outcomes: In this fine tuning guide to building your own medical reasoning model with GRPO, we notice how the fine tuned model not only learns how to maximize a balanced reward (across accuracy, fluency, structure), but also adopts the prescribed <reasoning> and <answer> format. Quantitatively, the reward score shifts from around -0.3 (pre-training) to +0.6 (post training), and qualitatively, the output becomes neatly structured and domain focused. Overall Benefit: This approach converts a general language model into a domain-specific expert medical reasoning model that can efficiently provide accurate and structured medical answers. These steps can be adapted by organisations with complex datasets for their own domain specific use cases. As you continue exploring GRPO, consider experimenting with different group sizes, datasets, base models, reward functions, and KL penalty coefficients to see how they affect your model’s performance. Additional Resources: Accelerator for ML Projects (AMP) on tuning models with GRPO GitHub repository. References: Unsloth Documentation | Hugging Face Reasoning Models Course | A Visual Guide to Reasoning LLMs    ... View more

1. Introduction Problem: Most readily-available Medical AI models lack structured reasoning, which significantly limits their reliability and trustworthiness in critical clinical decision-making. GRPO offers a robust solution for building reasoning Medical AI models by reinforcing logical step-by-step explanations. Takeaway: This post explains how enterprises can fine-tune Large Language Models (LLMs)  with reinforcement learning (RL) using GRPO for more structured, explainable AI, demonstrating its application with a medical reasoning use case. 2. The Business Case  Why should enterprises care? Reasoning is critical in high-stakes industries like healthcare, finance, and legal sectors.  It provides better transparency, explainability, and tracing on how models generate their outputs, building crucial trust and facilitating accountability.  Why GRPO? GRPO efficiently delivers advanced reasoning capabilities. It significantly reduces  compute requirements, cutting them by nearly half compared to traditional Reinforcement Learning from Human Feedback (RLHF) methods like PPO. Concurrently, GRPO develops models that self-verify answers, provide step-by-step reasoning, explore multiple problem-solving approaches, and even reflect on their own reasoning process. This democratizes access to reasoning models, enabling their deployment on modest hardware (for example, systems with even 16GB VRAM) and making sophisticated AI reasoning accessible  for organizations of all sizes, without requiring enterprise-scale infrastructure. Who benefits? Data science teams, AI engineers, business leaders, and decision makers seeking to optimize LLMs for complex reasoning tasks. 3. GRPO Deep Dive What is GRPO?  How does GRPO improve reasoning?  3.1. What is GRPO? GRPO is a reinforcement learning (RL) algorithm specifically designed to improve the reasoning capabilities  of LLMs. Unlike traditional supervised fine-tuning (SFT), GRPO does not just teach the model to predict the next word, it optimizes the model for specific outcomes, such as  correctness, formatting, and other task-specific rewards. At its core, GRPO: Compares multiple model outputs (candidates) per prompt in a batch. Assigns rewards based on correctness, formatting, and other predefined metrics. Adjusts the model to increase the likelihood of generating better reasoning paths and answers in future iterations. Traditional reinforcement learning methods rely heavily on separate value functions, but GRPO simplifies this, creating a “group-based” learning experience. This is achieved through a clever iterative process: Group Sampling: The model simultaneously generates multiple diverse answers for each question. Reward Scoring: Each generated answer is evaluated for accuracy, format, and consistency. Group Advantage: Answers outperforming the batch average are rewarded; lower performers are discouraged. Policy Update: The model's policy is updated, increasingly favoring the generation of more logical and structurally sound answers.. Iterative Refinement: The entire process repeats, continuously refining the model towards optimal reasoning. GRPO is memory efficient and aligns particularly with tasks where correctness is objectively verifiable but labeled data is scarce. Examples include medical Q&A, legal reasoning, and code generation. In essence, it represents a shift from text-style traditional fine-tuning to outcome-based fine-tuning for LLMs. Tools like Unsloth are making this method more  accessible, which can lead to superior performance on verifiable tasks while improving explainability with reasoning chains. 3.2. How Does GRPO Improve Reasoning? Fig 1: “Regular” vs “Reasoning” LLMs  Medical Reasoning Example Let’s walk through a practical medical reasoning example to illustrate the difference: Traditional Fine-Tuning Model Behavior: Given a patient case, a model trained with traditional Supervised Fine-Tuning (SFT) methods might hallucinate by generating  the most likely next word based on patterns in its training dataset. It can generate a diagnosis but often lacks structured reasoning or explanation for its conclusion. GRPO-Enhanced Model Behavior: In contrast, a GRPO-trained model is rewarded not only for the correct final diagnosis but for providing a detailed, reasoned trace:  <reasoning> The patient presents with chest pain, high blood pressure, and shortness of breath. These symptoms are indicative of cardiovascular issues. Given the elevated blood pressure and the patient's age, the most probable diagnosis is acute coronary syndrome. </reasoning> <answer> Acute Coronary Syndrome </answer> The model is rewarded for: Clinical accuracy (matching ground truth diagnosis) Providing traceable reasoning in a structured format Using medically valid explanations 4. Real-World Outcomes By fine-tuning with tailored datasets, base models, and reward functions, you achieve: Custom Expertise: The model becomes a specialised domain-specific assistant, capable of answering questions with a high degree of accuracy and relevance within its designated field. Consistent Responses: Enforced formatting ensures that each response includes clear reasoning and a final answer in a predictable format. Efficiency: Leveraging Unsloth’s fine-tuning optimizations, the GRPO-trained model can efficiently deliver medical suggestions with a variety of open-source language models. 5. Conclusion GRPO offers a powerful approach to developing highly specialized and reliable LLMs: Domain specific reasoning model: A curated dataset (e.g. for medical reasoning) is instrumental in transforming a general-purpose language model into a domain-specific expert.  Reward Functions: Serve as the model's scorecard by evaluating correctness and format, guiding the model to produce improved responses. Overall Benefit: GRPO is a powerful technique for aligning language models with specific behaviors. It helps convert a general LLM into an expert system that can efficiently provide accurate and detailed answers for organisations working with complex datasets.  6. Resources   GRPO is a powerful technique for aligning language models with specific behaviors. It helps convert a general language model into an expert system that can efficiently provide accurate and detailed answers for organisations with complex datasets. Run this notebook within your environment One-click deploy and experiment with the GRPO AMP within your Cloudera AI environment If you don’t have a Cloudera notebook, then register for 5-day Cloudera trial and experience this AMP within Cloudera AI: https://www.cloudera.com/products/cloudera-public-cloud-trial.html Suggested links: Part II, A Practical Guide to Fine-Tuning Language Models with GRPO ... View more