Inference Guide

This guide covers how to use your trained models for inference, including model loading, interactive testing, and common troubleshooting steps.

1 Quick Start

1.1 Basic Inference

LoRA Models
Full Fine-tuned Models

axolotl inference your_config.yml --lora-model-dir="./lora-output-dir"

axolotl inference your_config.yml --base-model="./completed-model"

2 Advanced Usage

2.1 Gradio Interface

Launch an interactive web interface:

axolotl inference your_config.yml --gradio

2.2 File-based Prompts

Process prompts from a text file:

cat /tmp/prompt.txt | axolotl inference your_config.yml \
  --base-model="./completed-model" --prompter=None

2.3 Memory Optimization

For large models or limited memory:

axolotl inference your_config.yml --load-in-8bit=True

3 Merging LoRA Weights

Merge LoRA adapters with the base model:

axolotl merge-lora your_config.yml --lora-model-dir="./completed-model"

gpu_memory_limit: 20GiB  # Adjust based on your GPU
lora_on_cpu: true        # Process on CPU if needed

CUDA_VISIBLE_DEVICES="" axolotl merge-lora ...

4 Tokenization

4.1 Common Issues

Warning

Tokenization mismatches between training and inference are a common source of problems.

To debug:

Check training tokenization:

axolotl preprocess your_config.yml --debug

Verify inference tokenization by decoding tokens before model input
Compare token IDs between training and inference

4.2 Special Tokens

Configure special tokens in your YAML:

special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"
tokens:
  - "<|im_start|>"
  - "<|im_end|>"

5 Troubleshooting

5.1 Common Problems

Use 8-bit loading
Reduce batch sizes
Try CPU offloading

Verify special tokens
Check tokenizer settings
Compare training and inference preprocessing

Verify model loading
Check prompt formatting
Ensure temperature/sampling settings

For more details, see our debugging guide.

--- title: "Inference Guide" format: html: toc: true toc-depth: 3 number-sections: true code-tools: true execute: enabled: false --- This guide covers how to use your trained models for inference, including model loading, interactive testing, and common troubleshooting steps. ## Quick Start {#sec-quickstart} ### Basic Inference {#sec-basic} ::: {.panel-tabset} ## LoRA Models ```{.bash} axolotl inference your_config.yml --lora-model-dir="./lora-output-dir" ``` ## Full Fine-tuned Models ```{.bash} axolotl inference your_config.yml --base-model="./completed-model" ``` ::: ## Advanced Usage {#sec-advanced} ### Gradio Interface {#sec-gradio} Launch an interactive web interface: ```{.bash} axolotl inference your_config.yml --gradio ``` ### File-based Prompts {#sec-file-prompts} Process prompts from a text file: ```{.bash} cat /tmp/prompt.txt | axolotl inference your_config.yml \ --base-model="./completed-model" --prompter=None ``` ### Memory Optimization {#sec-memory} For large models or limited memory: ```{.bash} axolotl inference your_config.yml --load-in-8bit=True ``` ## Merging LoRA Weights {#sec-merging} Merge LoRA adapters with the base model: ```{.bash} axolotl merge-lora your_config.yml --lora-model-dir="./completed-model" ``` ### Memory Management for Merging {#sec-memory-management} ::: {.panel-tabset} ## Configuration Options ```{.yaml} gpu_memory_limit: 20GiB # Adjust based on your GPU lora_on_cpu: true # Process on CPU if needed ``` ## Force CPU Merging ```{.bash} CUDA_VISIBLE_DEVICES="" axolotl merge-lora ... ``` ::: ## Tokenization {#sec-tokenization} ### Common Issues {#sec-tokenization-issues} ::: {.callout-warning} Tokenization mismatches between training and inference are a common source of problems. ::: To debug: 1. Check training tokenization: ```{.bash} axolotl preprocess your_config.yml --debug ``` 2. Verify inference tokenization by decoding tokens before model input 3. Compare token IDs between training and inference ### Special Tokens {#sec-special-tokens} Configure special tokens in your YAML: ```{.yaml} special_tokens: bos_token: "<s>" eos_token: "</s>" unk_token: "<unk>" tokens: - "<|im_start|>" - "<|im_end|>" ``` ## Troubleshooting {#sec-troubleshooting} ### Common Problems {#sec-common-problems} ::: {.panel-tabset} ## Memory Issues - Use 8-bit loading - Reduce batch sizes - Try CPU offloading ## Token Issues - Verify special tokens - Check tokenizer settings - Compare training and inference preprocessing ## Performance Issues - Verify model loading - Check prompt formatting - Ensure temperature/sampling settings ::: For more details, see our [debugging guide](debugging.qmd).