cli.inference

cli.inference

CLI to run inference on a trained model.

Functions

Name	Description
do_cli	Parses axolotl config, CLI args, and calls do_inference or do_inference_gradio.
do_inference	Runs inference on the command line in a loop. User input is accepted, a chat template
do_inference_gradio	Runs inference in a Gradio interface. User input is accepted, a chat template is
get_multi_line_input	Gets multi-line input from terminal.

do_cli

cli.inference.do_cli(config=Path('examples/'), gradio=False, **kwargs)

Parses axolotl config, CLI args, and calls do_inference or do_inference_gradio.

Parameters

Name	Type	Description	Default
config	Union[Path, str]	Path to axolotl config YAML file.	Path('examples/')
kwargs		Additional keyword arguments to override config file values.	{}

do_inference

cli.inference.do_inference(cfg, cli_args)

Runs inference on the command line in a loop. User input is accepted, a chat template is (optionally) applied, and the model specified in the axolotl config is used to generate completions according to a default generation config.

Parameters

Name	Type	Description	Default
cfg	DictDefault	Dictionary mapping axolotl config keys to values.	required
cli_args	InferenceCliArgs	Inference-specific CLI arguments.	required

do_inference_gradio

cli.inference.do_inference_gradio(cfg, cli_args)

Runs inference in a Gradio interface. User input is accepted, a chat template is (optionally) applied, and the model specified in the axolotl config is used to generate completions according to a default generation config.

Parameters

Name	Type	Description	Default
cfg	DictDefault	Dictionary mapping axolotl config keys to values.	required
cli_args	InferenceCliArgs	Inference-specific CLI arguments.	required

get_multi_line_input

cli.inference.get_multi_line_input()

Gets multi-line input from terminal.

Returns

Name	Type	Description
	str	Possibly multi-line, possibly empty stdin input as a string.