cli.utils

cli.utils

Utility methods for axolotl CLI.

Functions

Name	Description
add_options_from_config	Create Click options from the fields of a Pydantic model.
add_options_from_dataclass	Create Click options from the fields of a dataclass.
build_command	Build command list from base command and options.
download_file	Download a single file and return its processing status.
fetch_from_github	Sync files from a specific directory in the GitHub repository.
filter_none_kwargs	Wraps function to remove `None`-valued `kwargs`.
load_model_and_tokenizer	Helper function for loading a model, tokenizer, and processor specified in the given `axolotl`
strip_optional_type	Extracts the non-`None` type from an `Optional` / `Union` type.

add_options_from_config

cli.utils.add_options_from_config(config_class)

Create Click options from the fields of a Pydantic model.

Parameters

Name	Type	Description	Default
config_class	Type[BaseModel]	PyDantic model with fields to parse from the CLI	required

Returns

Name	Type	Description
	Callable	Function decorator for Axolotl CLI command.

add_options_from_dataclass

cli.utils.add_options_from_dataclass(config_class)

Create Click options from the fields of a dataclass.

Parameters

Name	Type	Description	Default
config_class	Type[Any]	Dataclass with fields to parse from the CLI.	required

Returns

Name	Type	Description
	Callable	Function decorator for Axolotl CLI command.

build_command

cli.utils.build_command(base_cmd, options)

Build command list from base command and options.

Parameters

Name	Type	Description	Default
base_cmd	list[str]	Command without options.	required
options	dict[str, Any]	Options to parse and append to base command.	required

Returns

Name	Type	Description
	list[str]	List of strings giving shell command.

download_file

cli.utils.download_file(file_info, raw_base_url, dest_path, dir_prefix)

Download a single file and return its processing status.

Parameters

Name	Type	Description	Default
file_info	tuple	Tuple of (file_path, remote_sha).	required
raw_base_url	str	Base URL for raw GitHub content.	required
dest_path	Path	Local destination directory.	required
dir_prefix	str	Directory prefix to filter files.	required

Returns

Name	Type	Description
	tuple[str, str]	Tuple of (file_path, status) where status is ‘new’, ‘updated’, or ‘unchanged’.

fetch_from_github

cli.utils.fetch_from_github(dir_prefix, dest_dir=None, max_workers=5)

Sync files from a specific directory in the GitHub repository. Only downloads files that don’t exist locally or have changed.

Parameters

Name	Type	Description	Default
dir_prefix	str	Directory prefix to filter files (e.g., ‘examples/’, ‘deepspeed_configs/’).	required
dest_dir	str \| None	Local destination directory.	`None`
max_workers	int	Maximum number of concurrent downloads.	`5`

filter_none_kwargs

cli.utils.filter_none_kwargs(func)

Wraps function to remove None-valued kwargs.

Parameters

Name	Type	Description	Default
func	Callable	Function to wrap.	required

Returns

Name	Type	Description
	Callable	Wrapped function.

load_model_and_tokenizer

cli.utils.load_model_and_tokenizer(cfg, inference=False)

Helper function for loading a model, tokenizer, and processor specified in the given axolotl config.

Parameters

Name	Type	Description	Default
cfg	DictDefault	Dictionary mapping `axolotl` config keys to values.	required
inference	bool	Boolean denoting inference mode.	`False`

Returns

Name	Type	Description
	tuple[PreTrainedModel, PreTrainedTokenizer \| PreTrainedTokenizerFast \| Any, ProcessorMixin \| None]	Tuple of (PreTrainedModel, PreTrainedTokenizer, ProcessorMixin).

strip_optional_type

cli.utils.strip_optional_type(field_type)

Extracts the non-None type from an Optional / Union type.

Parameters

Name	Type	Description	Default
field_type	type \| str \| None	Type of field for Axolotl CLI command.	required

Returns

Name	Type	Description
		If the input type is `Union[T, None]` or `Optional[T]`, returns `T`. Otherwise returns the input type unchanged.