Skip to content

Components

Finetune

lazyllm.components.finetune.AlpacaloraFinetune

Bases: LazyLLMFinetuneBase

This class is a subclass of LazyLLMFinetuneBase, based on the LoRA fine-tuning capabilities provided by the alpaca-lora project, used for LoRA fine-tuning of large language models.

Parameters:

  • base_model (str) –

    The base model used for fine-tuning. It is required to be the path of the base model.

  • target_path (str) –

    The path where the LoRA weights of the fine-tuned model are saved.

  • merge_path (str, default: None ) –

    The path where the model merges the LoRA weights, default to None. If not specified, "lazyllm_lora" and "lazyllm_merge" directories will be created under target_path as target_path and merge_path respectively.

  • model_name (str, default: 'LLM' ) –

    The name of the model, used as the prefix for setting the log name, default to "LLM".

  • cp_files (str, default: 'tokeniz*' ) –

    Specify configuration files to be copied from the base model path, which will be copied to merge_path, default to tokeniz*

  • launcher (launcher, default: remote(ngpus=1) ) –

    The launcher for fine-tuning, default to launchers.remote(ngpus=1).

  • kw –

    Keyword arguments, used to update the default training parameters. Note that additional keyword arguments cannot be arbitrarily specified.

The keyword arguments and their default values for this class are as follows:

Other Parameters:

  • data_path (str) –

    Data path, default to None; generally passed as the only positional argument when this object is called.

  • batch_size (int) –

    Batch size, default to 64.

  • micro_batch_size (int) –

    Micro-batch size, default to 4.

  • num_epochs (int) –

    Number of training epochs, default to 2.

  • learning_rate (float) –

    Learning rate, default to 5.e-4.

  • cutoff_len (int) –

    Cutoff length, default to 1030; input data tokens will be truncated if they exceed this length.

  • filter_nums (int) –

    Number of filters, default to 1024; only input with token length below this value is preserved.

  • val_set_size (int) –

    Validation set size, default to 200.

  • lora_r (int) –

    LoRA rank, default to 8; this value determines the amount of parameters added, the smaller the value, the fewer the parameters.

  • lora_alpha (int) –

    LoRA fusion factor, default to 32; this value determines the impact of LoRA parameters on the base model parameters, the larger the value, the greater the impact.

  • lora_dropout (float) –

    LoRA dropout rate, default to 0.05, generally used to prevent overfitting.

  • lora_target_modules (str) –

    LoRA target modules, default to [wo,wqkv], which is the default for InternLM2 model; this configuration item varies for different models.

  • modules_to_save (str) –

    Modules for full fine-tuning, default to [tok_embeddings,output], which is the default for InternLM2 model; this configuration item varies for different models.

  • deepspeed (str) –

    The path of the DeepSpeed configuration file, default to use the pre-made configuration file in the LazyLLM code repository: ds.json.

  • prompt_template_name (str) –

    The name of the prompt template, default to "alpaca", i.e., use the prompt template provided by LazyLLM by default.

  • train_on_inputs (bool) –

    Whether to train on inputs, default to True.

  • show_prompt (bool) –

    Whether to show the prompt, default to False.

  • nccl_port (int) –

    NCCL port, default to 19081.

Examples:

>>> from lazyllm import finetune
>>> trainer = finetune.alpacalora('path/to/base/model', 'path/to/target')
Source code in lazyllm/components/finetune/alpacalora.py 
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
class AlpacaloraFinetune(LazyLLMFinetuneBase):
    """This class is a subclass of ``LazyLLMFinetuneBase``, based on the LoRA fine-tuning capabilities provided by the [alpaca-lora](https://github.com/tloen/alpaca-lora) project, used for LoRA fine-tuning of large language models.

Args:
    base_model (str): The base model used for fine-tuning. It is required to be the path of the base model.
    target_path (str): The path where the LoRA weights of the fine-tuned model are saved.
    merge_path (str): The path where the model merges the LoRA weights, default to `None`. If not specified, "lazyllm_lora" and "lazyllm_merge" directories will be created under ``target_path`` as ``target_path`` and ``merge_path`` respectively.
    model_name (str): The name of the model, used as the prefix for setting the log name, default to "LLM".
    cp_files (str): Specify configuration files to be copied from the base model path, which will be copied to ``merge_path``, default to ``tokeniz*``
    launcher (lazyllm.launcher): The launcher for fine-tuning, default to ``launchers.remote(ngpus=1)``.
    kw: Keyword arguments, used to update the default training parameters. Note that additional keyword arguments cannot be arbitrarily specified.

The keyword arguments and their default values for this class are as follows:

Keyword Args: 
    data_path (str): Data path, default to ``None``; generally passed as the only positional argument when this object is called.
    batch_size (int): Batch size, default to ``64``.
    micro_batch_size (int): Micro-batch size, default to ``4``.
    num_epochs (int): Number of training epochs, default to ``2``.
    learning_rate (float): Learning rate, default to ``5.e-4``.
    cutoff_len (int): Cutoff length, default to ``1030``; input data tokens will be truncated if they exceed this length.
    filter_nums (int): Number of filters, default to ``1024``; only input with token length below this value is preserved.
    val_set_size (int): Validation set size, default to ``200``.
    lora_r (int): LoRA rank, default to ``8``; this value determines the amount of parameters added, the smaller the value, the fewer the parameters.
    lora_alpha (int): LoRA fusion factor, default to ``32``; this value determines the impact of LoRA parameters on the base model parameters, the larger the value, the greater the impact.
    lora_dropout (float): LoRA dropout rate, default to ``0.05``, generally used to prevent overfitting.
    lora_target_modules (str): LoRA target modules, default to ``[wo,wqkv]``, which is the default for InternLM2 model; this configuration item varies for different models.
    modules_to_save (str): Modules for full fine-tuning, default to ``[tok_embeddings,output]``, which is the default for InternLM2 model; this configuration item varies for different models.
    deepspeed (str): The path of the DeepSpeed configuration file, default to use the pre-made configuration file in the LazyLLM code repository: ``ds.json``.
    prompt_template_name (str): The name of the prompt template, default to "alpaca", i.e., use the prompt template provided by LazyLLM by default.
    train_on_inputs (bool): Whether to train on inputs, default to ``True``.
    show_prompt (bool): Whether to show the prompt, default to ``False``.
    nccl_port (int): NCCL port, default to ``19081``.



Examples:
    >>> from lazyllm import finetune
    >>> trainer = finetune.alpacalora('path/to/base/model', 'path/to/target')
    """
    defatult_kw = ArgsDict({
        'data_path': None,
        'batch_size': 64,
        'micro_batch_size': 4,
        'num_epochs': 2,
        'learning_rate': 5.e-4,
        'cutoff_len': 1030,
        'filter_nums': 1024,
        'val_set_size': 200,
        'lora_r': 8,
        'lora_alpha': 32,
        'lora_dropout': 0.05,
        'lora_target_modules': '[wo,wqkv]',
        'modules_to_save': '[tok_embeddings,output]',
        'deepspeed': '',
        'prompt_template_name': 'alpaca',
        'train_on_inputs': True,
        'show_prompt': False,
        'nccl_port': 19081,
    })
    auto_map = {'micro_batch_size': 'micro_batch_size'}

    def __init__(self,
                 base_model,
                 target_path,
                 merge_path=None,
                 model_name='LLM',
                 cp_files='tokeniz*',
                 launcher=launchers.remote(ngpus=1),  # noqa B008
                 **kw
                 ):
        if not merge_path:
            save_path = os.path.join(lazyllm.config['train_target_root'], target_path)
            target_path, merge_path = os.path.join(save_path, "lazyllm_lora"), os.path.join(save_path, "lazyllm_merge")
            os.makedirs(target_path, exist_ok=True)
            os.makedirs(merge_path, exist_ok=True)
        super().__init__(
            base_model,
            target_path,
            launcher=launcher,
        )
        self.folder_path = os.path.dirname(os.path.abspath(__file__))
        deepspeed_config_path = os.path.join(self.folder_path, 'alpaca-lora', 'ds.json')
        self.kw = copy.deepcopy(self.defatult_kw)
        self.kw['deepspeed'] = deepspeed_config_path
        self.kw['nccl_port'] = random.randint(19000, 20500)
        self.kw.check_and_update(kw)
        self.merge_path = merge_path
        self.cp_files = cp_files
        self.model_name = model_name

    def cmd(self, trainset, valset=None) -> str:
        """Generate shell command sequence for Alpaca-LoRA fine-tuning and model merging.

Args:
    trainset (str): Training dataset path, supports both relative path (to configured data_path) and absolute path
    valset (str, optional): Validation dataset path, will auto-split from trainset if not specified

Returns:
    str or list: Returns a single command string when no merging needed, otherwise returns a list containing:
                 [fine-tune command, merge command, file copy command]




Examples:
    >>> from lazyllm import finetune
    >>> trainer = finetune.alpacalora('path/to/base/model', 'path/to/target')
    >>> cmd = trainer.cmd("my_dataset.json")

    """
        thirdparty.check_packages(['datasets', 'deepspeed', 'fire', 'numpy', 'peft', 'torch', 'transformers'])
        if not os.path.exists(trainset):
            defatult_path = os.path.join(lazyllm.config['data_path'], trainset)
            if os.path.exists(defatult_path):
                trainset = defatult_path
        if not self.kw['data_path']:
            self.kw['data_path'] = trainset

        run_file_path = os.path.join(self.folder_path, 'alpaca-lora', 'finetune.py')
        cmd = (f'python {run_file_path} '
               f'--base_model={self.base_model} '
               f'--output_dir={self.target_path} '
            )
        cmd += self.kw.parse_kwargs()
        cmd += f' 2>&1 | tee {os.path.join(self.target_path, self.model_name)}_$(date +"%Y-%m-%d_%H-%M-%S").log'

        if self.merge_path:
            run_file_path = os.path.join(self.folder_path, 'alpaca-lora', 'utils', 'merge_weights.py')

            cmd = [cmd,
                   f'python {run_file_path} '
                   f'--base={self.base_model} '
                   f'--adapter={self.target_path} '
                   f'--save_path={self.merge_path} ',
                   f' cp {os.path.join(self.base_model, self.cp_files)} {self.merge_path} '
                ]

        # cmd = 'realpath .'
        return cmd

cmd(trainset, valset=None)

Generate shell command sequence for Alpaca-LoRA fine-tuning and model merging.

Parameters:

  • trainset (str) –

    Training dataset path, supports both relative path (to configured data_path) and absolute path

  • valset (str, default: None ) –

    Validation dataset path, will auto-split from trainset if not specified

Returns:

  • str –

    str or list: Returns a single command string when no merging needed, otherwise returns a list containing: [fine-tune command, merge command, file copy command]

Examples:

>>> from lazyllm import finetune
>>> trainer = finetune.alpacalora('path/to/base/model', 'path/to/target')
>>> cmd = trainer.cmd("my_dataset.json")
Source code in lazyllm/components/finetune/alpacalora.py 
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
    def cmd(self, trainset, valset=None) -> str:
        """Generate shell command sequence for Alpaca-LoRA fine-tuning and model merging.

Args:
    trainset (str): Training dataset path, supports both relative path (to configured data_path) and absolute path
    valset (str, optional): Validation dataset path, will auto-split from trainset if not specified

Returns:
    str or list: Returns a single command string when no merging needed, otherwise returns a list containing:
                 [fine-tune command, merge command, file copy command]




Examples:
    >>> from lazyllm import finetune
    >>> trainer = finetune.alpacalora('path/to/base/model', 'path/to/target')
    >>> cmd = trainer.cmd("my_dataset.json")

    """
        thirdparty.check_packages(['datasets', 'deepspeed', 'fire', 'numpy', 'peft', 'torch', 'transformers'])
        if not os.path.exists(trainset):
            defatult_path = os.path.join(lazyllm.config['data_path'], trainset)
            if os.path.exists(defatult_path):
                trainset = defatult_path
        if not self.kw['data_path']:
            self.kw['data_path'] = trainset

        run_file_path = os.path.join(self.folder_path, 'alpaca-lora', 'finetune.py')
        cmd = (f'python {run_file_path} '
               f'--base_model={self.base_model} '
               f'--output_dir={self.target_path} '
            )
        cmd += self.kw.parse_kwargs()
        cmd += f' 2>&1 | tee {os.path.join(self.target_path, self.model_name)}_$(date +"%Y-%m-%d_%H-%M-%S").log'

        if self.merge_path:
            run_file_path = os.path.join(self.folder_path, 'alpaca-lora', 'utils', 'merge_weights.py')

            cmd = [cmd,
                   f'python {run_file_path} '
                   f'--base={self.base_model} '
                   f'--adapter={self.target_path} '
                   f'--save_path={self.merge_path} ',
                   f' cp {os.path.join(self.base_model, self.cp_files)} {self.merge_path} '
                ]

        # cmd = 'realpath .'
        return cmd

lazyllm.components.finetune.CollieFinetune

Bases: LazyLLMFinetuneBase

This class is a subclass of LazyLLMFinetuneBase, based on the LoRA fine-tuning capabilities provided by the Collie framework, used for LoRA fine-tuning of large language models.

Parameters:

  • base_model (str) –

    The base model used for fine-tuning. It is required to be the path of the base model.

  • target_path (str) –

    The path where the LoRA weights of the fine-tuned model are saved.

  • merge_path (str, default: None ) –

    The path where the model merges the LoRA weights, default to None. If not specified, "lazyllm_lora" and "lazyllm_merge" directories will be created under target_path as target_path and merge_path respectively.

  • model_name (str, default: 'LLM' ) –

    The name of the model, used as the prefix for setting the log name, default to "LLM".

  • cp_files (str, default: 'tokeniz*' ) –

    Specify configuration files to be copied from the base model path, which will be copied to merge_path, default to "tokeniz*"

  • launcher (launcher, default: remote(ngpus=1) ) –

    The launcher for fine-tuning, default to launchers.remote(ngpus=1).

  • kw –

    Keyword arguments, used to update the default training parameters. Note that additional keyword arguments cannot be arbitrarily specified.

The keyword arguments and their default values for this class are as follows:

Other Parameters:

  • data_path (str) –

    Data path, default to None; generally passed as the only positional argument when this object is called.

  • batch_size (int) –

    Batch size, default to 64.

  • micro_batch_size (int) –

    Micro-batch size, default to 4.

  • num_epochs (int) –

    Number of training epochs, default to 2.

  • learning_rate (float) –

    Learning rate, default to 5.e-4.

  • dp_size (int) –

    Data parallelism parameter, default to 8.

  • pp_size (int) –

    Pipeline parallelism parameter, default to 1.

  • tp_size (int) –

    Tensor parallelism parameter, default to 1.

  • lora_r (int) –

    LoRA rank, default to 8; this value determines the amount of parameters added, the smaller the value, the fewer the parameters.

  • lora_alpha (int) –

    LoRA fusion factor, default to 32; this value determines the impact of LoRA parameters on the base model parameters, the larger the value, the greater the impact.

  • lora_dropout (float) –

    LoRA dropout rate, default to 0.05, generally used to prevent overfitting.

  • lora_target_modules (str) –

    LoRA target modules, default to [wo,wqkv], which is the default for InternLM2 model; this configuration item varies for different models.

  • modules_to_save (str) –

    Modules for full fine-tuning, default to [tok_embeddings,output], which is the default for InternLM2 model; this configuration item varies for different models.

  • prompt_template_name (str) –

    The name of the prompt template, default to alpaca, i.e., use the prompt template provided by LazyLLM by default.

Examples:

>>> from lazyllm import finetune
>>> trainer = finetune.collie('path/to/base/model', 'path/to/target')
Source code in lazyllm/components/finetune/collie.py 
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
class CollieFinetune(LazyLLMFinetuneBase):
    """This class is a subclass of ``LazyLLMFinetuneBase``, based on the LoRA fine-tuning capabilities provided by the [Collie](https://github.com/OpenLMLab/collie) framework, used for LoRA fine-tuning of large language models.

Args:
    base_model (str): The base model used for fine-tuning. It is required to be the path of the base model.
    target_path (str): The path where the LoRA weights of the fine-tuned model are saved.
    merge_path (str): The path where the model merges the LoRA weights, default to ``None``. If not specified, "lazyllm_lora" and "lazyllm_merge" directories will be created under ``target_path`` as ``target_path`` and ``merge_path`` respectively.
    model_name (str): The name of the model, used as the prefix for setting the log name, default to "LLM".
    cp_files (str): Specify configuration files to be copied from the base model path, which will be copied to ``merge_path``, default to "tokeniz*"
    launcher (lazyllm.launcher): The launcher for fine-tuning, default to ``launchers.remote(ngpus=1)``.
    kw: Keyword arguments, used to update the default training parameters. Note that additional keyword arguments cannot be arbitrarily specified.

The keyword arguments and their default values for this class are as follows:

Keyword Args: 
    data_path (str): Data path, default to ``None``; generally passed as the only positional argument when this object is called.
    batch_size (int): Batch size, default to ``64``.
    micro_batch_size (int): Micro-batch size, default to ``4``.
    num_epochs (int): Number of training epochs, default to ``2``.
    learning_rate (float): Learning rate, default to ``5.e-4``.
    dp_size (int): Data parallelism parameter, default to `` 8``.
    pp_size (int): Pipeline parallelism parameter, default to ``1``.
    tp_size (int): Tensor parallelism parameter, default to ``1``.
    lora_r (int): LoRA rank, default to ``8``; this value determines the amount of parameters added, the smaller the value, the fewer the parameters.
    lora_alpha (int): LoRA fusion factor, default to ``32``; this value determines the impact of LoRA parameters on the base model parameters, the larger the value, the greater the impact.
    lora_dropout (float): LoRA dropout rate, default to ``0.05``, generally used to prevent overfitting.
    lora_target_modules (str): LoRA target modules, default to ``[wo,wqkv]``, which is the default for InternLM2 model; this configuration item varies for different models.
    modules_to_save (str): Modules for full fine-tuning, default to ``[tok_embeddings,output]``, which is the default for InternLM2 model; this configuration item varies for different models.
    prompt_template_name (str): The name of the prompt template, default to ``alpaca``, i.e., use the prompt template provided by LazyLLM by default.



Examples:
    >>> from lazyllm import finetune
    >>> trainer = finetune.collie('path/to/base/model', 'path/to/target')
    """
    defatult_kw = ArgsDict({
        'data_path': None,
        'batch_size': 64,
        'micro_batch_size': 4,
        'num_epochs': 3,
        'learning_rate': 5.e-4,
        'dp_size': 8,
        'pp_size': 1,
        'tp_size': 1,
        'lora_r': 8,
        'lora_alpha': 16,
        'lora_dropout': 0.05,
        'lora_target_modules': '[wo,wqkv]',
        'modules_to_save': '[tok_embeddings,output]',
        'prompt_template_name': 'alpaca',
    })
    auto_map = {
        'ddp': 'dp_size',
        'micro_batch_size': 'micro_batch_size',
        'tp': 'tp_size',
    }

    def __init__(self,
                 base_model,
                 target_path,
                 merge_path=None,
                 model_name='LLM',
                 cp_files='tokeniz*',
                 launcher=launchers.remote(ngpus=1),  # noqa B008
                 **kw
                 ):
        if not merge_path:
            save_path = os.path.join(lazyllm.config['train_target_root'], target_path)
            target_path, merge_path = os.path.join(save_path, "lazyllm_lora"), os.path.join(save_path, "lazyllm_merge")
            os.makedirs(target_path, exist_ok=True)
            os.makedirs(merge_path, exist_ok=True)
        super().__init__(
            base_model,
            target_path,
            launcher=launcher,
        )
        self.folder_path = os.path.dirname(os.path.abspath(__file__))
        self.kw = copy.deepcopy(self.defatult_kw)
        self.kw.check_and_update(kw)
        self.merge_path = merge_path
        self.cp_files = cp_files
        self.model_name = model_name

    def cmd(self, trainset, valset=None) -> str:
        thirdparty.check_packages(['numpy', 'peft', 'torch', 'transformers'])
        if not os.path.exists(trainset):
            defatult_path = os.path.join(lazyllm.config['data_path'], trainset)
            if os.path.exists(defatult_path):
                trainset = defatult_path
        if not self.kw['data_path']:
            self.kw['data_path'] = trainset

        run_file_path = os.path.join(self.folder_path, 'collie', 'finetune.py')
        cmd = (f'python {run_file_path} '
               f'--base_model={self.base_model} '
               f'--output_dir={self.target_path} '
            )
        cmd += self.kw.parse_kwargs()
        cmd += f' 2>&1 | tee {os.path.join(self.target_path, self.model_name)}_$(date +"%Y-%m-%d_%H-%M-%S").log'

        if self.merge_path:
            run_file_path = os.path.join(self.folder_path, 'alpaca-lora', 'utils', 'merge_weights.py')

            cmd = [cmd,
                   f'python {run_file_path} '
                   f'--base={self.base_model} '
                   f'--adapter={self.target_path} '
                   f'--save_path={self.merge_path} ',
                   f' cp {os.path.join(self.base_model,self.cp_files)} {self.merge_path} '
                ]

        return cmd

lazyllm.components.finetune.LlamafactoryFinetune

Bases: LazyLLMFinetuneBase

This class is a subclass of LazyLLMFinetuneBase, based on the training capabilities provided by the LLaMA-Factory framework, used for training large language models(or visual language models).

Parameters:

  • base_model (str) –

    Path to the base model used for training. Supports local paths; if the path does not exist, it will attempt to locate it from the configured model directory.

  • target_path (str) –

    Target directory to save model weights after training is completed.

  • merge_path (str, default: None ) –

    Path to save the model after merging LoRA weights. Defaults to None. If not specified, two directories will be automatically created under target_path: - "lazyllm_lora" (for storing LoRA fine-tuned weights) - "lazyllm_merge" (for storing the merged model weights)

  • config_path (str, default: None ) –

    Path to the YAML file containing training configuration. Defaults to None. If not specified, the default config file llama_factory/sft.yaml will be used. This file can override default training parameters.

  • export_config_path (str, default: None ) –

    Path to the YAML file for LoRA weight export/merging configuration. Defaults to None. If not specified, the default config file llama_factory/lora_export.yaml will be used.

  • lora_r (int, default: None ) –

    Rank of the LoRA adaptation. If provided, overrides the lora_rank value in the configuration.

  • modules_to_save (str, default: None ) –

    List of additional module names to be saved. Should be provided as a string in Python list format, e.g., "[module1, module2]".

  • lora_target_modules (str, default: None ) –

    List of module names to apply LoRA fine-tuning to. Format is the same as above.

  • launcher (launcher, default: remote(ngpus=1, sync=True) ) –

    Launcher for the fine-tuning task. Defaults to a single-GPU, synchronous remote launcher: launchers.remote(ngpus=1, sync=True).

  • **kw –

    Additional keyword arguments used to dynamically override default parameters in the training configuration.

Other Parameters:

  • stage (Literal['pt', 'sft', 'rm', 'ppo', 'dpo', 'kto']) –

    Default is: sft. Which stage will be performed in training.

  • do_train (bool) –

    Default is: True. Whether to run training.

  • finetuning_type (Literal['lora', 'freeze', 'full']) –

    Default is: lora. Which fine-tuning method to use.

  • lora_target (str) –

    Default is: all. Name(s) of target modules to apply LoRA. Use commas to separate multiple modules. Use all to specify all the linear modules.

  • template (Optional[str]) –

    Default is: None. Which template to use for constructing prompts in training and inference.

  • cutoff_len (int) –

    Default is: 1024. The cutoff length of the tokenized inputs in the dataset.

  • max_samples (Optional[int]) –

    Default is: 1000. For debugging purposes, truncate the number of examples for each dataset.

  • overwrite_cache (bool) –

    Default is: True. Overwrite the cached training and evaluation sets.

  • preprocessing_num_workers (Optional[int]) –

    Default is: 16. The number of processes to use for the pre-processing.

  • dataset_dir (str) –

    Default is: lazyllm_temp_dir. Path to the folder containing the datasets. If not explicitly specified, LazyLLM will generate a dataset_info.json file in the .temp folder in the current working directory for use by LLaMA-Factory.

  • logging_steps (float) –

    Default is: 10. Log every X updates steps. Should be an integer or a float in range [0,1). If smaller than 1, will be interpreted as ratio of total training steps.

  • save_steps (float) –

    Default is: 500. Save checkpoint every X updates steps. Should be an integer or a float in range [0,1). If smaller than 1, will be interpreted as ratio of total training steps.

  • plot_loss (bool) –

    Default is: True. Whether or not to save the training loss curves.

  • overwrite_output_dir (bool) –

    Default is: True. Overwrite the content of the output directory.

  • per_device_train_batch_size (int) –

    Default is: 1. Batch size per GPU/TPU/MPS/NPU core/CPU for training.

  • gradient_accumulation_steps (int) –

    Default is: 8. Number of updates steps to accumulate before performing a backward/update pass.

  • learning_rate (float) –

    Default is: 1e-04. The initial learning rate for AdamW.

  • num_train_epochs (float) –

    Default is: 3.0. Total number of training epochs to perform.

  • lr_scheduler_type (Union[SchedulerType, str]) –

    Default is: cosine. The scheduler type to use.

  • warmup_ratio (float) –

    Default is: 0.1. Linear warmup over warmup_ratio fraction of total steps.

  • fp16 (bool) –

    Default is: True. Whether to use fp16 (mixed) precision instead of 32-bit.

  • ddp_timeout (Optional[int]) –

    Default is: 180000000. Overrides the default timeout for distributed training (value should be given in seconds).

  • report_to (Union[NoneType, str, List[str]]) –

    Default is: tensorboard. The list of integrations to report the results and logs to.

  • val_size (float) –

    Default is: 0.1. Size of the development set, should be an integer or a float in range [0,1).

  • per_device_eval_batch_size (int) –

    Default is: 1. Batch size per GPU/TPU/MPS/NPU core/CPU for evaluation.

  • eval_strategy (Union[IntervalStrategy, str]) –

    Default is: steps. The evaluation strategy to use.

  • eval_steps (Optional[float]) –

    Default is: 500. Run an evaluation every X steps. Should be an integer or a float in range [0,1). If smaller than 1, will be interpreted as ratio of total training steps.

Examples:

>>> from lazyllm import finetune
>>> trainer = finetune.llamafactory('internlm2-chat-7b', 'path/to/target')
<lazyllm.llm.finetune type=LlamafactoryFinetune>
Source code in lazyllm/components/finetune/llamafactory.py 
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
class LlamafactoryFinetune(LazyLLMFinetuneBase):
    """This class is a subclass of ``LazyLLMFinetuneBase``, based on the training capabilities provided by the [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) framework, used for training large language models(or visual language models).

Args:
    base_model (str): Path to the base model used for training. Supports local paths; if the path does not exist, it will attempt to locate it from the configured model directory.
    target_path (str): Target directory to save model weights after training is completed.
    merge_path (str, optional): Path to save the model after merging LoRA weights. Defaults to None.
        If not specified, two directories will be automatically created under ``target_path``:
        - "lazyllm_lora" (for storing LoRA fine-tuned weights)
        - "lazyllm_merge" (for storing the merged model weights)
    config_path (str, optional): Path to the YAML file containing training configuration. Defaults to None.
        If not specified, the default config file ``llama_factory/sft.yaml`` will be used.
        This file can override default training parameters.
    export_config_path (str, optional): Path to the YAML file for LoRA weight export/merging configuration. Defaults to None.
        If not specified, the default config file ``llama_factory/lora_export.yaml`` will be used.
    lora_r (int, optional): Rank of the LoRA adaptation. If provided, overrides the ``lora_rank`` value in the configuration.
    modules_to_save (str, optional): List of additional module names to be saved. Should be provided as a string in Python list format, e.g., "[module1, module2]".
    lora_target_modules (str, optional): List of module names to apply LoRA fine-tuning to. Format is the same as above.
    launcher (lazyllm.launcher, optional): Launcher for the fine-tuning task. Defaults to a single-GPU, synchronous remote launcher: ``launchers.remote(ngpus=1, sync=True)``.
    **kw: Additional keyword arguments used to dynamically override default parameters in the training configuration.

Keyword Args:
    stage (typing.Literal['pt', 'sft', 'rm', 'ppo', 'dpo', 'kto']): Default is: ``sft``. Which stage will be performed in training.
    do_train (bool): Default is: ``True``. Whether to run training.
    finetuning_type (typing.Literal['lora', 'freeze', 'full']): Default is: ``lora``. Which fine-tuning method to use.
    lora_target (str): Default is: ``all``. Name(s) of target modules to apply LoRA. Use commas to separate multiple modules. Use `all` to specify all the linear modules.
    template (typing.Optional[str]): Default is: ``None``. Which template to use for constructing prompts in training and inference.
    cutoff_len (int): Default is: ``1024``. The cutoff length of the tokenized inputs in the dataset.
    max_samples (typing.Optional[int]): Default is: ``1000``. For debugging purposes, truncate the number of examples for each dataset.
    overwrite_cache (bool): Default is: ``True``. Overwrite the cached training and evaluation sets.
    preprocessing_num_workers (typing.Optional[int]): Default is: ``16``. The number of processes to use for the pre-processing.
    dataset_dir (str): Default is: ``lazyllm_temp_dir``. Path to the folder containing the datasets. If not explicitly specified, LazyLLM will generate a ``dataset_info.json`` file in the ``.temp`` folder in the current working directory for use by LLaMA-Factory.
    logging_steps (float): Default is: ``10``. Log every X updates steps. Should be an integer or a float in range ``[0,1)``. If smaller than 1, will be interpreted as ratio of total training steps.
    save_steps (float): Default is: ``500``. Save checkpoint every X updates steps. Should be an integer or a float in range ``[0,1)``. If smaller than 1, will be interpreted as ratio of total training steps.
    plot_loss (bool): Default is: ``True``. Whether or not to save the training loss curves.
    overwrite_output_dir (bool): Default is: ``True``. Overwrite the content of the output directory.
    per_device_train_batch_size (int): Default is: ``1``. Batch size per GPU/TPU/MPS/NPU core/CPU for training.
    gradient_accumulation_steps (int): Default is: ``8``. Number of updates steps to accumulate before performing a backward/update pass.
    learning_rate (float): Default is: ``1e-04``. The initial learning rate for AdamW.
    num_train_epochs (float): Default is: ``3.0``. Total number of training epochs to perform.
    lr_scheduler_type (typing.Union[transformers.trainer_utils.SchedulerType, str]): Default is: ``cosine``. The scheduler type to use.
    warmup_ratio (float): Default is: ``0.1``. Linear warmup over warmup_ratio fraction of total steps.
    fp16 (bool): Default is: ``True``. Whether to use fp16 (mixed) precision instead of 32-bit.
    ddp_timeout (typing.Optional[int]): Default is: ``180000000``. Overrides the default timeout for distributed training (value should be given in seconds).
    report_to (typing.Union[NoneType, str, typing.List[str]]): Default is: ``tensorboard``. The list of integrations to report the results and logs to.
    val_size (float): Default is: ``0.1``. Size of the development set, should be an integer or a float in range `[0,1)`.
    per_device_eval_batch_size (int): Default is: ``1``. Batch size per GPU/TPU/MPS/NPU core/CPU for evaluation.
    eval_strategy (typing.Union[transformers.trainer_utils.IntervalStrategy, str]): Default is: ``steps``. The evaluation strategy to use.
    eval_steps (typing.Optional[float]): Default is: ``500``. Run an evaluation every X steps. Should be an integer or a float in range `[0,1)`. If smaller than 1, will be interpreted as ratio of total training steps.



Examples:
    >>> from lazyllm import finetune
    >>> trainer = finetune.llamafactory('internlm2-chat-7b', 'path/to/target')
    <lazyllm.llm.finetune type=LlamafactoryFinetune>
    """
    auto_map = {
        'gradient_step': 'gradient_accumulation_steps',
        'micro_batch_size': 'per_device_train_batch_size',
    }

    def __init__(self,
                 base_model,
                 target_path,
                 merge_path=None,
                 config_path=None,
                 export_config_path=None,
                 lora_r=None,
                 modules_to_save=None,
                 lora_target_modules=None,
                 launcher=launchers.remote(ngpus=1, sync=True),  # noqa B008
                 **kw
                 ):
        if not os.path.exists(base_model):
            defatult_path = os.path.join(lazyllm.config['model_path'], base_model)
            if os.path.exists(defatult_path):
                base_model = defatult_path
        if not merge_path:
            save_path = os.path.join(lazyllm.config['train_target_root'], target_path)
            target_path, merge_path = os.path.join(save_path, "lazyllm_lora"), os.path.join(save_path, "lazyllm_merge")
            os.system(f'mkdir -p {target_path} {merge_path}')
        super().__init__(
            base_model,
            target_path,
            launcher=launcher,
        )
        self.merge_path = merge_path
        self.temp_yaml_file = None
        self.temp_export_yaml_file = None
        self.config_path = config_path
        self.export_config_path = export_config_path
        self.config_folder_path = os.path.dirname(os.path.abspath(__file__))

        default_config_path = os.path.join(self.config_folder_path, 'llama_factory', 'sft.yaml')
        self.template_dict = ArgsDict(self._load_yaml(default_config_path))

        if self.config_path:
            self.template_dict.update(self._load_yaml(self.config_path))

        if lora_r:
            self.template_dict['lora_rank'] = lora_r
        if modules_to_save:
            self.template_dict['additional_target'] = modules_to_save.strip('[]')
        if lora_target_modules:
            self.template_dict['lora_target'] = lora_target_modules.strip('[]')
        self.template_dict['model_name_or_path'] = base_model
        self.template_dict['output_dir'] = target_path
        self.template_dict['template'] = self._get_template_name(base_model)
        self.template_dict.check_and_update(kw)

        default_export_config_path = os.path.join(self.config_folder_path, 'llama_factory', 'lora_export.yaml')
        self.export_dict = ArgsDict(self._load_yaml(default_export_config_path))

        if self.export_config_path:
            self.export_dict.update(self._load_yaml(self.export_config_path))

        self.export_dict['model_name_or_path'] = base_model
        self.export_dict['adapter_name_or_path'] = target_path
        self.export_dict['export_dir'] = merge_path
        self.export_dict['template'] = self.template_dict['template']

        self.temp_folder = os.path.join(lazyllm.config['temp_dir'], 'llamafactory_config', str(uuid.uuid4())[:10])
        if not os.path.exists(self.temp_folder):
            os.makedirs(self.temp_folder)
        self.log_file_path = None

    def _get_template_name(self, base_model):
        base_name = os.path.basename(base_model).lower()
        key_value = match_longest_prefix(base_name)
        if key_value:
            return key_value
        else:
            raise RuntimeError(f'Cannot find prfix of base_model({base_model}) '
                               f'in DEFAULT_TEMPLATE of LLaMA_Factory: {llamafactory_mapping_dict}')

    def _load_yaml(self, config_path):
        with open(config_path, 'r') as file:
            config_dict = yaml.safe_load(file)
        return config_dict

    def _build_temp_yaml(self, updated_template_str, prefix='train_'):
        fd, temp_yaml_file = tempfile.mkstemp(prefix=prefix, suffix='.yaml', dir=self.temp_folder)
        with os.fdopen(fd, 'w') as temp_file:
            temp_file.write(updated_template_str)
        return temp_yaml_file

    def _build_temp_dataset_info(self, datapaths):
        if isinstance(datapaths, str):
            datapaths = [datapaths]
        elif isinstance(datapaths, list) and all(isinstance(item, str) for item in datapaths):
            pass
        else:
            raise TypeError(f'datapaths({datapaths}) should be str or list of str.')
        temp_dataset_dict = dict()
        for datapath in datapaths:
            datapath = os.path.join(lazyllm.config['data_path'], datapath)
            assert os.path.isfile(datapath)
            file_name, _ = os.path.splitext(os.path.basename(datapath))
            temp_dataset_dict[file_name] = {'file_name': datapath}
            formatting = 'alpaca'
            try:
                with open(datapath, 'r', encoding='utf-8') as file:
                    data = json.load(file)
                if 'messages' in data[0]:
                    formatting = 'sharegpt'
                media_types = []
                for media in ['images', 'videos', 'audios']:
                    if media in data[0]:
                        media_types.append(media)
                if media_types:
                    columns = {item: item for item in media_types}
                    columns.update({"messages": "messages"})
                    temp_dataset_dict[file_name].update({
                        "tags": {
                            "role_tag": "role",
                            "content_tag": "content",
                            "user_tag": "user",
                            "assistant_tag": "assistant"
                        },
                        "columns": columns
                    })
            except Exception:
                pass
            temp_dataset_dict[file_name].update({'formatting': formatting})
        self.temp_dataset_info_path = os.path.join(self.temp_folder, 'dataset_info.json')
        with open(self.temp_dataset_info_path, 'w') as json_file:
            json.dump(temp_dataset_dict, json_file, indent=4)
        return self.temp_dataset_info_path, ','.join(temp_dataset_dict.keys())

    def _rm_temp_yaml(self):
        if self.temp_yaml_file:
            if os.path.exists(self.temp_yaml_file):
                os.remove(self.temp_yaml_file)
            self.temp_yaml_file = None

    def cmd(self, trainset, valset=None) -> str:
        """Generate LLaMA-Factory fine-tuning command sequence, including training and model merge commands.

Args:
    trainset (str): Training dataset path (supports relative path to lazyllm.config['data_path'])
    valset (str, optional): Validation dataset path (not directly used in current implementation)

Returns:
    str: Complete shell command string containing:
         - Training command (with auto-configured parameters)
         - Log redirection (saved to target path)
         - Optional model merge command (when LoRA is configured)

Notes:
    - Automatically generates timestamped training log files
    - Temporary files are automatically cleaned up after use
    - Supports multiple data formats (alpaca/sharegpt etc.)
    - Multimodal data (images/videos/audios) is automatically detected and handled
"""
        thirdparty.check_packages(['datasets', 'deepspeed', 'numpy', 'peft', 'torch', 'transformers', 'trl'])
        # train config update
        if 'dataset_dir' in self.template_dict and self.template_dict['dataset_dir'] == 'lazyllm_temp_dir':
            _, datasets = self._build_temp_dataset_info(trainset)
            self.template_dict['dataset_dir'] = self.temp_folder
        else:
            datasets = trainset
        self.template_dict['dataset'] = datasets

        # save config update
        if self.template_dict['finetuning_type'] == 'lora':
            updated_export_str = yaml.dump(dict(self.export_dict), default_flow_style=False)
            self.temp_export_yaml_file = self._build_temp_yaml(updated_export_str, prefix='merge_')

        updated_template_str = yaml.dump(dict(self.template_dict), default_flow_style=False)
        self.temp_yaml_file = self._build_temp_yaml(updated_template_str)

        formatted_date = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
        random_value = random.randint(1000, 9999)
        self.log_file_path = f'{self.target_path}/train_log_{formatted_date}_{random_value}.log'

        cmds = f'export DISABLE_VERSION_CHECK=1 && llamafactory-cli train {self.temp_yaml_file}'
        cmds += f' 2>&1 | tee {self.log_file_path}'
        if self.temp_export_yaml_file:
            cmds += f' && llamafactory-cli export {self.temp_export_yaml_file}'
        return cmds

cmd(trainset, valset=None)

Generate LLaMA-Factory fine-tuning command sequence, including training and model merge commands.

Parameters:

  • trainset (str) –

    Training dataset path (supports relative path to lazyllm.config['data_path'])

  • valset (str, default: None ) –

    Validation dataset path (not directly used in current implementation)

Returns:

  • str ( str ) –

    Complete shell command string containing: - Training command (with auto-configured parameters) - Log redirection (saved to target path) - Optional model merge command (when LoRA is configured)

Notes
  • Automatically generates timestamped training log files
  • Temporary files are automatically cleaned up after use
  • Supports multiple data formats (alpaca/sharegpt etc.)
  • Multimodal data (images/videos/audios) is automatically detected and handled
Source code in lazyllm/components/finetune/llamafactory.py 
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
    def cmd(self, trainset, valset=None) -> str:
        """Generate LLaMA-Factory fine-tuning command sequence, including training and model merge commands.

Args:
    trainset (str): Training dataset path (supports relative path to lazyllm.config['data_path'])
    valset (str, optional): Validation dataset path (not directly used in current implementation)

Returns:
    str: Complete shell command string containing:
         - Training command (with auto-configured parameters)
         - Log redirection (saved to target path)
         - Optional model merge command (when LoRA is configured)

Notes:
    - Automatically generates timestamped training log files
    - Temporary files are automatically cleaned up after use
    - Supports multiple data formats (alpaca/sharegpt etc.)
    - Multimodal data (images/videos/audios) is automatically detected and handled
"""
        thirdparty.check_packages(['datasets', 'deepspeed', 'numpy', 'peft', 'torch', 'transformers', 'trl'])
        # train config update
        if 'dataset_dir' in self.template_dict and self.template_dict['dataset_dir'] == 'lazyllm_temp_dir':
            _, datasets = self._build_temp_dataset_info(trainset)
            self.template_dict['dataset_dir'] = self.temp_folder
        else:
            datasets = trainset
        self.template_dict['dataset'] = datasets

        # save config update
        if self.template_dict['finetuning_type'] == 'lora':
            updated_export_str = yaml.dump(dict(self.export_dict), default_flow_style=False)
            self.temp_export_yaml_file = self._build_temp_yaml(updated_export_str, prefix='merge_')

        updated_template_str = yaml.dump(dict(self.template_dict), default_flow_style=False)
        self.temp_yaml_file = self._build_temp_yaml(updated_template_str)

        formatted_date = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
        random_value = random.randint(1000, 9999)
        self.log_file_path = f'{self.target_path}/train_log_{formatted_date}_{random_value}.log'

        cmds = f'export DISABLE_VERSION_CHECK=1 && llamafactory-cli train {self.temp_yaml_file}'
        cmds += f' 2>&1 | tee {self.log_file_path}'
        if self.temp_export_yaml_file:
            cmds += f' && llamafactory-cli export {self.temp_export_yaml_file}'
        return cmds

lazyllm.components.deploy.LazyLLMDeployBase

Bases: ComponentBase

This class is a subclass of ComponentBase that provides basic functionality for LazyLLM deployment. It supports encoding conversion for various media types and provides configuration options for result extraction and streaming processing.

Parameters:

  • launcher (LauncherBase, default: remote() ) –

    Launcher instance for deployment, defaults to remote launcher (launchers.remote()).

Notes
  • Need to implement specific deployment logic when inheriting this class
  • Can customize result extraction logic by overriding the extract_result method

Examples:

>>> import lazyllm
>>> from lazyllm.components.deploy.base import LazyLLMDeployBase
>>> class MyDeployer(LazyLLMDeployBase):
...     def __call__(self, inputs):
...         return processed_result
        def extract_result(output, inputs):
...         return output.json()['result']
>>> deployer = MyDeployer()
>>> result = deployer.extract_result(raw_output, input_data)
Source code in lazyllm/components/deploy/base.py 
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
class LazyLLMDeployBase(ComponentBase):
    """This class is a subclass of ``ComponentBase`` that provides basic functionality for LazyLLM deployment. It supports encoding conversion for various media types and provides configuration options for result extraction and streaming processing.

Args:
    launcher (LauncherBase): Launcher instance for deployment, defaults to remote launcher (``launchers.remote()``).

Notes: 
    - Need to implement specific deployment logic when inheriting this class
    - Can customize result extraction logic by overriding the extract_result method


Examples:
    >>> import lazyllm
    >>> from lazyllm.components.deploy.base import LazyLLMDeployBase
    >>> class MyDeployer(LazyLLMDeployBase):
    ...     def __call__(self, inputs):
    ...         return processed_result
            def extract_result(output, inputs):
    ...         return output.json()['result']
    >>> deployer = MyDeployer()
    >>> result = deployer.extract_result(raw_output, input_data)
    """
    keys_name_handle = None
    message_format = None
    default_headers = {'Content-Type': 'application/json'}
    stream_url_suffix = ''
    stream_parse_parameters = {}

    encoder_map = dict(image=_image_to_base64, audio=_audio_to_base64, ocr_files=ocr_to_base64)

    @staticmethod
    def extract_result(output, inputs):
        return output

    def __init__(self, *, launcher=launchers.remote()):  # noqa B008
        super().__init__(launcher=launcher)

lazyllm.components.deploy.LazyLLMDeployBase.extract_result(output, inputs) staticmethod

Source code in lazyllm/components/deploy/base.py 
39
40
41
@staticmethod
def extract_result(output, inputs):
    return output

lazyllm.components.finetune.FlagembeddingFinetune

Bases: LazyLLMFinetuneBase

This class is a subclass of LazyLLMFinetuneBase, based on the training capabilities provided by the FlagEmbedding framework, used for training embedding and reranker models.

Parameters:

  • base_model (str) –

    The base model used for training. It is required to be the path of the base model.

  • target_path (str) –

    The path where the trained model weights are saved.

  • launcher (launcher, default: remote(ngpus=1, sync=True) ) –

    The launcher for fine-tuning, default is launchers.remote(ngpus=1, sync=True).

  • kw –

    Keyword arguments used to update the default training parameters.

The keyword arguments and their default values for this class of embedding model are as follows:

Other Parameters:

  • train_group_size (int) –

    Default is: 8. The size of train group. It is used to control the number of negative samples in each training set.

  • query_max_len (int) –

    Default is: 512. The maximum total input sequence length after tokenization for passage. Sequences longer than this will be truncated, sequences shorter will be padded.

  • passage_max_len (int) –

    Default is: 512. The maximum total input sequence length after tokenization for passage. Sequences longer than this will be truncated, sequences shorter will be padded.

  • pad_to_multiple_of (int) –

    Default is: 8. If set will pad the sequence to be a multiple of the provided value.

  • query_instruction_for_retrieval (str) –

    Default is: Represent this sentence for searching relevant passages:. Instruction for query.

  • query_instruction_format (str) –

    Default is: {}{}. Format for query instruction.

  • learning_rate (float) –

    Default is: 1e-5. Learning rate.

  • num_train_epochs (int) –

    Default is: 1. Total number of training epochs to perform.

  • per_device_train_batch_size (int) –

    Default is: 2. Train batch size

  • gradient_accumulation_steps (int) –

    Default is: 1. Number of updates steps to accumulate before performing a backward/update pass.

  • dataloader_drop_last (bool) –

    Default is: True. When it='True', the last incomplete batch is dropped if the dataset size is not divisible by the batch size, meaning DataLoader only returns complete batches.

  • warmup_ratio (float) –

    Default is: 0.1. Warmup ratio for linear scheduler.

  • weight_decay (float) –

    Default is: 0.01. Weight decay in AdamW.

  • deepspeed (str) –

    Default is: ``. The path of the DeepSpeed configuration file, default to use the pre-made configuration file in the LazyLLM code repository:ds_stage0.json``.

  • logging_steps (int) –

    Default is: 1. Logging frequency according to logging strategy.

  • save_steps (int) –

    Default is: 1000. Saving frequency.

  • temperature (float) –

    Default is: 0.02. Temperature used for similarity score

  • sentence_pooling_method (str) –

    Default is: cls. The pooling method. Available options: 'cls', 'mean', 'last_token'.

  • normalize_embeddings (bool) –

    Default is: True. Whether to normalize the embeddings.

  • kd_loss_type (str) –

    Default is: kl_div. The loss type for knowledge distillation. Available options:'kl_div', 'm3_kd_loss'.

  • overwrite_output_dir (bool) –

    Default is: True. It is used to allow the program to overwrite an existing output directory.

  • fp16 (bool) –

    Default is: True. Whether to use fp16 (mixed) precision instead of 32-bit.

  • gradient_checkpointing (bool) –

    Default is: True. Whether enable gradient checkpointing.

  • negatives_cross_device (bool) –

    Default is: True. Whether share negatives across devices.

The keyword arguments and their default values for this class of reranker model are as follows:

Other Parameters:

  • train_group_size (int) –

    Default is: 8. The size of train group. It is used to control the number of negative samples in each training set.

  • query_max_len (int) –

    Default is: 256. The maximum total input sequence length after tokenization for passage. Sequences longer than this will be truncated, sequences shorter will be padded.

  • passage_max_len (int) –

    Default is: 256. The maximum total input sequence length after tokenization for passage. Sequences longer than this will be truncated, sequences shorter will be padded.

  • pad_to_multiple_of (int) –

    Default is: 8. If set will pad the sequence to be a multiple of the provided value.

  • learning_rate (float) –

    Default is: 6e-5. Learning rate.

  • num_train_epochs (int) –

    Default is: 1. Total number of training epochs to perform.

  • per_device_train_batch_size (int) –

    Default is: 2. Train batch size

  • gradient_accumulation_steps (int) –

    Default is: 1. Number of updates steps to accumulate before performing a backward/update pass.

  • dataloader_drop_last (bool) –

    Default is: True. When it='True', the last incomplete batch is dropped if the dataset size is not divisible by the batch size, meaning DataLoader only returns complete batches.

  • warmup_ratio (float) –

    Default is: 0.1. Warmup ratio for linear scheduler.

  • weight_decay (float) –

    Default is: 0.01. Weight decay in AdamW.

  • deepspeed (str) –

    Default is: ``. The path of the DeepSpeed configuration file, default to use the pre-made configuration file in the LazyLLM code repository:ds_stage0.json``.

  • logging_steps (int) –

    Default is: 1. Logging frequency according to logging strategy.

  • save_steps (int) –

    Default is: 1000. Saving frequency.

  • overwrite_output_dir (bool) –

    Default is: True. It is used to allow the program to overwrite an existing output directory.

  • fp16 (bool) –

    Default is: True. Whether to use fp16 (mixed) precision instead of 32-bit.

  • gradient_checkpointing (bool) –

    Default is: True. Whether enable gradient checkpointing.

Examples:

>>> from lazyllm import finetune
>>> finetune.FlagembeddingFinetune('bge-m3', 'path/to/target')
<lazyllm.llm.finetune type=FlagembeddingFinetune>
Source code in lazyllm/components/finetune/flagembedding.py 
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
class FlagembeddingFinetune(LazyLLMFinetuneBase):
    """This class is a subclass of ``LazyLLMFinetuneBase``, based on the training capabilities provided by the [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding) framework, used for training embedding and reranker models.

Args:
    base_model (str): The base model used for training. It is required to be the path of the base model.
    target_path (str): The path where the trained model weights are saved.
    launcher (lazyllm.launcher): The launcher for fine-tuning, default is ``launchers.remote(ngpus=1, sync=True)``.
    kw: Keyword arguments used to update the default training parameters.

The keyword arguments and their default values for this class of embedding model are as follows:

Keyword Args:
    train_group_size (int): Default is: ``8``. The size of train group. It is used to control the number of negative samples in each training set.
    query_max_len (int): Default is: ``512``. The maximum total input sequence length after tokenization for passage. Sequences longer than this will be truncated, sequences shorter will be padded.
    passage_max_len (int): Default is: ``512``. The maximum total input sequence length after tokenization for passage. Sequences longer than this will be truncated, sequences shorter will be padded.
    pad_to_multiple_of (int): Default is: ``8``. If set will pad the sequence to be a multiple of the provided value.
    query_instruction_for_retrieval (str): Default is: ``Represent this sentence for searching relevant passages: ``. Instruction for query.
    query_instruction_format (str): Default is: ``{}{}``. Format for query instruction.
    learning_rate (float): Default is: ``1e-5``. Learning rate.
    num_train_epochs (int): Default is: ``1``. Total number of training epochs to perform.
    per_device_train_batch_size (int): Default is: ``2``. Train batch size
    gradient_accumulation_steps (int): Default is: ``1``. Number of updates steps to accumulate before performing a backward/update pass.
    dataloader_drop_last (bool): Default is: ``True``. When it='True', the last incomplete batch is dropped if the dataset size is not divisible by the batch size, meaning DataLoader only returns complete batches.
    warmup_ratio (float): Default is: ``0.1``. Warmup ratio for linear scheduler.
    weight_decay (float): Default is: ``0.01``. Weight decay in AdamW.
    deepspeed (str): Default is: ````. The path of the DeepSpeed configuration file, default to use the pre-made configuration file in the LazyLLM code repository: ``ds_stage0.json``.
    logging_steps (int): Default is: ``1``. Logging frequency according to logging strategy.
    save_steps (int): Default is: ``1000``. Saving frequency.
    temperature (float): Default is: ``0.02``. Temperature used for similarity score
    sentence_pooling_method (str): Default is: ``cls``. The pooling method. Available options: 'cls', 'mean', 'last_token'.
    normalize_embeddings (bool): Default is: ``True``. Whether to normalize the embeddings.
    kd_loss_type (str): Default is: ``kl_div``. The loss type for knowledge distillation. Available options:'kl_div', 'm3_kd_loss'.
    overwrite_output_dir (bool): Default is: ``True``. It is used to allow the program to overwrite an existing output directory.
    fp16 (bool): Default is: ``True``.  Whether to use fp16 (mixed) precision instead of 32-bit.
    gradient_checkpointing (bool): Default is: ``True``. Whether enable gradient checkpointing.
    negatives_cross_device (bool): Default is: ``True``. Whether share negatives across devices.

The keyword arguments and their default values for this class of reranker model are as follows:

Keyword Args:
    train_group_size (int): Default is: ``8``. The size of train group. It is used to control the number of negative samples in each training set.
    query_max_len (int): Default is: ``256``. The maximum total input sequence length after tokenization for passage. Sequences longer than this will be truncated, sequences shorter will be padded.
    passage_max_len (int): Default is: ``256``. The maximum total input sequence length after tokenization for passage. Sequences longer than this will be truncated, sequences shorter will be padded.
    pad_to_multiple_of (int): Default is: ``8``. If set will pad the sequence to be a multiple of the provided value.
    learning_rate (float): Default is: ``6e-5``. Learning rate.
    num_train_epochs (int): Default is: ``1``. Total number of training epochs to perform.
    per_device_train_batch_size (int): Default is: ``2``. Train batch size
    gradient_accumulation_steps (int): Default is: ``1``. Number of updates steps to accumulate before performing a backward/update pass.
    dataloader_drop_last (bool): Default is: ``True``. When it='True', the last incomplete batch is dropped if the dataset size is not divisible by the batch size, meaning DataLoader only returns complete batches.
    warmup_ratio (float): Default is: ``0.1``. Warmup ratio for linear scheduler.
    weight_decay (float): Default is: ``0.01``. Weight decay in AdamW.
    deepspeed (str): Default is: ````. The path of the DeepSpeed configuration file, default to use the pre-made configuration file in the LazyLLM code repository: ``ds_stage0.json``.
    logging_steps (int): Default is: ``1``. Logging frequency according to logging strategy.
    save_steps (int): Default is: ``1000``. Saving frequency.
    overwrite_output_dir (bool): Default is: ``True``. It is used to allow the program to overwrite an existing output directory.
    fp16 (bool): Default is: ``True``.  Whether to use fp16 (mixed) precision instead of 32-bit.
    gradient_checkpointing (bool): Default is: ``True``. Whether enable gradient checkpointing.



Examples:
    >>> from lazyllm import finetune
    >>> finetune.FlagembeddingFinetune('bge-m3', 'path/to/target')
    <lazyllm.llm.finetune type=FlagembeddingFinetune>
    """
    defatult_embed_kw = ArgsDict({
        'train_group_size': 8,
        'query_max_len': 512,
        'passage_max_len': 512,
        'pad_to_multiple_of': 8,
        'query_instruction_for_retrieval': 'Represent this sentence for searching relevant passages: ',
        'query_instruction_format': '{}{}',
        'learning_rate': 1e-5,
        'num_train_epochs': 1,
        'per_device_train_batch_size': 2,
        'gradient_accumulation_steps': 1,
        'dataloader_drop_last': True,
        'warmup_ratio': 0.1,
        'weight_decay': 0.01,
        'deepspeed': '',
        'logging_steps': 1,
        'save_steps': 1000,
        'temperature': 0.02,
        'sentence_pooling_method': 'cls',
        'normalize_embeddings': True,
        'kd_loss_type': 'kl_div',
        'overwrite_output_dir': True,
        'fp16': True,
        'gradient_checkpointing': True,
        'negatives_cross_device': True
    })
    defatult_rerank_kw = ArgsDict({
        'train_group_size': 8,
        'query_max_len': 256,
        'passage_max_len': 256,
        'pad_to_multiple_of': 8,
        'learning_rate': 6e-5,
        'num_train_epochs': 1,
        'per_device_train_batch_size': 2,
        'gradient_accumulation_steps': 1,
        'dataloader_drop_last': True,
        'warmup_ratio': 0.1,
        'weight_decay': 0.01,
        'deepspeed': '',
        'logging_steps': 1,
        'save_steps': 1000,
        'overwrite_output_dir': True,
        'fp16': True,
        'gradient_checkpointing': True
    })
    store_true_embed_kw = {'overwrite_output_dir', 'fp16', 'gradient_checkpointing', 'negatives_cross_device'}
    store_true_rerank_kw = {'overwrite_output_dir', 'fp16', 'gradient_checkpointing'}

    def __init__(
        self,
        base_model,
        target_path,
        launcher=launchers.remote(ngpus=1, sync=True),  # noqa B008
        **kw
    ):
        model_type = ModelManager.get_model_type(base_model.split('/')[-1])
        if model_type not in ('embed', 'reranker'):
            raise RuntimeError(f'Not supported {model_type} type to finetune.')
        if not os.path.exists(base_model):
            defatult_path = os.path.join(lazyllm.config['model_path'], base_model)
            if os.path.exists(defatult_path):
                base_model = defatult_path
        save_path = os.path.join(lazyllm.config['train_target_root'], target_path)
        target_path = os.path.join(save_path, model_type)
        os.system(f'mkdir -p {target_path}')
        super().__init__(
            base_model,
            target_path,
            launcher=launcher,
        )
        if model_type == 'reranker':
            self.kw = copy.deepcopy(self.defatult_rerank_kw)
            self.store_true_kw = copy.deepcopy(self.store_true_rerank_kw)
            self.module_run_path = 'FlagEmbedding.finetune.reranker.encoder_only.base'
        else:
            self.kw = copy.deepcopy(self.defatult_embed_kw)
            self.store_true_kw = copy.deepcopy(self.store_true_embed_kw)
            self.module_run_path = 'FlagEmbedding.finetune.embedder.encoder_only.base'
        self.kw.check_and_update(kw)
        if not self.kw['deepspeed']:
            folder_path = os.path.dirname(os.path.abspath(__file__))
            deepspeed_config_path = os.path.join(folder_path, 'flag_embedding', 'ds_stage0.json')
            self.kw['deepspeed'] = deepspeed_config_path
        self.nproc_per_node = launcher.ngpus

    def cmd(self, trainset, valset=None) -> str:
        thirdparty.check_packages(['flagembedding'])
        self.kw['train_data'] = trainset

        formatted_date = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
        self.log_file_path = f'{self.target_path}/train_log_{formatted_date}_{random.randint(1000, 9999)}.log'
        cache_path = os.path.join(os.path.expanduser('~'), '.lazyllm', 'fintune', 'embeding')
        cache_model_path = os.path.join(cache_path, "model")
        cache_data_path = os.path.join(cache_path, "data")
        os.system(f'mkdir -p {cache_model_path} {cache_data_path}')

        cmds = (f'export WANDB_MODE=disabled && torchrun --nproc_per_node {self.nproc_per_node} '
                f'-m {self.module_run_path} '
                f'--model_name_or_path {self.base_model} '
                f'--output_dir {self.target_path} '
                f'--cache_dir {cache_model_path} '
                f'--cache_path {cache_data_path} '
            )
        for key in self.store_true_kw:
            cmds += f'--{key} ' if self.kw.pop(key) else ''
        cmds += self.kw.parse_kwargs()
        cmds += f' 2>&1 | tee {self.log_file_path}'
        return cmds

lazyllm.components.auto.AutoFinetune

Bases: LazyLLMFinetuneBase

This class is a subclass of LazyLLMFinetuneBase and can automatically select the appropriate fine-tuning framework and parameters based on the input arguments to fine-tune large language models.

Specifically, based on the input model parameters of base_model, ctx_len, batch_size, lora_r, the type and number of GPUs in launcher, this class can automatically select the appropriate fine-tuning framework (such as: AlpacaloraFinetune or CollieFinetune) and the required parameters.

Parameters:

  • base_model (str) –

    The base model used for fine-tuning. It is required to be the path of the base model.

  • source (config[model_source]) –

    Specifies the model download source. This can be configured by setting the environment variable LAZYLLM_MODEL_SOURCE.

  • target_path (str) –

    The path where the LoRA weights of the fine-tuned model are saved.

  • merge_path (str) –

    The path where the model merges the LoRA weights, default to None. If not specified, "lazyllm_lora" and "lazyllm_merge" directories will be created under target_path as target_path and merge_path respectively.

  • ctx_len (int) –

    The maximum token length for input to the fine-tuned model, default to 1024.

  • batch_size (int) –

    Batch size, default to 32.

  • lora_r (int) –

    LoRA rank, default to 8; this value determines the amount of parameters added, the smaller the value, the fewer the parameters.

  • launcher (launcher, default: remote() ) –

    The launcher for fine-tuning, default to launchers.remote(ngpus=1).

  • kw –

    Keyword arguments, used to update the default training parameters. Note that additional keyword arguments cannot be arbitrarily specified, as they depend on the framework inferred by LazyLLM, so it is recommended to set them with caution.

Examples:

>>> from lazyllm import finetune
>>> finetune.auto("internlm2-chat-7b", 'path/to/target')
<lazyllm.llm.finetune type=AlpacaloraFinetune>
Source code in lazyllm/components/auto/autofinetune.py 
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
class AutoFinetune(LazyLLMFinetuneBase):
    """This class is a subclass of ``LazyLLMFinetuneBase`` and can automatically select the appropriate fine-tuning framework and parameters based on the input arguments to fine-tune large language models.

Specifically, based on the input model parameters of ``base_model``, ``ctx_len``, ``batch_size``, ``lora_r``, the type and number of GPUs in ``launcher``, this class can automatically select the appropriate fine-tuning framework (such as: ``AlpacaloraFinetune`` or ``CollieFinetune``) and the required parameters.

Args:
    base_model (str): The base model used for fine-tuning. It is required to be the path of the base model.
    source (lazyllm.config['model_source']): Specifies the model download source. This can be configured by setting the environment variable ``LAZYLLM_MODEL_SOURCE``.
    target_path (str): The path where the LoRA weights of the fine-tuned model are saved.
    merge_path (str): The path where the model merges the LoRA weights, default to ``None``. If not specified, "lazyllm_lora" and "lazyllm_merge" directories will be created under ``target_path`` as ``target_path`` and ``merge_path`` respectively.
    ctx_len (int): The maximum token length for input to the fine-tuned model, default to ``1024``.
    batch_size (int): Batch size, default to ``32``.
    lora_r (int): LoRA rank, default to ``8``; this value determines the amount of parameters added, the smaller the value, the fewer the parameters.
    launcher (lazyllm.launcher): The launcher for fine-tuning, default to ``launchers.remote(ngpus=1)``.
    kw: Keyword arguments, used to update the default training parameters. Note that additional keyword arguments cannot be arbitrarily specified, as they depend on the framework inferred by LazyLLM, so it is recommended to set them with caution.



Examples:
    >>> from lazyllm import finetune
    >>> finetune.auto("internlm2-chat-7b", 'path/to/target')
    <lazyllm.llm.finetune type=AlpacaloraFinetune>
    """
    def __new__(cls, base_model, target_path, source=lazyllm.config['model_source'], merge_path=None, ctx_len=1024,
                batch_size=32, lora_r=8, launcher=launchers.remote(ngpus=1), **kw):  # noqa B008
        base_model = ModelManager(source).download(base_model) or ''
        model_name = get_model_name(base_model)
        model_type = ModelManager.get_model_type(model_name)
        if model_type in ['embed', 'tts', 'vlm', 'stt', 'sd']:
            raise RuntimeError(f'Fine-tuning of the {model_type} model is not currently supported.')
        map_name, _ = model_map(model_name)
        base_name = model_name.split('-')[0].split('_')[0].lower()
        candidates = get_configer().query_finetune(lazyllm.config['gpu_type'], launcher.ngpus,
                                                   map_name, ctx_len, batch_size, lora_r)
        configs = get_configs(base_name)

        for k, v in configs.items():
            if k not in kw: kw[k] = v

        for c in candidates:
            if check_requirements(requirements[c.framework.lower()]):
                finetune_cls = getattr(finetune, c.framework.lower())
                for key, value in finetune_cls.auto_map.items():
                    if value:
                        kw[value] = getattr(c, key)
                if finetune_cls.__name__ == 'LlamafactoryFinetune':
                    return finetune_cls(base_model, target_path, lora_r=lora_r, launcher=launcher, **kw)
                return finetune_cls(base_model, target_path, merge_path, cp_files='tokeniz*',
                                    batch_size=batch_size, lora_r=lora_r, launcher=launcher, **kw)
        raise RuntimeError(f'No valid framework found, candidates are {[c.framework.lower() for c in candidates]}')

lazyllm.components.finetune.base.DummyFinetune

Bases: LazyLLMFinetuneBase

DummyFinetune is a subclass of [LazyLLMFinetuneBase][lazyllm.components.LazyLLMFinetuneBase] that serves as a placeholder implementation for fine-tuning. The class is primarily used for demonstration or testing purposes, as it does not perform any actual fine-tuning logic. Args: base_model: A string specifying the base model name. Defaults to 'base'. target_path: A string specifying the target path for fine-tuning outputs. Defaults to 'target'. launcher: A launcher instance for executing commands. Defaults to [launchers.remote()][lazyllm.launchers.remote]. kw: Additional keyword arguments that are stored for later use. cmd(self, *args, **kw) -> str Generates a dummy command string for fine-tuning. This method is for testing purposes only. Args: *args: Positional arguments to be included in the command. kw: Keyword arguments to be included in the command. Returns: A string representing a dummy command. The string includes the initial arguments passed during initialization.

Examples:

>>> from lazyllm.components import DummyFinetune
>>> from lazyllm import launchers
>>> # 创建一个 DummyFinetune 实例
>>> finetuner = DummyFinetune(base_model='example-base', target_path='example-target', launcher=launchers.local(), custom_arg='custom_value')
>>> # 调用 cmd 方法生成占位命令
>>> command = finetuner.cmd('--example-arg', key='value')
>>> print(command)
... echo 'dummy finetune!, and init-args is {'custom_arg': 'custom_value'}'
Source code in lazyllm/components/finetune/base.py 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
class DummyFinetune(LazyLLMFinetuneBase):
    """DummyFinetune is a subclass of [LazyLLMFinetuneBase][lazyllm.components.LazyLLMFinetuneBase] that serves as a placeholder implementation for fine-tuning.
The class is primarily used for demonstration or testing purposes, as it does not perform any actual fine-tuning logic.
Args:
    base_model: A string specifying the base model name. Defaults to 'base'.
    target_path: A string specifying the target path for fine-tuning outputs. Defaults to 'target'.
    launcher: A launcher instance for executing commands. Defaults to [launchers.remote()][lazyllm.launchers.remote].
    **kw: Additional keyword arguments that are stored for later use.
     `cmd(self, *args, **kw) -> str`
Generates a dummy command string for fine-tuning. This method is for testing purposes only.
Args:
    *args: Positional arguments to be included in the command.
    **kw: Keyword arguments to be included in the command.
Returns:
    A string representing a dummy command. The string includes the initial arguments passed during initialization.


Examples:
    >>> from lazyllm.components import DummyFinetune
    >>> from lazyllm import launchers
    >>> # 创建一个 DummyFinetune 实例
    >>> finetuner = DummyFinetune(base_model='example-base', target_path='example-target', launcher=launchers.local(), custom_arg='custom_value')
    >>> # 调用 cmd 方法生成占位命令
    >>> command = finetuner.cmd('--example-arg', key='value')
    >>> print(command)
    ... echo 'dummy finetune!, and init-args is {'custom_arg': 'custom_value'}'
    """
    def __init__(self, base_model='base', target_path='target', *, launcher=launchers.remote(), **kw):  # noqa B008
        super().__init__(base_model, target_path, launcher=launchers.empty)
        self.kw = kw

    def cmd(self, *args, **kw) -> str:
        """The `cmd` method generates a dummy command string for fine-tuning. This method is primarily for testing or demonstration purposes.
Args:
    *args: Positional arguments to be included in the command (not used in this implementation).
    **kw: Keyword arguments to be included in the command (not used in this implementation).
Returns:
    A string representing a dummy command. The string includes the initial arguments (`**kw`) passed during the instance initialization, which are stored in `self.kw`.
Example:
    If the class is initialized with `custom_arg='value'`, calling the `cmd` method will return:
    `"echo 'dummy finetune!, and init-args is {'custom_arg': 'value'}'"`


Examples:
    >>> from lazyllm.components import DummyFinetune
    >>> from lazyllm import launchers
    >>> # 创建一个 DummyFinetune 实例，并传递初始化参数
    >>> finetuner = DummyFinetune(base_model='example-base', target_path='example-target', launcher=launchers.local(), custom_arg='value')
    >>> # 调用 cmd 方法生成占位命令
    >>> command = finetuner.cmd()
    >>> # 打印生成的占位命令
    >>> print(command)
    ... echo 'dummy finetune!, and init-args is {'custom_arg': 'value'}'
    """
        return f'echo \'dummy finetune!, and init-args is {self.kw}\''

cmd(*args, **kw)

The cmd method generates a dummy command string for fine-tuning. This method is primarily for testing or demonstration purposes. Args: args: Positional arguments to be included in the command (not used in this implementation). *kw: Keyword arguments to be included in the command (not used in this implementation). Returns: A string representing a dummy command. The string includes the initial arguments (**kw) passed during the instance initialization, which are stored in self.kw. Example: If the class is initialized with custom_arg='value', calling the cmd method will return: "echo 'dummy finetune!, and init-args is {'custom_arg': 'value'}'"

Examples:

>>> from lazyllm.components import DummyFinetune
>>> from lazyllm import launchers
>>> # 创建一个 DummyFinetune 实例，并传递初始化参数
>>> finetuner = DummyFinetune(base_model='example-base', target_path='example-target', launcher=launchers.local(), custom_arg='value')
>>> # 调用 cmd 方法生成占位命令
>>> command = finetuner.cmd()
>>> # 打印生成的占位命令
>>> print(command)
... echo 'dummy finetune!, and init-args is {'custom_arg': 'value'}'
Source code in lazyllm/components/finetune/base.py 
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
    def cmd(self, *args, **kw) -> str:
        """The `cmd` method generates a dummy command string for fine-tuning. This method is primarily for testing or demonstration purposes.
Args:
    *args: Positional arguments to be included in the command (not used in this implementation).
    **kw: Keyword arguments to be included in the command (not used in this implementation).
Returns:
    A string representing a dummy command. The string includes the initial arguments (`**kw`) passed during the instance initialization, which are stored in `self.kw`.
Example:
    If the class is initialized with `custom_arg='value'`, calling the `cmd` method will return:
    `"echo 'dummy finetune!, and init-args is {'custom_arg': 'value'}'"`


Examples:
    >>> from lazyllm.components import DummyFinetune
    >>> from lazyllm import launchers
    >>> # 创建一个 DummyFinetune 实例，并传递初始化参数
    >>> finetuner = DummyFinetune(base_model='example-base', target_path='example-target', launcher=launchers.local(), custom_arg='value')
    >>> # 调用 cmd 方法生成占位命令
    >>> command = finetuner.cmd()
    >>> # 打印生成的占位命令
    >>> print(command)
    ... echo 'dummy finetune!, and init-args is {'custom_arg': 'value'}'
    """
        return f'echo \'dummy finetune!, and init-args is {self.kw}\''

Deploy

lazyllm.components.deploy.Lightllm

Bases: LazyLLMDeployBase

This class is a subclass of LazyLLMDeployBase, based on the inference capabilities provided by the LightLLM framework, used for inference with large language models.

Parameters:

  • trust_remote_code (bool, default: True ) –

    Whether to allow loading of model code from remote servers, default is True.

  • launcher (launcher, default: remote(ngpus=1) ) –

    The launcher for fine-tuning, default is launchers.remote(ngpus=1).

  • stream (bool) –

    Whether the response is streaming, default is False.

  • kw –

    Keyword arguments used to update default training parameters. Note that not any additional keyword arguments can be specified here.

The keyword arguments and their default values for this class are as follows:

Other Parameters:

  • tp (int) –

    Tensor parallelism parameter, default is 1.

  • max_total_token_num (int) –

    Maximum total token number, default is 64000.

  • eos_id (int) –

    End-of-sentence ID, default is 2.

  • port (int) –

    Service port number, default is None, in which case LazyLLM will automatically generate a random port number.

  • host (str) –

    Service IP address, default is 0.0.0.0.

  • nccl_port (int) –

    NCCL port, default is None, in which case LazyLLM will automatically generate a random port number.

  • tokenizer_mode (str) –

    Tokenizer loading mode, default is auto.

  • running_max_req_size (int) –

    Maximum number of parallel requests for the inference engine, default is 256.

  • data_type (str) –

    Data type for model weights, default is float16.

  • max_req_total_len (int) –

    Maximum total length for requests, default is 64000.

  • max_req_input_len (int) –

    Maximum input length, default is 4096.

  • long_truncation_mode (str) –

    Truncation mode for long texts, default is head.

Examples:

>>> from lazyllm import deploy
>>> infer = deploy.lightllm()
Source code in lazyllm/components/deploy/lightllm.py 
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
class Lightllm(LazyLLMDeployBase):
    """This class is a subclass of ``LazyLLMDeployBase``, based on the inference capabilities provided by the [LightLLM](https://github.com/ModelTC/lightllm) framework, used for inference with large language models.

Args:
    trust_remote_code (bool): Whether to allow loading of model code from remote servers, default is ``True``.
    launcher (lazyllm.launcher): The launcher for fine-tuning, default is ``launchers.remote(ngpus=1)``.
    stream (bool): Whether the response is streaming, default is ``False``.
    kw: Keyword arguments used to update default training parameters. Note that not any additional keyword arguments can be specified here.

The keyword arguments and their default values for this class are as follows:

Keyword Args: 
    tp (int): Tensor parallelism parameter, default is ``1``.
    max_total_token_num (int): Maximum total token number, default is ``64000``.
    eos_id (int): End-of-sentence ID, default is ``2``.
    port (int): Service port number, default is ``None``, in which case LazyLLM will automatically generate a random port number.
    host (str): Service IP address, default is ``0.0.0.0``.
    nccl_port (int): NCCL port, default is ``None``, in which case LazyLLM will automatically generate a random port number.
    tokenizer_mode (str): Tokenizer loading mode, default is ``auto``.
    running_max_req_size (int): Maximum number of parallel requests for the inference engine, default is ``256``.
    data_type (str): Data type for model weights, default is ``float16``.
    max_req_total_len (int): Maximum total length for requests, default is ``64000``.
    max_req_input_len (int): Maximum input length, default is ``4096``.
    long_truncation_mode (str): Truncation mode for long texts, default is ``head``.



Examples:
    >>> from lazyllm import deploy
    >>> infer = deploy.lightllm()
    """
    keys_name_handle = {
        'inputs': 'inputs',
        'stop': 'stop_sequences'
    }
    default_headers = {'Content-Type': 'application/json'}
    message_format = {
        'inputs': 'Who are you ?',
        'parameters': {
            'do_sample': False,
            "presence_penalty": 0.0,
            "frequency_penalty": 0.0,
            "repetition_penalty": 1.0,
            'temperature': 1.0,
            "top_p": 1,
            "top_k": -1,  # -1 is for all
            "ignore_eos": False,
            'max_new_tokens': 8192,
            "stop_sequences": None,
        }
    }
    auto_map = {}
    stream_url_suffix = '_stream'
    stream_parse_parameters = {"delimiter": b"\n\n"}

    def __init__(self, trust_remote_code=True, launcher=launchers.remote(ngpus=1), log_path=None, **kw):  # noqa B008
        super().__init__(launcher=launcher)
        self.kw = ArgsDict({
            'tp': 1,
            'max_total_token_num': 64000,
            'eos_id': 2,
            'port': None,
            'host': '0.0.0.0',
            'nccl_port': None,
            'tokenizer_mode': 'auto',
            "running_max_req_size": 256,
            "data_type": 'float16',
            "max_req_total_len": 64000,
            "max_req_input_len": 4096,
            "long_truncation_mode": "head",
        })
        self.trust_remote_code = trust_remote_code
        self.kw.check_and_update(kw)
        self.random_port = False if 'port' in kw and kw['port'] else True
        self.random_nccl_port = False if 'nccl_port' in kw and kw['nccl_port'] else True
        self.temp_folder = make_log_dir(log_path, 'lightllm') if log_path else None

    def cmd(self, finetuned_model=None, base_model=None):
        """This method generates the command to start the LightLLM service.

Args:
    finetuned_model (str): Path to the fine-tuned model.
    base_model (str): Path to the base model, used when finetuned_model is invalid.

Returns:
    LazyLLMCMD: A LazyLLMCMD object containing the startup command.
"""
        if not os.path.exists(finetuned_model) or \
            not any(filename.endswith('.bin') or filename.endswith('.safetensors')
                    for filename in os.listdir(finetuned_model)):
            if not finetuned_model:
                LOG.warning(f"Note! That finetuned_model({finetuned_model}) is an invalid path, "
                            f"base_model({base_model}) will be used")
            finetuned_model = base_model

        def impl():
            if self.random_port:
                self.kw['port'] = random.randint(30000, 40000)
            if self.random_nccl_port:
                self.kw['nccl_port'] = random.randint(20000, 30000)
            cmd = f'python -m lightllm.server.api_server --model_dir {finetuned_model} '
            cmd += self.kw.parse_kwargs()
            if self.trust_remote_code:
                cmd += ' --trust_remote_code '
            if self.temp_folder: cmd += f' 2>&1 | tee {get_log_path(self.temp_folder)}'
            return cmd

        return LazyLLMCMD(cmd=impl, return_value=self.geturl, checkf=verify_fastapi_func)

    def geturl(self, job=None):
        """Get the URL address of the LightLLM service.

Args:
    job (optional): Job object, defaults to None, in which case self.job is used.

Returns:
    str: The service URL address in the format "http://{ip}:{port}/generate".
"""
        if job is None:
            job = self.job
        if lazyllm.config['mode'] == lazyllm.Mode.Display:
            return 'http://{ip}:{port}/generate'
        else:
            return f'http://{job.get_jobip()}:{self.kw["port"]}/generate'

    @staticmethod
    def extract_result(x, inputs):
        try:
            if x.startswith("data:"): return json.loads(x[len("data:"):])['token']['text']
            else: return json.loads(x)['generated_text'][0]
        except Exception as e:
            LOG.warning(f'JSONDecodeError on load {x}')
            raise e

cmd(finetuned_model=None, base_model=None)

This method generates the command to start the LightLLM service.

Parameters:

  • finetuned_model (str, default: None ) –

    Path to the fine-tuned model.

  • base_model (str, default: None ) –

    Path to the base model, used when finetuned_model is invalid.

Returns:

  • LazyLLMCMD –

    A LazyLLMCMD object containing the startup command.

Source code in lazyllm/components/deploy/lightllm.py 
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
    def cmd(self, finetuned_model=None, base_model=None):
        """This method generates the command to start the LightLLM service.

Args:
    finetuned_model (str): Path to the fine-tuned model.
    base_model (str): Path to the base model, used when finetuned_model is invalid.

Returns:
    LazyLLMCMD: A LazyLLMCMD object containing the startup command.
"""
        if not os.path.exists(finetuned_model) or \
            not any(filename.endswith('.bin') or filename.endswith('.safetensors')
                    for filename in os.listdir(finetuned_model)):
            if not finetuned_model:
                LOG.warning(f"Note! That finetuned_model({finetuned_model}) is an invalid path, "
                            f"base_model({base_model}) will be used")
            finetuned_model = base_model

        def impl():
            if self.random_port:
                self.kw['port'] = random.randint(30000, 40000)
            if self.random_nccl_port:
                self.kw['nccl_port'] = random.randint(20000, 30000)
            cmd = f'python -m lightllm.server.api_server --model_dir {finetuned_model} '
            cmd += self.kw.parse_kwargs()
            if self.trust_remote_code:
                cmd += ' --trust_remote_code '
            if self.temp_folder: cmd += f' 2>&1 | tee {get_log_path(self.temp_folder)}'
            return cmd

        return LazyLLMCMD(cmd=impl, return_value=self.geturl, checkf=verify_fastapi_func)

geturl(job=None)

Get the URL address of the LightLLM service.

Parameters:

  • job (optional, default: None ) –

    Job object, defaults to None, in which case self.job is used.

Returns:

  • str –

    The service URL address in the format "http://{ip}:{port}/generate".

Source code in lazyllm/components/deploy/lightllm.py 
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
    def geturl(self, job=None):
        """Get the URL address of the LightLLM service.

Args:
    job (optional): Job object, defaults to None, in which case self.job is used.

Returns:
    str: The service URL address in the format "http://{ip}:{port}/generate".
"""
        if job is None:
            job = self.job
        if lazyllm.config['mode'] == lazyllm.Mode.Display:
            return 'http://{ip}:{port}/generate'
        else:
            return f'http://{job.get_jobip()}:{self.kw["port"]}/generate'

extract_result(x, inputs) staticmethod

Source code in lazyllm/components/deploy/lightllm.py 
136
137
138
139
140
141
142
143
@staticmethod
def extract_result(x, inputs):
    try:
        if x.startswith("data:"): return json.loads(x[len("data:"):])['token']['text']
        else: return json.loads(x)['generated_text'][0]
    except Exception as e:
        LOG.warning(f'JSONDecodeError on load {x}')
        raise e

lazyllm.components.deploy.Vllm

Bases: LazyLLMDeployBase

Source code in lazyllm/components/deploy/vllm.py 
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
class Vllm(LazyLLMDeployBase, metaclass=_VllmStreamParseParametersMeta):
    # keys_name_handle/default_headers/message_format will lose efficacy when openai_api is True
    keys_name_handle = {'inputs': 'prompt', 'stop': 'stop'}
    default_headers = {'Content-Type': 'application/json'}
    message_format = {
        'prompt': 'Who are you ?',
        'stream': False,
        'stop': ['<|im_end|>', '<|im_start|>', '</s>', '<|assistant|>', '<|user|>', '<|system|>', '<eos>'],
        'skip_special_tokens': False,
        'temperature': 0.6,
        'top_p': 0.8,
        'max_tokens': 4096
    }
    auto_map = {'tp': 'tensor-parallel-size'}
    optional_keys = set(["max-model-len"])

    # TODO(wangzhihong): change default value for `openai_api` argument to True
    def __init__(self, trust_remote_code: bool = True,
                 launcher: LazyLLMLaunchersBase = launchers.remote(ngpus=1),  # noqa B008
                 log_path: str = None, openai_api: bool = False, **kw):
        self.launcher_list, launcher = reallocate_launcher(launcher)
        super().__init__(launcher=launcher)
        self.kw = ArgsDict({
            'dtype': 'auto',
            'kv-cache-dtype': 'auto',
            'tokenizer-mode': 'auto',
            'device': 'auto',
            'block-size': 16,
            'tensor-parallel-size': 1,
            'seed': 0,
            'port': 'auto',
            'host': '0.0.0.0',
            'max-num-seqs': 256,
            'pipeline-parallel-size': 1,
            'max-num-batched-tokens': 64000,
        })
        self._vllm_cmd = 'vllm.entrypoints.openai.api_server' if openai_api else 'vllm.entrypoints.api_server'
        self.trust_remote_code = trust_remote_code
        self.kw.update(**{key: kw[key] for key in self.optional_keys if key in kw})
        self.kw.check_and_update(kw)
        self.random_port = False if 'port' in kw and kw['port'] and kw['port'] != 'auto' else True
        self.temp_folder = make_log_dir(log_path, 'vllm') if log_path else None
        if self.launcher_list:
            ray_launcher = [Distributed(launcher=launcher) for launcher in self.launcher_list]
            parall_launcher = [lazyllm.pipeline(sleep_moment, launcher) for launcher in ray_launcher[1:]]
            self._prepare_deploy = lazyllm.pipeline(
                ray_launcher[0], post_action=(lazyllm.parallel(*parall_launcher) if len(parall_launcher) else None))

    def cmd(self, finetuned_model=None, base_model=None, master_ip=None):
        if not os.path.exists(finetuned_model) or \
            not any(filename.endswith('.bin') or filename.endswith('.safetensors')
                    for filename in os.listdir(finetuned_model)):
            if not finetuned_model:
                LOG.warning(f"Note! That finetuned_model({finetuned_model}) is an invalid path, "
                            f"base_model({base_model}) will be used")
            finetuned_model = base_model

        def impl():
            if self.random_port:
                self.kw['port'] = random.randint(30000, 40000)

            cmd = ''
            if self.launcher_list:
                cmd += f"ray start --address='{master_ip}' && "
            cmd += f'{sys.executable} -m {self._vllm_cmd} --model {finetuned_model} '
            cmd += self.kw.parse_kwargs()
            if self.trust_remote_code:
                cmd += ' --trust-remote-code '
            if self.temp_folder: cmd += f' 2>&1 | tee {get_log_path(self.temp_folder)}'
            return cmd

        return LazyLLMCMD(cmd=impl, return_value=self.geturl, checkf=verify_fastapi_func)

    def geturl(self, job=None):
        if job is None:
            job = self.job
        if lazyllm.config['mode'] == lazyllm.Mode.Display:
            return 'http://{ip}:{port}/generate'
        else:
            return f'http://{job.get_jobip()}:{self.kw["port"]}/generate'

    @staticmethod
    def extract_result(x, inputs):
        return json.loads(x)['text'][0]

lazyllm.components.deploy.LMDeploy

Bases: LazyLLMDeployBase

This class is a subclass of LazyLLMDeployBase, leveraging the inference capabilities provided by the LMDeploy framework for inference on large language models.

Parameters:

  • launcher (launcher, default: remote(ngpus=1) ) –

    The launcher for fine-tuning, defaults to launchers.remote(ngpus=1).

  • stream (bool) –

    Whether to enable streaming response, defaults to False.

  • trust_remote_code (bool, default: True ) –

    Whether to trust remote code, defaults to True.

  • log_path (str, default: None ) –

    Path for log file, defaults to None.

  • kw –

    Keyword arguments for updating default training parameters. Note that no additional keyword arguments beyond those listed below can be passed.

Other Parameters:

  • tp (int) –

    Tensor parallelism parameter, defaults to 1.

  • server-name (str) –

    The IP address of the service, defaults to 0.0.0.0.

  • server-port (int) –

    The port number of the service, defaults to None. In this case, LazyLLM will automatically generate a random port number.

  • max-batch-size (int) –

    Maximum batch size, defaults to 128.

  • chat-template (str) –

    Path to chat template file, defaults to None. If the model is not a vision-language model and no template is specified, a default template will be used.

  • eager-mode (bool) –

    Whether to enable eager mode, controlled by environment variable LMDEPLOY_EAGER_MODE, defaults to False.

Examples:

>>> # Basic use:
>>> from lazyllm import deploy
>>> infer = deploy.LMDeploy()
>>>
>>> # MultiModal:
>>> import lazyllm
>>> from lazyllm import deploy, globals
>>> from lazyllm.components.formatter import encode_query_with_filepaths
>>> chat = lazyllm.TrainableModule('Mini-InternVL-Chat-2B-V1-5').deploy_method(deploy.LMDeploy)
>>> chat.update_server()
>>> inputs = encode_query_with_filepaths('What is it?', ['path/to/image'])
>>> res = chat(inputs)
Source code in lazyllm/components/deploy/lmdeploy.py 
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
class LMDeploy(LazyLLMDeployBase):
    """This class is a subclass of ``LazyLLMDeployBase``, leveraging the inference capabilities provided by the [LMDeploy](https://github.com/InternLM/lmdeploy) framework for inference on large language models.

Args:
    launcher (lazyllm.launcher): The launcher for fine-tuning, defaults to ``launchers.remote(ngpus=1)``.
    stream (bool): Whether to enable streaming response, defaults to ``False``.
    trust_remote_code (bool): Whether to trust remote code, defaults to ``True``.
    log_path (str): Path for log file, defaults to ``None``.
    kw: Keyword arguments for updating default training parameters. Note that no additional keyword arguments beyond those listed below can be passed.

Keyword Args: 
    tp (int): Tensor parallelism parameter, defaults to ``1``.
    server-name (str): The IP address of the service, defaults to ``0.0.0.0``.
    server-port (int): The port number of the service, defaults to ``None``. In this case, LazyLLM will automatically generate a random port number.
    max-batch-size (int): Maximum batch size, defaults to ``128``.
    chat-template (str): Path to chat template file, defaults to ``None``. If the model is not a vision-language model and no template is specified, a default template will be used.
    eager-mode (bool): Whether to enable eager mode, controlled by environment variable ``LMDEPLOY_EAGER_MODE``, defaults to ``False``.



Examples:
    >>> # Basic use:
    >>> from lazyllm import deploy
    >>> infer = deploy.LMDeploy()
    >>>
    >>> # MultiModal:
    >>> import lazyllm
    >>> from lazyllm import deploy, globals
    >>> from lazyllm.components.formatter import encode_query_with_filepaths
    >>> chat = lazyllm.TrainableModule('Mini-InternVL-Chat-2B-V1-5').deploy_method(deploy.LMDeploy)
    >>> chat.update_server()
    >>> inputs = encode_query_with_filepaths('What is it?', ['path/to/image'])
    >>> res = chat(inputs)
    """
    keys_name_handle = {
        'inputs': 'prompt',
        'stop': 'stop',
        'image': 'image_url',
    }
    default_headers = {'Content-Type': 'application/json'}
    message_format = {
        'prompt': 'Who are you ?',
        "image_url": None,
        "session_id": -1,
        "interactive_mode": False,
        "stream": False,
        "stop": None,
        "request_output_len": None,
        "top_p": 0.8,
        "top_k": 40,
        "temperature": 0.8,
        "repetition_penalty": 1,
        "ignore_eos": False,
        "skip_special_tokens": True,
        "cancel": False,
        "adapter_name": None
    }
    auto_map = {
        'port': 'server-port',
        'host': 'server-name',
        'max_batch_size': 'max-batch-size',
        'chat_template': 'chat-template',
    }
    stream_parse_parameters = {"delimiter": b"\n"}

    def __init__(self, launcher=launchers.remote(ngpus=1), trust_remote_code=True, log_path=None, **kw):  # noqa B008
        super().__init__(launcher=launcher)
        self.kw = ArgsDict({
            'server-name': '0.0.0.0',
            'server-port': None,
            'tp': 1,
            "max-batch-size": 128,
            "chat-template": None,
        })
        self.kw.check_and_update(kw)
        self._trust_remote_code = trust_remote_code
        self.random_port = False if 'server-port' in kw and kw['server-port'] else True
        self.temp_folder = make_log_dir(log_path, 'lmdeploy') if log_path else None

    def cmd(self, finetuned_model=None, base_model=None):
        """This method generates the command to start the LMDeploy service.

Args:
    finetuned_model (str): Path to the fine-tuned model.
    base_model (str): Path to the base model, used when finetuned_model is invalid.

Returns:
    LazyLLMCMD: A LazyLLMCMD object containing the startup command.
"""
        if not os.path.exists(finetuned_model) or \
            not any(filename.endswith('.bin') or filename.endswith('.safetensors')
                    for filename in os.listdir(finetuned_model)):
            if not finetuned_model:
                LOG.warning(f"Note! That finetuned_model({finetuned_model}) is an invalid path, "
                            f"base_model({base_model}) will be used")
            finetuned_model = base_model

        model_type = ModelManager.get_model_type(base_model or finetuned_model)
        if model_type == 'vlm':
            self.kw.pop("chat-template")
        else:
            if not self.kw["chat-template"] and 'vl' not in finetuned_model and 'lava' not in finetuned_model:
                self.kw["chat-template"] = os.path.join(os.path.dirname(os.path.abspath(__file__)),
                                                        'lmdeploy', 'chat_template.json')
            else:
                self.kw.pop("chat-template")

        def impl():
            if self.random_port:
                self.kw['server-port'] = random.randint(30000, 40000)
            cmd = f"lmdeploy serve api_server {finetuned_model} "

            if importlib.util.find_spec("torch_npu") is not None: cmd += '--device ascend '
            if config['lmdeploy_eager_mode']: cmd += '--eager-mode '
            cmd += self.kw.parse_kwargs()
            if self.temp_folder: cmd += f' 2>&1 | tee {get_log_path(self.temp_folder)}'
            return cmd

        return LazyLLMCMD(cmd=impl, return_value=self.geturl, checkf=verify_fastapi_func)

    def geturl(self, job=None):
        """Get the URL address of the LMDeploy service.

Args:
    job (optional): Job object, defaults to None, in which case self.job is used.

Returns:
    str: The service URL address in the format "http://{ip}:{port}/v1/chat/interactive".
"""
        if job is None:
            job = self.job
        if lazyllm.config['mode'] == lazyllm.Mode.Display:
            return 'http://{ip}:{port}/v1/chat/interactive'
        else:
            return f'http://{job.get_jobip()}:{self.kw["server-port"]}/v1/chat/interactive'

    @staticmethod
    def extract_result(x, inputs):
        return json.loads(x)['text']

cmd(finetuned_model=None, base_model=None)

This method generates the command to start the LMDeploy service.

Parameters:

  • finetuned_model (str, default: None ) –

    Path to the fine-tuned model.

  • base_model (str, default: None ) –

    Path to the base model, used when finetuned_model is invalid.

Returns:

  • LazyLLMCMD –

    A LazyLLMCMD object containing the startup command.

Source code in lazyllm/components/deploy/lmdeploy.py 
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
    def cmd(self, finetuned_model=None, base_model=None):
        """This method generates the command to start the LMDeploy service.

Args:
    finetuned_model (str): Path to the fine-tuned model.
    base_model (str): Path to the base model, used when finetuned_model is invalid.

Returns:
    LazyLLMCMD: A LazyLLMCMD object containing the startup command.
"""
        if not os.path.exists(finetuned_model) or \
            not any(filename.endswith('.bin') or filename.endswith('.safetensors')
                    for filename in os.listdir(finetuned_model)):
            if not finetuned_model:
                LOG.warning(f"Note! That finetuned_model({finetuned_model}) is an invalid path, "
                            f"base_model({base_model}) will be used")
            finetuned_model = base_model

        model_type = ModelManager.get_model_type(base_model or finetuned_model)
        if model_type == 'vlm':
            self.kw.pop("chat-template")
        else:
            if not self.kw["chat-template"] and 'vl' not in finetuned_model and 'lava' not in finetuned_model:
                self.kw["chat-template"] = os.path.join(os.path.dirname(os.path.abspath(__file__)),
                                                        'lmdeploy', 'chat_template.json')
            else:
                self.kw.pop("chat-template")

        def impl():
            if self.random_port:
                self.kw['server-port'] = random.randint(30000, 40000)
            cmd = f"lmdeploy serve api_server {finetuned_model} "

            if importlib.util.find_spec("torch_npu") is not None: cmd += '--device ascend '
            if config['lmdeploy_eager_mode']: cmd += '--eager-mode '
            cmd += self.kw.parse_kwargs()
            if self.temp_folder: cmd += f' 2>&1 | tee {get_log_path(self.temp_folder)}'
            return cmd

        return LazyLLMCMD(cmd=impl, return_value=self.geturl, checkf=verify_fastapi_func)

geturl(job=None)

Get the URL address of the LMDeploy service.

Parameters:

  • job (optional, default: None ) –

    Job object, defaults to None, in which case self.job is used.

Returns:

  • str –

    The service URL address in the format "http://{ip}:{port}/v1/chat/interactive".

Source code in lazyllm/components/deploy/lmdeploy.py 
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
    def geturl(self, job=None):
        """Get the URL address of the LMDeploy service.

Args:
    job (optional): Job object, defaults to None, in which case self.job is used.

Returns:
    str: The service URL address in the format "http://{ip}:{port}/v1/chat/interactive".
"""
        if job is None:
            job = self.job
        if lazyllm.config['mode'] == lazyllm.Mode.Display:
            return 'http://{ip}:{port}/v1/chat/interactive'
        else:
            return f'http://{job.get_jobip()}:{self.kw["server-port"]}/v1/chat/interactive'

extract_result(x, inputs) staticmethod

Source code in lazyllm/components/deploy/lmdeploy.py 
151
152
153
@staticmethod
def extract_result(x, inputs):
    return json.loads(x)['text']

lazyllm.components.deploy.base.DummyDeploy

Bases: LazyLLMDeployBase, Pipeline

DummyDeploy(launcher=launchers.remote(sync=False), , stream=False, *kw)

A mock deployment class for testing purposes. It extends both LazyLLMDeployBase and flows.Pipeline, simulating a simple pipeline-style deployable service with optional streaming support.

This class is primarily intended for internal testing and demonstration. It receives inputs in the format defined by message_format, and returns a dummy response or a streaming response depending on the stream flag.

Attributes: - keys_name_handle (dict): Mapping of input keys for request formatting. - message_format (dict): Default request template including input and generation parameters.

Parameters: - launcher: Deployment launcher instance, defaulting to launchers.remote(sync=False). - stream (bool): Whether to simulate streaming output. - kw: Additional keyword arguments passed to the superclass.

Methods: - call(*args): Starts the deployment and returns the service URL. - repr(): Returns a string representation of the underlying pipeline.

Source code in lazyllm/components/deploy/base.py 
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
class DummyDeploy(LazyLLMDeployBase, flows.Pipeline):
    """DummyDeploy(launcher=launchers.remote(sync=False), *, stream=False, **kw)

A mock deployment class for testing purposes. It extends both `LazyLLMDeployBase` and `flows.Pipeline`,
simulating a simple pipeline-style deployable service with optional streaming support.

This class is primarily intended for internal testing and demonstration. It receives inputs in the format defined
by `message_format`, and returns a dummy response or a streaming response depending on the `stream` flag.

Attributes:
- keys_name_handle (dict): Mapping of input keys for request formatting.
- message_format (dict): Default request template including input and generation parameters.

Parameters:
- launcher: Deployment launcher instance, defaulting to `launchers.remote(sync=False)`.
- stream (bool): Whether to simulate streaming output.
- kw: Additional keyword arguments passed to the superclass.

Methods:
- __call__(*args): Starts the deployment and returns the service URL.
- __repr__(): Returns a string representation of the underlying pipeline.
"""
    keys_name_handle = {'inputs': 'inputs'}
    message_format = {
        'inputs': '',
        'parameters': {
            'do_sample': False,
            'temperature': 0.1,
        }
    }

    def __init__(self, launcher=launchers.remote(sync=False), *, stream=False, **kw):  # noqa B008
        super().__init__(launcher=launcher)

        def func():

            def impl(x):
                LOG.info(f'input is {x["inputs"]}, parameters is {x["parameters"]}')
                return f'reply for {x["inputs"]}, and parameters is {x["parameters"]}'

            def impl_stream(x):
                for s in ['reply', ' for', f' {x["inputs"]}', ', and',
                          ' parameters', ' is', f' {x["parameters"]}']:
                    yield s
                    time.sleep(0.2)
            return impl_stream if stream else impl
        flows.Pipeline.__init__(self, func,
                                lazyllm.deploy.RelayServer(port=random.randint(30000, 40000), launcher=launcher))

    def __call__(self, *args):
        url = flows.Pipeline.__call__(self)
        LOG.info(f'dummy deploy url is : {url}')
        return url

    def __repr__(self):
        return flows.Pipeline.__repr__(self)

lazyllm.components.auto.AutoDeploy

Bases: LazyLLMDeployBase

This class is a subclass of LazyLLMDeployBase that automatically selects the appropriate inference framework and parameters based on the input arguments for inference with large language models.

Specifically, based on the input base_model parameters, max_token_num, the type and number of GPUs in launcher, this class can automatically select the appropriate inference framework (such as Lightllm or Vllm) and the required parameters.

Parameters:

  • base_model (str) –

    The base model for fine-tuning, which is required to be the name or the path to the base model. Used to provide base model information.

  • source (config[model_source]) –

    Specifies the model download source. This can be configured by setting the environment variable LAZYLLM_MODEL_SOURCE.

  • trust_remote_code (bool) –

    Whether to allow loading of model code from remote servers, default is True.

  • launcher (launcher, default: remote() ) –

    The launcher for fine-tuning, default is launchers.remote(ngpus=1).

  • stream (bool) –

    Whether the response is streaming, default is False.

  • type (str) –

    Type parameter, default is None, which corresponds to the llm type. Additionally, the embed type is also supported.

  • max_token_num (int) –

    The maximum token length for the input fine-tuning model, default is 1024.

  • launcher (launcher, default: remote() ) –

    The launcher for fine-tuning, default is launchers.remote(ngpus=1).

  • kw –

    Keyword arguments used to update default training parameters. Note that whether additional keyword arguments can be specified depends on the framework inferred by LazyLLM, so it is recommended to set them carefully.

Examples:

>>> from lazyllm import deploy
>>> deploy.auto('internlm2-chat-7b')
<lazyllm.llm.deploy type=Lightllm>
Source code in lazyllm/components/auto/autodeploy.py 
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
class AutoDeploy(LazyLLMDeployBase):
    """This class is a subclass of ``LazyLLMDeployBase`` that automatically selects the appropriate inference framework and parameters based on the input arguments for inference with large language models.

Specifically, based on the input ``base_model`` parameters, ``max_token_num``, the type and number of GPUs in ``launcher``, this class can automatically select the appropriate inference framework (such as ``Lightllm`` or ``Vllm``) and the required parameters.

Args:
    base_model (str): The base model for fine-tuning, which is required to be the name or the path to the base model. Used to provide base model information.
    source (lazyllm.config['model_source']): Specifies the model download source. This can be configured by setting the environment variable ``LAZYLLM_MODEL_SOURCE``.
    trust_remote_code (bool): Whether to allow loading of model code from remote servers, default is ``True``.
    launcher (lazyllm.launcher): The launcher for fine-tuning, default is ``launchers.remote(ngpus=1)``.
    stream (bool): Whether the response is streaming, default is ``False``.
    type (str): Type parameter, default is ``None``, which corresponds to the ``llm`` type. Additionally, the ``embed`` type is also supported.
    max_token_num (int): The maximum token length for the input fine-tuning model, default is ``1024``.
    launcher (lazyllm.launcher): The launcher for fine-tuning, default is ``launchers.remote(ngpus=1)``.
    kw: Keyword arguments used to update default training parameters. Note that whether additional keyword arguments can be specified depends on the framework inferred by LazyLLM, so it is recommended to set them carefully.



Examples:
    >>> from lazyllm import deploy
    >>> deploy.auto('internlm2-chat-7b')
    <lazyllm.llm.deploy type=Lightllm> 
    """
    @staticmethod
    def _get_embed_deployer(launcher, type, kw):
        launcher = launcher or launchers.remote(ngpus=1)
        kw['model_type'] = type
        if lazyllm.config['default_embedding_engine'].lower() in ('transformers', 'flagembedding') \
            or kw.get('embed_type')=='sparse' or not check_requirements('infinity_emb'):
            return deploy.Rerank if type == 'reranker' else deploy.Embedding, launcher, kw
        else:
            return deploy.InfinityRerank if type == 'reranker' else deploy.Infinity, launcher, kw

    @classmethod
    def get_deployer(cls, base_model: str, source: Optional[str] = None, trust_remote_code: bool = True,
                     launcher: Optional[LazyLLMLaunchersBase] = None, type: Optional[str] = None,
                     log_path: Optional[str] = None, **kw):
        model_name = get_model_name(base_model)
        kw['log_path'], kw['trust_remote_code'] = log_path, trust_remote_code
        if not type:
            type = ModelManager.get_model_type(model_name)

        if type in ('embed', 'cross_modal_embed', 'reranker'):
            return AutoDeploy._get_embed_deployer(launcher, type, kw)
        elif type == 'sd':
            return StableDiffusionDeploy, launcher or launchers.remote(ngpus=1), kw
        elif type == 'stt':
            return SenseVoiceDeploy, launcher or launchers.remote(ngpus=1), kw
        elif type == 'tts':
            return TTSDeploy.get_deploy_cls(model_name), launcher or launchers.remote(ngpus=1), kw
        elif type == 'vlm':
            return deploy.LMDeploy, launcher or launchers.remote(ngpus=1), kw
        elif type == 'ocr':
            return OCRDeploy, launcher or launchers.remote(ngpus=1), kw

        if not launcher:
            match = re.search(r'(\d+)[bB]', model_name)
            size = int(match.group(1)) if match else 0
            size = (size * 2) if 'awq' not in model_name.lower() else (size / 1.5)
            ngpus = (1 << (math.ceil(size * 2 / config['gpu_memory']) - 1).bit_length())
            launcher = launchers.remote(ngpus = ngpus)

        for deploy_cls in ['vllm', 'lightllm', 'lmdeploy', 'mindie']:
            if check_cmd(deploy_cls) or check_requirements(requirements.get(deploy_cls)):
                deploy_cls = getattr(deploy, deploy_cls)
                return deploy_cls, launcher, kw
        return deploy.auto, launcher, kw

    def __new__(cls, base_model, source=lazyllm.config['model_source'], trust_remote_code=True,
                launcher=None, type=None, log_path=None, **kw):
        deploy_cls, launcher, kw = __class__.get_deployer(
            base_model=base_model, source=source, trust_remote_code=trust_remote_code,
            launcher=launcher, type=type, log_path=log_path, **kw)
        return deploy_cls(launcher=launcher, **kw)

lazyllm.components.deploy.embed.AbstractEmbedding

Bases: ABC

Source code in lazyllm/components/deploy/embed.py 
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
class AbstractEmbedding(ABC):
    def __init__(self, base_embed, source=None, init=False):
        from ..utils.downloader import ModelManager
        self._source = source or lazyllm.config['model_source']
        self._base_embed = ModelManager(self._source).download(base_embed) or ''
        self._embed = None
        self._init = lazyllm.once_flag()
        if init:
            lazyllm.call_once(self._init, self.load_embed)

    @abstractmethod
    def load_embed(self) -> None:
        pass

    @abstractmethod
    def _call(self, data: Dict[str, Union[str, List[str]]]) -> str:
        pass

    def __call__(self, data: Dict[str, Union[str, List[str]]]) -> str:
        lazyllm.call_once(self._init, self.load_embed)
        return self._call(data)

    def __reduce__(self):
        init = bool(os.getenv('LAZYLLM_ON_CLOUDPICKLE', None) == 'ON' or self._init)
        return self.__class__, (self._base_embed, self._source, init)

lazyllm.components.deploy.EmbeddingDeploy

Bases: LazyLLMDeployBase

Source code in lazyllm/components/deploy/embed.py 
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
class EmbeddingDeploy(LazyLLMDeployBase):
    message_format = {
        'text': 'text',  # str,
        'images': []  # Union[str, List[str]]
    }
    keys_name_handle = {
        'inputs': 'text',
        'image': 'images'
    }
    default_headers = {'Content-Type': 'application/json'}

    def __init__(self, launcher: LazyLLMLaunchersBase = None, model_type: str = 'embed', log_path: Optional[str] = None,
                 embed_type: Optional[str] = 'dense', trust_remote_code: bool = True, port: Optional[int] = None):
        super().__init__(launcher=launcher)
        self._launcher = launcher
        self._port = port
        self._model_type = model_type
        self._log_path = log_path
        self._sparse_embed = True if embed_type == 'sparse' else False
        self._trust_remote_code = trust_remote_code
        self._port = port

    def _get_model_path(self, finetuned_model=None, base_model=None):
        if not os.path.exists(finetuned_model) or \
            not any(filename.endswith('.bin') or filename.endswith('.safetensors')
                    for filename in os.listdir(finetuned_model)):
            if not finetuned_model:
                LOG.warning(f"Note! That finetuned_model({finetuned_model}) is an invalid path, "
                            f"base_model({base_model}) will be used")
            finetuned_model = base_model
        return finetuned_model

    def __call__(self, finetuned_model=None, base_model=None):
        finetuned_model = self._get_model_path(finetuned_model, base_model)
        if self._sparse_embed or lazyllm.config['default_embedding_engine'] == 'flagEmbedding':
            return lazyllm.deploy.RelayServer(port=self._port, func=LazyFlagEmbedding(
                finetuned_model, sparse=self._sparse_embed),
                launcher=self._launcher, log_path=self._log_path, cls='embedding')()
        else:
            return lazyllm.deploy.RelayServer(port=self._port, func=HuggingFaceEmbedding(finetuned_model),
                                              launcher=self._launcher, log_path=self._log_path, cls='embedding')()

lazyllm.components.deploy.embed.RerankDeploy

Bases: EmbeddingDeploy

Source code in lazyllm/components/deploy/embed.py 
239
240
241
242
243
244
245
246
247
class RerankDeploy(EmbeddingDeploy):
    message_format = {'query': 'query', 'documents': ['string'], 'top_n': 1}
    keys_name_handle = {'inputs': 'query', 'documents': 'documents', 'top_n': 'top_n'}
    default_headers = {'Content-Type': 'application/json'}

    def __call__(self, finetuned_model=None, base_model=None):
        finetuned_model = self._get_model_path(finetuned_model, base_model)
        return lazyllm.deploy.RelayServer(port=self._port, func=LazyHuggingFaceRerank(
            finetuned_model), launcher=self._launcher, log_path=self._log_path, cls='embedding')()

lazyllm.components.deploy.embed.LazyHuggingFaceRerank

Bases: object

Source code in lazyllm/components/deploy/embed.py 
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
class LazyHuggingFaceRerank(object):
    def __init__(self, base_rerank, source=None, init=False):
        from ..utils.downloader import ModelManager
        source = lazyllm.config['model_source'] if not source else source
        self.base_rerank = ModelManager(source).download(base_rerank) or ''
        self.reranker = None
        self.init_flag = lazyllm.once_flag()
        if init:
            lazyllm.call_once(self.init_flag, self.load_reranker)

    def load_reranker(self):
        self.reranker = sentence_transformers.CrossEncoder(self.base_rerank)

    def __call__(self, inps):
        lazyllm.call_once(self.init_flag, self.load_reranker)
        query, documents, top_n = inps['query'], inps['documents'], inps['top_n']
        query_pairs = [(query, doc) for doc in documents]
        scores = self.reranker.predict(query_pairs)
        sorted_indices = [(index, scores[index]) for index in np.argsort(scores)[::-1]]
        if top_n > 0:
            sorted_indices = sorted_indices[:top_n]
        return sorted_indices

    @classmethod
    def rebuild(cls, base_rerank, init):
        return cls(base_rerank, init)

    def __reduce__(self):
        init = bool(os.getenv('LAZYLLM_ON_CLOUDPICKLE', None) == 'ON' or self.init_flag)
        return LazyHuggingFaceRerank.rebuild, (self.base_rerank, init)

load_reranker()

Source code in lazyllm/components/deploy/embed.py 
142
143
def load_reranker(self):
    self.reranker = sentence_transformers.CrossEncoder(self.base_rerank)

rebuild(base_rerank, init) classmethod

Source code in lazyllm/components/deploy/embed.py 
155
156
157
@classmethod
def rebuild(cls, base_rerank, init):
    return cls(base_rerank, init)

lazyllm.components.deploy.Mindie

Bases: LazyLLMDeployBase

This class is a subclass of LazyLLMDeployBase, designed for deploying and managing the MindIE large language model inference service. It encapsulates the full workflow including configuration generation, process launching, and API interaction for the MindIE service. Args: trust_remote_code (bool): Whether to trust remote code (e.g., from HuggingFace models). Default is True. launcher: Instance of the task launcher. Default is launchers.remote(). log_path (str): Path to save logs. If None, logs will not be saved. **kw: Other configuration parameters. Supports the following keys: - npuDeviceIds: List of NPU device IDs (e.g., [[0,1]] indicates using 2 devices) - worldSize: Model parallelism size - port: Service port (set to 'auto' for auto-assignment between 30000–40000) - maxSeqLen: Maximum sequence length - maxInputTokenLen: Maximum number of tokens per input - maxPrefillTokens: Maximum number of prefill tokens - config: Custom configuration file Note: You must set the environment variable LAZYLLM_MINDIE_HOME to point to the MindIE installation directory. If finetuned_model is not specified or the path is invalid, it will automatically fall back to base_model.

Examples:

>>> import lazyllm
>>> from lazyllm.components.deploy import Mindie            
>>> deployer = Mindie(
...     port=30000,
...     launcher=lazyllm.launchers.remote(),
...     max_seq_len=32000,
...     log_path="/path/to/logs"
... )
>>> cmd = deployer.cmd(
...     finetuned_model="/path/to/finetuned_model",
...     base_model="/path/to/base_model")
>>> print("Service URL:", cmd.geturl())
Source code in lazyllm/components/deploy/mindie.py 
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
class Mindie(LazyLLMDeployBase):
    """This class is a subclass of ``LazyLLMDeployBase``, designed for deploying and managing the MindIE large language model inference service. It encapsulates the full workflow including configuration generation, process launching, and API interaction for the MindIE service.
Args:
    trust_remote_code (bool): Whether to trust remote code (e.g., from HuggingFace models). Default is ``True``.
    launcher: Instance of the task launcher. Default is ``launchers.remote()``.
    log_path (str): Path to save logs. If ``None``, logs will not be saved.
    **kw: Other configuration parameters. Supports the following keys:
        - npuDeviceIds: List of NPU device IDs (e.g., ``[[0,1]]`` indicates using 2 devices)
        - worldSize: Model parallelism size
        - port: Service port (set to ``'auto'`` for auto-assignment between 30000–40000)
        - maxSeqLen: Maximum sequence length
        - maxInputTokenLen: Maximum number of tokens per input
        - maxPrefillTokens: Maximum number of prefill tokens
        - config: Custom configuration file
Note:
    You must set the environment variable ``LAZYLLM_MINDIE_HOME`` to point to the MindIE installation directory. 
    If ``finetuned_model`` is not specified or the path is invalid, it will automatically fall back to ``base_model``.


Examples:
    >>> import lazyllm
    >>> from lazyllm.components.deploy import Mindie            
    >>> deployer = Mindie(
    ...     port=30000,
    ...     launcher=lazyllm.launchers.remote(),
    ...     max_seq_len=32000,
    ...     log_path="/path/to/logs"
    ... )
    >>> cmd = deployer.cmd(
    ...     finetuned_model="/path/to/finetuned_model",
    ...     base_model="/path/to/base_model")
    >>> print("Service URL:", cmd.geturl())

    """
    keys_name_handle = {
        'inputs': 'prompt',
    }
    default_headers = {'Content-Type': 'application/json'}
    message_format = {
        'prompt': 'Who are you ?',
        'stream': False,
        'max_tokens': 4096,
        'presence_penalty': 1.03,
        'frequency_penalty': 1.0,
        'temperature': 0.5,
        'top_p': 0.95
    }
    auto_map = {
        'port': int,
        'tp': ('world_size', int),
        'max_input_token_len': ('maxInputTokenLen', int),
        'max_prefill_tokens': ('maxPrefillTokens', int),
        'max_seq_len': ('maxSeqLen', int)
    }

    def __init__(self, trust_remote_code=True, launcher=launchers.remote(), log_path=None, **kw):  # noqa B008
        super().__init__(launcher=launcher)
        assert lazyllm.config['mindie_home'], 'Ensure you have installed MindIE and \
                                  "export LAZYLLM_MINDIE_HOME=/path/to/mindie/latest"'
        self.mindie_home = lazyllm.config['mindie_home']
        self.mindie_config_path = os.path.join(self.mindie_home, 'mindie-service/conf/config.json')
        self.backup_path = self.mindie_config_path + '.backup'
        self.custom_config = kw.pop('config', None)
        self.kw = ArgsDict({
            'npuDeviceIds': [[0]],
            'worldSize': 1,
            'port': 'auto',
            'host': '0.0.0.0',
            'maxSeqLen': 64000,
            'maxInputTokenLen': 4096,
            'maxPrefillTokens': 8192,
        })
        self.trust_remote_code = trust_remote_code
        self.kw.check_and_update(kw)
        self.kw['npuDeviceIds'] = [[i for i in range(self.kw.get('worldSize', 1))]]
        self.random_port = False if 'port' in kw and kw['port'] and kw['port'] != 'auto' else True
        self.temp_folder = make_log_dir(log_path, 'mindie') if log_path else None

        if self.custom_config:
            self.config_dict = (ArgsDict(self.load_config(self.custom_config))
                                if isinstance(self.custom_config, str) else ArgsDict(self.custom_config))
            self.kw['host'] = self.config_dict["ServerConfig"]["ipAddress"]
            self.kw['port'] = self.config_dict["ServerConfig"]["port"]
        else:
            default_config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'mindie', 'config.json')
            self.config_dict = ArgsDict(self.load_config(default_config_path))

    def __del__(self):
        if hasattr(self, 'backup_path') and os.path.isfile(self.backup_path):
            shutil.copy2(self.backup_path, self.mindie_config_path)

    def load_config(self, config_path):
        """Loads and parses the MindIE configuration file.

Args:
    config_path (str): Path to the JSON configuration file

Returns:
    dict: Parsed configuration dictionary

Notes:
    - Handles both default and custom configuration files
    - Uses JSON format for configuration
    - Creates backup of original config before modification
"""
        with open(config_path, 'r') as file:
            config_dict = json.load(file)
        return config_dict

    def save_config(self):
        """Saves the current configuration to file.

Notes:
    - Automatically creates backup of existing config
    - Writes to the standard MindIE config location
    - Uses JSON format with proper indentation
    - Called automatically during deployment
"""
        if os.path.isfile(self.mindie_config_path):
            shutil.copy2(self.mindie_config_path, self.backup_path)

        with open(self.mindie_config_path, 'w') as file:
            json.dump(self.config_dict, file)

    def update_config(self):
        """Updates the configuration dictionary with current settings.

Notes:
    - Handles multiple configuration sections:
        - Model deployment parameters
        - Server settings
        - Scheduling parameters
"""
        backend_config = self.config_dict["BackendConfig"]
        backend_config["npuDeviceIds"] = self.kw["npuDeviceIds"]
        model_config = {
            "modelName": self.finetuned_model.split('/')[-1],
            "modelWeightPath": self.finetuned_model,
            "worldSize": self.kw["worldSize"],
            "trust_remote_code": self.trust_remote_code
        }
        backend_config["ModelDeployConfig"]["ModelConfig"][0].update(model_config)
        backend_config["ModelDeployConfig"]["maxSeqLen"] = self.kw["maxSeqLen"]
        backend_config["ModelDeployConfig"]["maxInputTokenLen"] = self.kw["maxInputTokenLen"]
        backend_config["ScheduleConfig"]["maxPrefillTokens"] = self.kw["maxPrefillTokens"]
        self.config_dict["BackendConfig"] = backend_config
        if self.kw["host"] != '0.0.0.0':
            self.config_dict["ServerConfig"]["ipAddress"] = self.kw["host"]
        self.config_dict["ServerConfig"]["port"] = self.kw["port"]

    def cmd(self, finetuned_model=None, base_model=None, master_ip=None):
        """Generates the command to start the MindIE service.

Args:
    finetuned_model (str): Path to the fine-tuned model
    base_model (str): Path to the base model (fallback if finetuned_model is invalid)
    master_ip (str): Master node IP address (currently unused)

Returns:
    LazyLLMCMD: Command object for starting the service

Notes:
    - Automatically handles model path validation
    - Updates configuration before service start
    - Supports random port allocation when configured
"""
        if self.custom_config is None:
            self.finetuned_model = finetuned_model
            if finetuned_model or base_model:
                if not os.path.exists(finetuned_model) or \
                    not any(filename.endswith('.bin') or filename.endswith('.safetensors')
                            for filename in os.listdir(finetuned_model)):
                    if not finetuned_model:
                        LOG.warning(f"Note! That finetuned_model({finetuned_model}) is an invalid path, "
                                    f"base_model({base_model}) will be used")
                    self.finetuned_model = base_model

            if self.random_port:
                self.kw['port'] = random.randint(30000, 40000)

            self.update_config()

        self.save_config()

        def impl():
            cmd = f'{os.path.join(self.mindie_home, "mindie-service/bin/mindieservice_daemon")}'
            if self.temp_folder: cmd += f' 2>&1 | tee {get_log_path(self.temp_folder)}'
            return cmd

        return LazyLLMCMD(cmd=impl, return_value=self.geturl, checkf=verify_fastapi_func)

    def geturl(self, job=None):
        """Gets the service URL after deployment.

Args:
    job: Job object (optional, defaults to self.job)

Returns:
    str: The generate endpoint URL

Notes:
    - Returns different formats based on display mode
    - Includes port number from configuration
"""
        if job is None:
            job = self.job
        if lazyllm.config['mode'] == lazyllm.Mode.Display:
            return f'http://{job.get_jobip()}:{self.kw["port"]}/generate'
        else:
            LOG.info(f"MindIE Server running on http://{job.get_jobip()}:{self.kw['port']}")
            return f'http://{job.get_jobip()}:{self.kw["port"]}/generate'

    @staticmethod
    def extract_result(x, inputs):
        return json.loads(x)['text'][0]

cmd(finetuned_model=None, base_model=None, master_ip=None)

Generates the command to start the MindIE service.

Parameters:

  • finetuned_model (str, default: None ) –

    Path to the fine-tuned model

  • base_model (str, default: None ) –

    Path to the base model (fallback if finetuned_model is invalid)

  • master_ip (str, default: None ) –

    Master node IP address (currently unused)

Returns:

  • LazyLLMCMD –

    Command object for starting the service

Notes
  • Automatically handles model path validation
  • Updates configuration before service start
  • Supports random port allocation when configured
Source code in lazyllm/components/deploy/mindie.py 
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
    def cmd(self, finetuned_model=None, base_model=None, master_ip=None):
        """Generates the command to start the MindIE service.

Args:
    finetuned_model (str): Path to the fine-tuned model
    base_model (str): Path to the base model (fallback if finetuned_model is invalid)
    master_ip (str): Master node IP address (currently unused)

Returns:
    LazyLLMCMD: Command object for starting the service

Notes:
    - Automatically handles model path validation
    - Updates configuration before service start
    - Supports random port allocation when configured
"""
        if self.custom_config is None:
            self.finetuned_model = finetuned_model
            if finetuned_model or base_model:
                if not os.path.exists(finetuned_model) or \
                    not any(filename.endswith('.bin') or filename.endswith('.safetensors')
                            for filename in os.listdir(finetuned_model)):
                    if not finetuned_model:
                        LOG.warning(f"Note! That finetuned_model({finetuned_model}) is an invalid path, "
                                    f"base_model({base_model}) will be used")
                    self.finetuned_model = base_model

            if self.random_port:
                self.kw['port'] = random.randint(30000, 40000)

            self.update_config()

        self.save_config()

        def impl():
            cmd = f'{os.path.join(self.mindie_home, "mindie-service/bin/mindieservice_daemon")}'
            if self.temp_folder: cmd += f' 2>&1 | tee {get_log_path(self.temp_folder)}'
            return cmd

        return LazyLLMCMD(cmd=impl, return_value=self.geturl, checkf=verify_fastapi_func)

geturl(job=None)

Gets the service URL after deployment.

Parameters:

  • job –

    Job object (optional, defaults to self.job)

Returns:

  • str –

    The generate endpoint URL

Notes
  • Returns different formats based on display mode
  • Includes port number from configuration
Source code in lazyllm/components/deploy/mindie.py 
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
    def geturl(self, job=None):
        """Gets the service URL after deployment.

Args:
    job: Job object (optional, defaults to self.job)

Returns:
    str: The generate endpoint URL

Notes:
    - Returns different formats based on display mode
    - Includes port number from configuration
"""
        if job is None:
            job = self.job
        if lazyllm.config['mode'] == lazyllm.Mode.Display:
            return f'http://{job.get_jobip()}:{self.kw["port"]}/generate'
        else:
            LOG.info(f"MindIE Server running on http://{job.get_jobip()}:{self.kw['port']}")
            return f'http://{job.get_jobip()}:{self.kw["port"]}/generate'

load_config(config_path)

Loads and parses the MindIE configuration file.

Parameters:

  • config_path (str) –

    Path to the JSON configuration file

Returns:

  • dict –

    Parsed configuration dictionary

Notes
  • Handles both default and custom configuration files
  • Uses JSON format for configuration
  • Creates backup of original config before modification
Source code in lazyllm/components/deploy/mindie.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
    def load_config(self, config_path):
        """Loads and parses the MindIE configuration file.

Args:
    config_path (str): Path to the JSON configuration file

Returns:
    dict: Parsed configuration dictionary

Notes:
    - Handles both default and custom configuration files
    - Uses JSON format for configuration
    - Creates backup of original config before modification
"""
        with open(config_path, 'r') as file:
            config_dict = json.load(file)
        return config_dict

save_config()

Saves the current configuration to file.

Notes
  • Automatically creates backup of existing config
  • Writes to the standard MindIE config location
  • Uses JSON format with proper indentation
  • Called automatically during deployment
Source code in lazyllm/components/deploy/mindie.py 
124
125
126
127
128
129
130
131
132
133
134
135
136
137
    def save_config(self):
        """Saves the current configuration to file.

Notes:
    - Automatically creates backup of existing config
    - Writes to the standard MindIE config location
    - Uses JSON format with proper indentation
    - Called automatically during deployment
"""
        if os.path.isfile(self.mindie_config_path):
            shutil.copy2(self.mindie_config_path, self.backup_path)

        with open(self.mindie_config_path, 'w') as file:
            json.dump(self.config_dict, file)

update_config()

Updates the configuration dictionary with current settings.

Notes
  • Handles multiple configuration sections:
    • Model deployment parameters
    • Server settings
    • Scheduling parameters
Source code in lazyllm/components/deploy/mindie.py 
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
    def update_config(self):
        """Updates the configuration dictionary with current settings.

Notes:
    - Handles multiple configuration sections:
        - Model deployment parameters
        - Server settings
        - Scheduling parameters
"""
        backend_config = self.config_dict["BackendConfig"]
        backend_config["npuDeviceIds"] = self.kw["npuDeviceIds"]
        model_config = {
            "modelName": self.finetuned_model.split('/')[-1],
            "modelWeightPath": self.finetuned_model,
            "worldSize": self.kw["worldSize"],
            "trust_remote_code": self.trust_remote_code
        }
        backend_config["ModelDeployConfig"]["ModelConfig"][0].update(model_config)
        backend_config["ModelDeployConfig"]["maxSeqLen"] = self.kw["maxSeqLen"]
        backend_config["ModelDeployConfig"]["maxInputTokenLen"] = self.kw["maxInputTokenLen"]
        backend_config["ScheduleConfig"]["maxPrefillTokens"] = self.kw["maxPrefillTokens"]
        self.config_dict["BackendConfig"] = backend_config
        if self.kw["host"] != '0.0.0.0':
            self.config_dict["ServerConfig"]["ipAddress"] = self.kw["host"]
        self.config_dict["ServerConfig"]["port"] = self.kw["port"]

lazyllm.components.deploy.OCRDeploy

Bases: LazyLLMDeployBase

OCRDeploy is a subclass of LazyLLMDeployBase that provides deployment for OCR (Optical Character Recognition) models. This class is designed to deploy OCR models with additional configurations such as logging, trust for remote code, and port customization. Attributes: keys_name_handle: A dictionary mapping input keys to their corresponding handler keys. For example: - "inputs": Handles general inputs. - "ocr_files": Also mapped to "inputs". message_format: A dictionary specifying the expected message format. For example: - {"inputs": "/path/to/pdf"} indicates that the model expects a PDF file path as input. default_headers: A dictionary specifying default headers for API requests. Defaults to: - {"Content-Type": "application/json"} Args: launcher: A launcher instance for deploying the model. Defaults to None. log_path: A string specifying the path where logs should be saved. Defaults to None. trust_remote_code: A boolean indicating whether to trust remote code execution. Defaults to True. port: An integer specifying the port for the deployment server. Defaults to None. finetuned_model: A string specifying the path or name of the fine-tuned OCR model. Defaults to None. base_model: A string specifying the base model name. If finetuned_model is not provided, base_model will be used. Defaults to None. Returns: An instance of [RelayServer][lazyllm.deploy.RelayServer], which acts as the deployment server for the OCR model. Example: 

deployer = OCRDeploy(launcher=launchers.local(), log_path='./logs', port=8080)
server = deployer(finetuned_model='ocr-model')
print(server)  # RelayServer instance ready to handle OCR requests

Examples:

>>> from lazyllm.components import OCRDeploy
>>> from lazyllm import launchers
>>> # 创建一个 OCRDeploy 实例
>>> deployer = OCRDeploy(launcher=launchers.local(), log_path='./logs', port=8080)
>>> # 使用微调的 OCR 模型部署服务器
>>> server = deployer(finetuned_model='ocr-model')
>>> # 打印部署服务器信息
>>> print(server)
... <RelayServer instance ready to handle OCR requests>
Source code in lazyllm/components/deploy/ocr/pp_ocr.py 
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
class OCRDeploy(LazyLLMDeployBase):
    """OCRDeploy is a subclass of [LazyLLMDeployBase][lazyllm.components.LazyLLMDeployBase] that provides deployment for OCR (Optical Character Recognition) models.
This class is designed to deploy OCR models with additional configurations such as logging, trust for remote code, and port customization.
     Attributes:
    keys_name_handle: A dictionary mapping input keys to their corresponding handler keys. For example:
        - "inputs": Handles general inputs.
        - "ocr_files": Also mapped to "inputs".
    message_format: A dictionary specifying the expected message format. For example:
        - {"inputs": "/path/to/pdf"} indicates that the model expects a PDF file path as input.
    default_headers: A dictionary specifying default headers for API requests. Defaults to:
        - {"Content-Type": "application/json"}
Args:
    launcher: A launcher instance for deploying the model. Defaults to `None`.
    log_path: A string specifying the path where logs should be saved. Defaults to `None`.
    trust_remote_code: A boolean indicating whether to trust remote code execution. Defaults to `True`.
    port: An integer specifying the port for the deployment server. Defaults to `None`.
    finetuned_model: A string specifying the path or name of the fine-tuned OCR model. Defaults to `None`.
    base_model: A string specifying the base model name. If `finetuned_model` is not provided, `base_model` will be used. Defaults to `None`.
Returns:
    An instance of [RelayServer][lazyllm.deploy.RelayServer], which acts as the deployment server for the OCR model.
Example:
    ```python
    deployer = OCRDeploy(launcher=launchers.local(), log_path='./logs', port=8080)
    server = deployer(finetuned_model='ocr-model')
    print(server)  # RelayServer instance ready to handle OCR requests
    ```


Examples:
    >>> from lazyllm.components import OCRDeploy
    >>> from lazyllm import launchers
    >>> # 创建一个 OCRDeploy 实例
    >>> deployer = OCRDeploy(launcher=launchers.local(), log_path='./logs', port=8080)
    >>> # 使用微调的 OCR 模型部署服务器
    >>> server = deployer(finetuned_model='ocr-model')
    >>> # 打印部署服务器信息
    >>> print(server)
    ... <RelayServer instance ready to handle OCR requests>
    """
    keys_name_handle = {
        "inputs": "inputs",
        "ocr_files": "inputs",
    }
    message_format = {"inputs": "/path/to/pdf"}
    default_headers = {"Content-Type": "application/json"}

    def __init__(self, launcher=None, log_path=None, trust_remote_code=True, port=None):
        super().__init__(launcher=launcher)
        self._log_path = log_path
        self._trust_remote_code = trust_remote_code
        self._port = port

    def __call__(self, finetuned_model=None, base_model=None):
        if not finetuned_model:
            finetuned_model = base_model
        return lazyllm.deploy.RelayServer(
            port=self._port, func=_OCR(finetuned_model), launcher=self._launcher, log_path=self._log_path, cls="ocr")()

lazyllm.components.deploy.relay.base.RelayServer

Bases: LazyLLMDeployBase

Source code in lazyllm/components/deploy/relay/base.py 
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
class RelayServer(LazyLLMDeployBase):
    keys_name_handle = None
    default_headers = {'Content-Type': 'application/json'}
    message_format = None

    def __init__(self, port=None, *, func=None, pre_func=None, post_func=None,
                 pythonpath=None, log_path=None, cls=None, launcher=launchers.remote(sync=False)):  # noqa B008
        # func must dump in __call__ to wait for dependancies.
        self.func = func
        self.pre = dump_obj(pre_func)
        self.post = dump_obj(post_func)
        self.port, self.real_port = port, None
        self.pythonpath = pythonpath
        super().__init__(launcher=launcher)
        self.temp_folder = make_log_dir(log_path, cls or 'relay') if log_path else None

    def cmd(self, func=None):
        FastapiApp.update()
        self.func = dump_obj(func or self.func)
        folder_path = os.path.dirname(os.path.abspath(__file__))
        run_file_path = os.path.join(folder_path, 'server.py')

        def impl():
            self.real_port = self.port if self.port else random.randint(30000, 40000)
            cmd = f'{sys.executable} {run_file_path} --open_port={self.real_port} --function="{self.func}" '
            if self.pre:
                cmd += f'--before_function="{self.pre}" '
            if self.post:
                cmd += f'--after_function="{self.post}" '
            if self.pythonpath:
                cmd += f'--pythonpath="{self.pythonpath}" '
            if self.temp_folder: cmd += f' 2>&1 | tee {get_log_path(self.temp_folder)}'
            return cmd

        return LazyLLMCMD(cmd=impl, return_value=self.geturl, checkf=verify_fastapi_func,
                          no_displays=['function', 'before_function', 'after_function'])

    def geturl(self, job=None):
        if job is None:
            job = self.job
        return f'http://{job.get_jobip()}:{self.real_port}/generate'

cmd(func=None)

Source code in lazyllm/components/deploy/relay/base.py 
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
def cmd(self, func=None):
    FastapiApp.update()
    self.func = dump_obj(func or self.func)
    folder_path = os.path.dirname(os.path.abspath(__file__))
    run_file_path = os.path.join(folder_path, 'server.py')

    def impl():
        self.real_port = self.port if self.port else random.randint(30000, 40000)
        cmd = f'{sys.executable} {run_file_path} --open_port={self.real_port} --function="{self.func}" '
        if self.pre:
            cmd += f'--before_function="{self.pre}" '
        if self.post:
            cmd += f'--after_function="{self.post}" '
        if self.pythonpath:
            cmd += f'--pythonpath="{self.pythonpath}" '
        if self.temp_folder: cmd += f' 2>&1 | tee {get_log_path(self.temp_folder)}'
        return cmd

    return LazyLLMCMD(cmd=impl, return_value=self.geturl, checkf=verify_fastapi_func,
                      no_displays=['function', 'before_function', 'after_function'])

geturl(job=None)

Source code in lazyllm/components/deploy/relay/base.py 
48
49
50
51
def geturl(self, job=None):
    if job is None:
        job = self.job
    return f'http://{job.get_jobip()}:{self.real_port}/generate'

lazyllm.components.deploy.OCRDeploy

Bases: LazyLLMDeployBase

OCRDeploy is a subclass of LazyLLMDeployBase that provides deployment for OCR (Optical Character Recognition) models. This class is designed to deploy OCR models with additional configurations such as logging, trust for remote code, and port customization. Attributes: keys_name_handle: A dictionary mapping input keys to their corresponding handler keys. For example: - "inputs": Handles general inputs. - "ocr_files": Also mapped to "inputs". message_format: A dictionary specifying the expected message format. For example: - {"inputs": "/path/to/pdf"} indicates that the model expects a PDF file path as input. default_headers: A dictionary specifying default headers for API requests. Defaults to: - {"Content-Type": "application/json"} Args: launcher: A launcher instance for deploying the model. Defaults to None. log_path: A string specifying the path where logs should be saved. Defaults to None. trust_remote_code: A boolean indicating whether to trust remote code execution. Defaults to True. port: An integer specifying the port for the deployment server. Defaults to None. finetuned_model: A string specifying the path or name of the fine-tuned OCR model. Defaults to None. base_model: A string specifying the base model name. If finetuned_model is not provided, base_model will be used. Defaults to None. Returns: An instance of [RelayServer][lazyllm.deploy.RelayServer], which acts as the deployment server for the OCR model. Example: 

deployer = OCRDeploy(launcher=launchers.local(), log_path='./logs', port=8080)
server = deployer(finetuned_model='ocr-model')
print(server)  # RelayServer instance ready to handle OCR requests

Examples:

>>> from lazyllm.components import OCRDeploy
>>> from lazyllm import launchers
>>> # 创建一个 OCRDeploy 实例
>>> deployer = OCRDeploy(launcher=launchers.local(), log_path='./logs', port=8080)
>>> # 使用微调的 OCR 模型部署服务器
>>> server = deployer(finetuned_model='ocr-model')
>>> # 打印部署服务器信息
>>> print(server)
... <RelayServer instance ready to handle OCR requests>
Source code in lazyllm/components/deploy/ocr/pp_ocr.py 
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
class OCRDeploy(LazyLLMDeployBase):
    """OCRDeploy is a subclass of [LazyLLMDeployBase][lazyllm.components.LazyLLMDeployBase] that provides deployment for OCR (Optical Character Recognition) models.
This class is designed to deploy OCR models with additional configurations such as logging, trust for remote code, and port customization.
     Attributes:
    keys_name_handle: A dictionary mapping input keys to their corresponding handler keys. For example:
        - "inputs": Handles general inputs.
        - "ocr_files": Also mapped to "inputs".
    message_format: A dictionary specifying the expected message format. For example:
        - {"inputs": "/path/to/pdf"} indicates that the model expects a PDF file path as input.
    default_headers: A dictionary specifying default headers for API requests. Defaults to:
        - {"Content-Type": "application/json"}
Args:
    launcher: A launcher instance for deploying the model. Defaults to `None`.
    log_path: A string specifying the path where logs should be saved. Defaults to `None`.
    trust_remote_code: A boolean indicating whether to trust remote code execution. Defaults to `True`.
    port: An integer specifying the port for the deployment server. Defaults to `None`.
    finetuned_model: A string specifying the path or name of the fine-tuned OCR model. Defaults to `None`.
    base_model: A string specifying the base model name. If `finetuned_model` is not provided, `base_model` will be used. Defaults to `None`.
Returns:
    An instance of [RelayServer][lazyllm.deploy.RelayServer], which acts as the deployment server for the OCR model.
Example:
    ```python
    deployer = OCRDeploy(launcher=launchers.local(), log_path='./logs', port=8080)
    server = deployer(finetuned_model='ocr-model')
    print(server)  # RelayServer instance ready to handle OCR requests
    ```


Examples:
    >>> from lazyllm.components import OCRDeploy
    >>> from lazyllm import launchers
    >>> # 创建一个 OCRDeploy 实例
    >>> deployer = OCRDeploy(launcher=launchers.local(), log_path='./logs', port=8080)
    >>> # 使用微调的 OCR 模型部署服务器
    >>> server = deployer(finetuned_model='ocr-model')
    >>> # 打印部署服务器信息
    >>> print(server)
    ... <RelayServer instance ready to handle OCR requests>
    """
    keys_name_handle = {
        "inputs": "inputs",
        "ocr_files": "inputs",
    }
    message_format = {"inputs": "/path/to/pdf"}
    default_headers = {"Content-Type": "application/json"}

    def __init__(self, launcher=None, log_path=None, trust_remote_code=True, port=None):
        super().__init__(launcher=launcher)
        self._log_path = log_path
        self._trust_remote_code = trust_remote_code
        self._port = port

    def __call__(self, finetuned_model=None, base_model=None):
        if not finetuned_model:
            finetuned_model = base_model
        return lazyllm.deploy.RelayServer(
            port=self._port, func=_OCR(finetuned_model), launcher=self._launcher, log_path=self._log_path, cls="ocr")()

Prompter

lazyllm.components.prompter.LazyLLMPrompterBase

The base class of Prompter. A custom Prompter needs to inherit from this base class and set the Prompt template and the Instruction template using the _init_prompt function provided by the base class, as well as the string used to capture results. Refer to prompt for further understanding of the design philosophy and usage of Prompts.

Both the Prompt template and the Instruction template use {} to indicate the fields to be filled in. The fields that can be included in the Prompt are system, history, tools, user etc., while the fields that can be included in the instruction_template are instruction and extra_keys. If the instruction field is a string, it is considered as a system instruction; if it is a dictionary, it can only contain the keys user and system. user represents the user input instruction, which is placed before the user input in the prompt, and system represents the system instruction, which is placed after the system prompt in the prompt. instruction is passed in by the application developer, and the instruction can also contain {} to define fillable fields, making it convenient for users to input additional information.

Examples:

>>> from lazyllm.components.prompter import PrompterBase
>>> class MyPrompter(PrompterBase):
...     def __init__(self, instruction = None, extra_keys = None, show = False):
...         super(__class__, self).__init__(show)
...         instruction_template = f'{instruction}\n{{extra_keys}}\n'.replace('{extra_keys}', PrompterBase._get_extro_key_template(extra_keys))
...         self._init_prompt("<system>{system}</system>\n</instruction>{instruction}</instruction>{history}\n{input}\n, ## Response::", instruction_template, '## Response::')
... 
>>> p = MyPrompter('ins {instruction}')
>>> p.generate_prompt('hello')
'<system>You are an AI-Agent developed by LazyLLM.</system>\n</instruction>ins hello\n\n</instruction>\n\n, ## Response::'
>>> p.generate_prompt('hello world', return_dict=True)
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nins hello world\n\n'}, {'role': 'user', 'content': ''}]}
Source code in lazyllm/components/prompter/builtinPrompt.py 
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
class LazyLLMPrompterBase(metaclass=LazyLLMRegisterMetaClass):
    """The base class of Prompter. A custom Prompter needs to inherit from this base class and set the Prompt template and the Instruction template using the `_init_prompt` function provided by the base class, as well as the string used to capture results. Refer to  [prompt](/Best%20Practice/prompt) for further understanding of the design philosophy and usage of Prompts.

Both the Prompt template and the Instruction template use ``{}`` to indicate the fields to be filled in. The fields that can be included in the Prompt are `system`, `history`, `tools`, `user` etc., while the fields that can be included in the instruction_template are `instruction` and `extra_keys`. If the ``instruction`` field is a string, it is considered as a system instruction; if it is a dictionary, it can only contain the keys ``user`` and ``system``. ``user`` represents the user input instruction, which is placed before the user input in the prompt, and ``system`` represents the system instruction, which is placed after the system prompt in the prompt.
``instruction`` is passed in by the application developer, and the ``instruction`` can also contain ``{}`` to define fillable fields, making it convenient for users to input additional information.


Examples:
    >>> from lazyllm.components.prompter import PrompterBase
    >>> class MyPrompter(PrompterBase):
    ...     def __init__(self, instruction = None, extra_keys = None, show = False):
    ...         super(__class__, self).__init__(show)
    ...         instruction_template = f'{instruction}\\n{{extra_keys}}\\n'.replace('{extra_keys}', PrompterBase._get_extro_key_template(extra_keys))
    ...         self._init_prompt("<system>{system}</system>\\n</instruction>{instruction}</instruction>{history}\\n{input}\\n, ## Response::", instruction_template, '## Response::')
    ... 
    >>> p = MyPrompter('ins {instruction}')
    >>> p.generate_prompt('hello')
    '<system>You are an AI-Agent developed by LazyLLM.</system>\\n</instruction>ins hello\\n\\n</instruction>\\n\\n, ## Response::'
    >>> p.generate_prompt('hello world', return_dict=True)
    {'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\nins hello world\\n\\n'}, {'role': 'user', 'content': ''}]}
    """
    ISA = "<!lazyllm-spliter!>"
    ISE = "</!lazyllm-spliter!>"

    def __init__(self, show=False, tools=None, history=None):
        self._set_model_configs(system='You are an AI-Agent developed by LazyLLM.', sos='',
                                soh='', soa='', eos='', eoh='', eoa='')
        self._show = show
        self._tools = tools
        self._pre_hook = None
        self._history = history or []

    def _init_prompt(self, template: str, instruction_template: str, split: Union[None, str] = None):
        self._template = template
        self._instruction_template = instruction_template
        if split:
            assert not hasattr(self, '_split')
            self._split = split

    @staticmethod
    def _get_extro_key_template(extra_keys, prefix='Here are some extra messages you can referred to:\n\n'):
        if extra_keys:
            if isinstance(extra_keys, str): extra_keys = [extra_keys]
            assert isinstance(extra_keys, (tuple, list)), 'Only str, tuple[str], list[str] are supported'
            return prefix + ''.join([f"### {k}:\n{{{k}}}\n\n" for k in extra_keys])
        return ''

    def _handle_tool_call_instruction(self):
        tool_dict = {}
        for key in ["tool_start_token", "tool_args_token", "tool_end_token"]:
            if getattr(self, f"_{key}", None) and key in self._instruction_template:
                tool_dict[key] = getattr(self, f"_{key}")
        return reduce(lambda s, kv: s.replace(f"{{{kv[0]}}}", kv[1]), tool_dict.items(), self._instruction_template)

    def _set_model_configs(self, system: str = None, sos: Union[None, str] = None, soh: Union[None, str] = None,
                           soa: Union[None, str] = None, eos: Union[None, str] = None,
                           eoh: Union[None, str] = None, eoa: Union[None, str] = None,
                           soe: Union[None, str] = None, eoe: Union[None, str] = None,
                           separator: Union[None, str] = None, plugin: Union[None, str] = None,
                           interpreter: Union[None, str] = None, stop_words: Union[None, List[str]] = None,
                           tool_start_token: Union[None, str] = None, tool_end_token: Union[None, str] = None,
                           tool_args_token: Union[None, str] = None):

        local = locals()
        for name in ['system', 'sos', 'soh', 'soa', 'eos', 'eoh', 'eoa', 'soe', 'eoe', 'tool_start_token',
                     'tool_end_token', 'tool_args_token']:
            if local[name] is not None: setattr(self, f'_{name}', local[name])

        if getattr(self, "_instruction_template", None):
            self._instruction_template = self._handle_tool_call_instruction()

    def _get_tools(self, tools, *, return_dict):
        if self._tools:
            assert tools is None
            tools = self._tools

        return tools if return_dict else '### Function-call Tools. \n\n' + json.dumps(tools) + '\n\n' if tools else ''

    def _get_histories(self, history, *, return_dict):  # noqa: C901
        if not self._history and not history: return ''
        if return_dict:
            content = []
            for item in self._history + (history or []):
                if isinstance(item, list):
                    assert len(item) <= 2, "history item length cannot be greater than 2"
                    if len(item) > 0: content.append({"role": "user", "content": item[0]})
                    if len(item) > 1: content.append({"role": "assistant", "content": item[1]})
                elif isinstance(item, dict):
                    content.append(item)
                else:
                    LOG.error(f"history: {history}")
                    raise ValueError("history must be a list of list or dict")
            return content
        else:
            ret = ''.join([f'{self._soh}{h}{self._eoh}{self._soa}{a}{self._eoa}' for h, a in self._history])
            if not history: return ret
            if isinstance(history[0], list):
                return ret + ''.join([f'{self._soh}{h}{self._eoh}{self._soa}{a}{self._eoa}' for h, a in history])
            elif isinstance(history[0], dict):
                for item in history:
                    if item['role'] == "user":
                        ret += f'{self._soh}{item["content"]}{self._eoh}'
                    elif item['role'] == "assistant":
                        ret += f'{self._soa}'
                        ret += f'{item.get("content", "")}'
                        for idx in range(len(item.get('tool_calls', []))):
                            tool = item['tool_calls'][idx]['function']
                            if getattr(self, "_tool_args_token", None):
                                tool = tool['name'] + self._tool_args_token + \
                                    json.dumps(tool['arguments'], ensure_ascii=False)
                            ret += (f'{getattr(self, "_tool_start_token", "")}' + '\n'
                                    f'{tool}'
                                    f'{getattr(self, "_tool_end_token", "")}' + '\n')
                        ret += f'{self._eoa}'
                    elif item['role'] == "tool":
                        try:
                            content = json.loads(item['content'].strip())
                        except Exception:
                            content = item['content']
                        ret += f'{getattr(self, "_soe", "")}{content}{getattr(self, "_eoe", "")}'

                return ret
            else:
                raise NotImplementedError('Cannot transform json history to {type(history[0])} now')

    def _get_instruction_and_input(self, input):
        prompt_keys = list(set(re.findall(r'\{(\w+)\}', self._instruction_template)))
        if isinstance(input, (str, int)):
            if len(prompt_keys) == 1:
                return self._instruction_template.format(**{prompt_keys[0]: input}), ''
            else:
                assert len(prompt_keys) == 0
                return self._instruction_template, input
        assert isinstance(input, dict), f'expected types are str, int and dict, bug get {type(input)}(`{input})`'
        kwargs = {k: input.pop(k) for k in prompt_keys}
        assert len(input) <= 1, f"Unexpected keys found in input: {list(input.keys())}"
        return (reduce(lambda s, kv: s.replace(f"{{{kv[0]}}}", kv[1]),
                       kwargs.items(),
                       self._instruction_template)
                if len(kwargs) > 0 else self._instruction_template,
                list(input.values())[0] if input else "")

    def _check_values(self, instruction, input, history, tools): pass

    # Used for TrainableModule(local deployed)
    def _generate_prompt_impl(self, instruction, input, user, history, tools, label):
        is_tool = False
        if isinstance(input, dict):
            input = input.get('content', '')
            is_tool = input.get('role') == 'tool'
        elif isinstance(input, list):
            is_tool = any(item.get('role') == 'tool' for item in input)
            input = "\n".join([item.get('content', '') for item in input])
        params = dict(system=self._system, instruction=instruction, input=input, user=user, history=history, tools=tools,
                      sos=self._sos, eos=self._eos, soh=self._soh, eoh=self._eoh, soa=self._soa, eoa=self._eoa)
        if is_tool:
            params['soh'] = getattr(self, "_soe", self._soh)
            params['eoh'] = getattr(self, "_eoe", self._eoh)
        return self._template.format(**params) + (label if label else '')

    # Used for OnlineChatModule
    def _generate_prompt_dict_impl(self, instruction, input, user, history, tools, label):
        if not history: history = []
        if isinstance(input, str):
            history.append({"role": "user", "content": input})
        elif isinstance(input, dict):
            history.append(input)
        elif isinstance(input, list) and all(isinstance(ele, dict) for ele in input):
            history.extend(input)
        elif isinstance(input, tuple) and len(input) == 1:
            # Note tuple size 1 with one single string is not expected
            history.append({"role": "user", "content": input[0]})
        else:
            raise TypeError("input must be a string or a dict")

        if user:
            history[-1]["content"] = user + history[-1]['content']

        history.insert(0, {"role": "system",
                           "content": self._system + "\n" + instruction if instruction else self._system})

        return dict(messages=history, tools=tools) if tools else dict(messages=history)

    def pre_hook(self, func: Optional[Callable] = None):
        """Sets a pre-processing hook function, allowing external custom processing of input data before prompt generation.

Args:
    func (Optional[Callable]): A callable object to be used as the pre-processing hook function, which receives and processes input data.

**Returns:**

- LazyLLMPrompterBase: Returns the instance itself to support method chaining.
"""
        self._pre_hook = func
        return self

    def _split_instruction(self, instruction: str):
        system_instruction = instruction
        user_instruction = ""
        if LazyLLMPrompterBase.ISA in instruction and LazyLLMPrompterBase.ISE in instruction:
            # The instruction includes system prompts and/or user prompts
            pattern = re.compile(r"%s(.*)%s" % (LazyLLMPrompterBase.ISA, LazyLLMPrompterBase.ISE), re.DOTALL)
            ret = re.split(pattern, instruction)
            system_instruction = ret[0]
            user_instruction = ret[1]

        return system_instruction, user_instruction

    def generate_prompt(self, input: Union[str, List, Dict[str, str], None] = None,
                        history: List[Union[List[str], Dict[str, Any]]] = None,
                        tools: Union[List[Dict[str, Any]], None] = None,
                        label: Union[str, None] = None,
                        *, show: bool = False, return_dict: bool = False) -> Union[str, Dict]:
        """
Generate a corresponding Prompt based on user input.

Args:
    input (Option[str | Dict]): The input from the prompter, if it's a dict, it will be filled into the slots of the instruction; if it's a str, it will be used as input.
    history (Option[List[List | Dict]]): Historical conversation, can be ``[[u, s], [u, s]]`` or in openai's history format, defaults to None.
    tools (Option[List[Dict]]): A collection of tools that can be used, used when the large model performs FunctionCall, defaults to None.
    label (Option[str]): Label, used during fine-tuning or training, defaults to None.
    show (bool): Flag indicating whether to print the generated Prompt, defaults to False.
    return_dict (bool): Flag indicating whether to return a dict, generally set to True when using ``OnlineChatModule``. If returning a dict, only the ``instruction`` will be filled. Defaults to False.
"""
        input = copy.deepcopy(input)
        if self._pre_hook:
            input, history, tools, label = self._pre_hook(input, history, tools, label)
        instruction, input = self._get_instruction_and_input(input)
        history = self._get_histories(history, return_dict=return_dict)
        tools = self._get_tools(tools, return_dict=return_dict)
        self._check_values(instruction, input, history, tools)
        instruction, user_instruction = self._split_instruction(instruction)
        func = self._generate_prompt_dict_impl if return_dict else self._generate_prompt_impl
        result = func(instruction, input, user_instruction, history, tools, label)
        if self._show or show: LOG.info(result)
        return result

    def get_response(self, output: str, input: Union[str, None] = None) -> str:
        """Used to truncate the Prompt, keeping only valuable output.

Args:
        output (str): The output of the large model.
        input (Option[str]): The input of the large model. If this parameter is specified, any part of the output that includes the input will be completely truncated. Defaults to None.
"""
        if input and output.startswith(input):
            return output[len(input):]
        return output if getattr(self, "_split", None) is None else output.split(self._split)[-1]

generate_prompt(input=None, history=None, tools=None, label=None, *, show=False, return_dict=False)

Generate a corresponding Prompt based on user input.

Parameters:

  • input (Option[str | Dict], default: None ) –

    The input from the prompter, if it's a dict, it will be filled into the slots of the instruction; if it's a str, it will be used as input.

  • history (Option[List[List | Dict]], default: None ) –

    Historical conversation, can be [[u, s], [u, s]] or in openai's history format, defaults to None.

  • tools (Option[List[Dict]], default: None ) –

    A collection of tools that can be used, used when the large model performs FunctionCall, defaults to None.

  • label (Option[str], default: None ) –

    Label, used during fine-tuning or training, defaults to None.

  • show (bool, default: False ) –

    Flag indicating whether to print the generated Prompt, defaults to False.

  • return_dict (bool, default: False ) –

    Flag indicating whether to return a dict, generally set to True when using OnlineChatModule. If returning a dict, only the instruction will be filled. Defaults to False.

Source code in lazyllm/components/prompter/builtinPrompt.py 
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
    def generate_prompt(self, input: Union[str, List, Dict[str, str], None] = None,
                        history: List[Union[List[str], Dict[str, Any]]] = None,
                        tools: Union[List[Dict[str, Any]], None] = None,
                        label: Union[str, None] = None,
                        *, show: bool = False, return_dict: bool = False) -> Union[str, Dict]:
        """
Generate a corresponding Prompt based on user input.

Args:
    input (Option[str | Dict]): The input from the prompter, if it's a dict, it will be filled into the slots of the instruction; if it's a str, it will be used as input.
    history (Option[List[List | Dict]]): Historical conversation, can be ``[[u, s], [u, s]]`` or in openai's history format, defaults to None.
    tools (Option[List[Dict]]): A collection of tools that can be used, used when the large model performs FunctionCall, defaults to None.
    label (Option[str]): Label, used during fine-tuning or training, defaults to None.
    show (bool): Flag indicating whether to print the generated Prompt, defaults to False.
    return_dict (bool): Flag indicating whether to return a dict, generally set to True when using ``OnlineChatModule``. If returning a dict, only the ``instruction`` will be filled. Defaults to False.
"""
        input = copy.deepcopy(input)
        if self._pre_hook:
            input, history, tools, label = self._pre_hook(input, history, tools, label)
        instruction, input = self._get_instruction_and_input(input)
        history = self._get_histories(history, return_dict=return_dict)
        tools = self._get_tools(tools, return_dict=return_dict)
        self._check_values(instruction, input, history, tools)
        instruction, user_instruction = self._split_instruction(instruction)
        func = self._generate_prompt_dict_impl if return_dict else self._generate_prompt_impl
        result = func(instruction, input, user_instruction, history, tools, label)
        if self._show or show: LOG.info(result)
        return result

get_response(output, input=None)

Used to truncate the Prompt, keeping only valuable output.

Parameters:

  • output (str) –

    The output of the large model.

  • input (Option[str], default: None ) –

    The input of the large model. If this parameter is specified, any part of the output that includes the input will be completely truncated. Defaults to None.

Source code in lazyllm/components/prompter/builtinPrompt.py 
246
247
248
249
250
251
252
253
254
255
    def get_response(self, output: str, input: Union[str, None] = None) -> str:
        """Used to truncate the Prompt, keeping only valuable output.

Args:
        output (str): The output of the large model.
        input (Option[str]): The input of the large model. If this parameter is specified, any part of the output that includes the input will be completely truncated. Defaults to None.
"""
        if input and output.startswith(input):
            return output[len(input):]
        return output if getattr(self, "_split", None) is None else output.split(self._split)[-1]

pre_hook(func=None)

Sets a pre-processing hook function, allowing external custom processing of input data before prompt generation.

Parameters:

  • func (Optional[Callable], default: None ) –

    A callable object to be used as the pre-processing hook function, which receives and processes input data.

Returns:

  • LazyLLMPrompterBase: Returns the instance itself to support method chaining.
Source code in lazyllm/components/prompter/builtinPrompt.py 
192
193
194
195
196
197
198
199
200
201
202
203
    def pre_hook(self, func: Optional[Callable] = None):
        """Sets a pre-processing hook function, allowing external custom processing of input data before prompt generation.

Args:
    func (Optional[Callable]): A callable object to be used as the pre-processing hook function, which receives and processes input data.

**Returns:**

- LazyLLMPrompterBase: Returns the instance itself to support method chaining.
"""
        self._pre_hook = func
        return self

lazyllm.components.prompter.EmptyPrompter

Bases: LazyLLMPrompterBase

An empty prompt generator that inherits from LazyLLMPrompterBase, and directly returns the original input.

This class performs no formatting and is useful for debugging, testing, or as a placeholder.

Examples:

>>> from lazyllm.components.prompter import EmptyPrompter

>>> prompter = EmptyPrompter()

>>> prompter.generate_prompt("Hello LazyLLM")
'Hello LazyLLM'

>>> prompter.generate_prompt({"query": "Tell me a joke"})
{'query': 'Tell me a joke'}

>>> # Even with additional parameters, the input is returned unchanged
>>> prompter.generate_prompt("No-op", history=[["Hi", "Hello"]], tools=[{"name": "search"}], label="debug")
'No-op'
Source code in lazyllm/components/prompter/builtinPrompt.py 
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
class EmptyPrompter(LazyLLMPrompterBase):
    """An empty prompt generator that inherits from `LazyLLMPrompterBase`, and directly returns the original input.

This class performs no formatting and is useful for debugging, testing, or as a placeholder.


Examples:
    >>> from lazyllm.components.prompter import EmptyPrompter

    >>> prompter = EmptyPrompter()

    >>> prompter.generate_prompt("Hello LazyLLM")
    'Hello LazyLLM'

    >>> prompter.generate_prompt({"query": "Tell me a joke"})
    {'query': 'Tell me a joke'}

    >>> # Even with additional parameters, the input is returned unchanged
    >>> prompter.generate_prompt("No-op", history=[["Hi", "Hello"]], tools=[{"name": "search"}], label="debug")
    'No-op'
    """

    def generate_prompt(self, input, history=None, tools=None, label=None, show=False):
        """A prompt passthrough implementation that inherits from `LazyLLMPrompterBase`.

This method directly returns the input without any formatting. Useful for debugging, testing, or placeholder use.

Args:
    input (Any): The input to be returned directly as the prompt.
    history (Option[List[List | Dict]]): Dialogue history, ignored. Defaults to None.
    tools (Option[List[Dict]]): Tool definitions, ignored. Defaults to None.
    label (Option[str]): Label, ignored. Defaults to None.
    show (bool): Whether to print the returned prompt. Defaults to False.
"""
        if self._show or show: LOG.info(input)
        return input

generate_prompt(input, history=None, tools=None, label=None, show=False)

A prompt passthrough implementation that inherits from LazyLLMPrompterBase.

This method directly returns the input without any formatting. Useful for debugging, testing, or placeholder use.

Parameters:

  • input (Any) –

    The input to be returned directly as the prompt.

  • history (Option[List[List | Dict]], default: None ) –

    Dialogue history, ignored. Defaults to None.

  • tools (Option[List[Dict]], default: None ) –

    Tool definitions, ignored. Defaults to None.

  • label (Option[str], default: None ) –

    Label, ignored. Defaults to None.

  • show (bool, default: False ) –

    Whether to print the returned prompt. Defaults to False.

Source code in lazyllm/components/prompter/builtinPrompt.py 
279
280
281
282
283
284
285
286
287
288
289
290
291
292
    def generate_prompt(self, input, history=None, tools=None, label=None, show=False):
        """A prompt passthrough implementation that inherits from `LazyLLMPrompterBase`.

This method directly returns the input without any formatting. Useful for debugging, testing, or placeholder use.

Args:
    input (Any): The input to be returned directly as the prompt.
    history (Option[List[List | Dict]]): Dialogue history, ignored. Defaults to None.
    tools (Option[List[Dict]]): Tool definitions, ignored. Defaults to None.
    label (Option[str]): Label, ignored. Defaults to None.
    show (bool): Whether to print the returned prompt. Defaults to False.
"""
        if self._show or show: LOG.info(input)
        return input

lazyllm.components.Prompter

Bases: object

Prompt generator class for LLM input formatting. Supports template-based prompting, history injection, and response extraction.

This class allows prompts to be defined via string templates, loaded from dicts, files, or predefined names. It supports history-aware formatting for multi-turn conversations and adapts to both mapping and string input types.

Parameters:

  • prompt (Optional[str], default: None ) –

    Prompt template string with format placeholders.

  • response_split (Optional[str], default: None ) –

    Optional delimiter to split model response and extract useful output.

  • chat_prompt (Optional[str], default: None ) –

    Chat template string, must contain a history placeholder.

  • history_symbol (str, default: 'llm_chat_history' ) –

    Name of the placeholder for historical messages, default is 'llm_chat_history'.

  • eoa (Optional[str], default: None ) –

    Delimiter between assistant/user in history items.

  • eoh (Optional[str], default: None ) –

    Delimiter between user-assistant pairs.

  • show (bool, default: False ) –

    Whether to print the final prompt when generating. Default is False.

Examples:

>>> from lazyllm import Prompter

>>> p = Prompter(prompt="Answer the following: {question}")
>>> p.generate_prompt("What is AI?")
'Answer the following: What is AI?'

>>> p.generate_prompt({"question": "Define machine learning"})
'Answer the following: Define machine learning'

>>> p = Prompter(
...     prompt="Instruction: {instruction}",
...     chat_prompt="Instruction: {instruction}\nHistory:\n{llm_chat_history}",
...     history_symbol="llm_chat_history",
...     eoa="</s>",
...     eoh="|"
... )
>>> p.generate_prompt(
...     input={"instruction": "Translate this."},
...     history=[["hello", "你好"], ["how are you", "你好吗"]]
... )
'Instruction: Translate this.\nHistory:\nhello|你好</s>how are you|你好吗'

>>> prompt_conf = {
...     "prompt": "Task: {task}",
...     "response_split": "---"
... }
>>> p = Prompter.from_dict(prompt_conf)
>>> p.generate_prompt("Summarize this article.")
'Task: Summarize this article.'

>>> full_output = "Task: Summarize this article.---This is the summary."
>>> p.get_response(full_output)
'This is the summary.'
Source code in lazyllm/components/prompter/prompter.py 
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
class Prompter(object):
    """Prompt generator class for LLM input formatting. Supports template-based prompting, history injection, and response extraction.

This class allows prompts to be defined via string templates, loaded from dicts, files, or predefined names.
It supports history-aware formatting for multi-turn conversations and adapts to both mapping and string input types.

Args:
    prompt (Optional[str]): Prompt template string with format placeholders.
    response_split (Optional[str]): Optional delimiter to split model response and extract useful output.
    chat_prompt (Optional[str]): Chat template string, must contain a history placeholder.
    history_symbol (str): Name of the placeholder for historical messages, default is 'llm_chat_history'.
    eoa (Optional[str]): Delimiter between assistant/user in history items.
    eoh (Optional[str]): Delimiter between user-assistant pairs.
    show (bool): Whether to print the final prompt when generating. Default is False.


Examples:
    >>> from lazyllm import Prompter

    >>> p = Prompter(prompt="Answer the following: {question}")
    >>> p.generate_prompt("What is AI?")
    'Answer the following: What is AI?'

    >>> p.generate_prompt({"question": "Define machine learning"})
    'Answer the following: Define machine learning'

    >>> p = Prompter(
    ...     prompt="Instruction: {instruction}",
    ...     chat_prompt="Instruction: {instruction}\\nHistory:\\n{llm_chat_history}",
    ...     history_symbol="llm_chat_history",
    ...     eoa="</s>",
    ...     eoh="|"
    ... )
    >>> p.generate_prompt(
    ...     input={"instruction": "Translate this."},
    ...     history=[["hello", "你好"], ["how are you", "你好吗"]]
    ... )
    'Instruction: Translate this.\\nHistory:\\nhello|你好</s>how are you|你好吗'

    >>> prompt_conf = {
    ...     "prompt": "Task: {task}",
    ...     "response_split": "---"
    ... }
    >>> p = Prompter.from_dict(prompt_conf)
    >>> p.generate_prompt("Summarize this article.")
    'Task: Summarize this article.'

    >>> full_output = "Task: Summarize this article.---This is the summary."
    >>> p.get_response(full_output)
    'This is the summary.'
    """
    def __init__(self, prompt=None, response_split=None, *, chat_prompt=None,
                 history_symbol='llm_chat_history', eoa=None, eoh=None, show=False):
        self._prompt, self._response_split = prompt, response_split
        self._chat_prompt = chat_prompt
        self._history_symbol, self._eoa, self._eoh = history_symbol, eoa, eoh
        self._show = show
        self._prompt_keys = list(set(re.findall(r'\{(\w+)\}', self._prompt))) if prompt else []
        if chat_prompt is not None:
            chat_keys = set(re.findall(r'\{(\w+)\}', self._chat_prompt))
            assert set(self._prompt_keys).issubset(chat_keys)
            assert chat_keys - set(self._prompt_keys) == set([self._history_symbol])
            self.use_history = True
        else:
            self.use_history = history_symbol in self._prompt_keys
            if self.use_history:
                self._prompt_keys.pop(self._prompt_keys.index(history_symbol))
                self._chat_prompt = self._prompt

    @classmethod
    def from_dict(cls, prompt, *, show=False):
        """Initializes a Prompter instance from a prompt configuration dictionary.

Args:
    prompt (Dict): A dictionary containing prompt-related configuration. Must include 'prompt' key.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        assert isinstance(prompt, dict)
        return cls(**prompt, show=show)

    @classmethod
    def from_template(cls, template_name, *, show=False):
        """Loads prompt configuration from a template name and initializes a Prompter instance.

Args:
    template_name (str): Name of the template. Must exist in the `templates` dictionary.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        return cls.from_dict(templates[template_name], show=show)

    @classmethod
    def from_file(cls, fname, *, show=False):
        """Loads prompt configuration from a JSON file and initializes a Prompter instance.

Args:
    fname (str): Path to the JSON configuration file.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        with open(fname) as fp:
            return cls.from_dict(json.load(fp), show=show)

    @classmethod
    def empty(cls):
        """Creates an empty Prompter instance.

Returns:
    Prompter: A Prompter instance without any prompt configuration.
"""
        return cls()

    def _is_empty(self):
        return self._prompt is None

    def generate_prompt(self, input, history=None, tools=None, label=None, show=False):
        """Generates a formatted prompt string based on input and optional conversation history.

Args:
    input (Union[str, Dict]): User input. Can be a single string or a dictionary with multiple fields.
    history (Optional[List[List[str]]]): Multi-turn dialogue history, e.g., [['u1', 'a1'], ['u2', 'a2']].
    tools (Optional[Any]): Not supported. Must be None.
    label (Optional[str]): Optional label to append to the prompt, commonly used for training.
    show (bool): Whether to print the generated prompt. Defaults to False.

Returns:
    str: The final formatted prompt string.
"""
        if not self._is_empty():
            assert tools is None
            # datasets.formatting.formatting.LazyDict is used in transformers
            if not isinstance(input, collections.abc.Mapping):
                assert len(self._prompt_keys) == 1, (
                    f'invalid prompt `{self._prompt}` for <{type(input)}> input `{input}`')
                input = {self._prompt_keys[0]: input}
            try:
                if self.use_history and isinstance(history, list) and len(history) > 0:
                    assert isinstance(history[0], list), 'history must be list of list'
                    input[self._history_symbol] = self._eoa.join([self._eoh.join(h) for h in history])
                    input = self._chat_prompt.format(**input)
                else:
                    if self.use_history: input[self._history_symbol] = ''
                    input = self._prompt.format(**input)
            except Exception:
                raise RuntimeError(f'Generate prompt failed, and prompt is {self._prompt}; chat-prompt'
                                   f' is {self._chat_prompt}; input is {input}; history is {history}')
            if label: input += label
        if self._show or show: LOG.info(input)
        return input

    def get_response(self, response, input=None):
        """Extracts the actual model answer from the full response returned by an LLM.

Args:
    response (str): The full raw output from the model.
    input (Optional[str]): If the response starts with the input, that part will be removed.

Returns:
    str: The cleaned model response.
"""
        if input and response.startswith(input):
            return response[len(input):]
        return response if self._response_split is None else response.split(self._response_split)[-1]

from_dict(prompt, *, show=False) classmethod

Initializes a Prompter instance from a prompt configuration dictionary.

Parameters:

  • prompt (Dict) –

    A dictionary containing prompt-related configuration. Must include 'prompt' key.

  • show (bool, default: False ) –

    Whether to display the generated prompt. Defaults to False.

Returns:

  • Prompter –

    An initialized Prompter instance.

Source code in lazyllm/components/prompter/prompter.py 
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
    @classmethod
    def from_dict(cls, prompt, *, show=False):
        """Initializes a Prompter instance from a prompt configuration dictionary.

Args:
    prompt (Dict): A dictionary containing prompt-related configuration. Must include 'prompt' key.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        assert isinstance(prompt, dict)
        return cls(**prompt, show=show)

from_template(template_name, *, show=False) classmethod

Loads prompt configuration from a template name and initializes a Prompter instance.

Parameters:

  • template_name (str) –

    Name of the template. Must exist in the templates dictionary.

  • show (bool, default: False ) –

    Whether to display the generated prompt. Defaults to False.

Returns:

  • Prompter –

    An initialized Prompter instance.

Source code in lazyllm/components/prompter/prompter.py 
102
103
104
105
106
107
108
109
110
111
112
113
    @classmethod
    def from_template(cls, template_name, *, show=False):
        """Loads prompt configuration from a template name and initializes a Prompter instance.

Args:
    template_name (str): Name of the template. Must exist in the `templates` dictionary.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        return cls.from_dict(templates[template_name], show=show)

from_file(fname, *, show=False) classmethod

Loads prompt configuration from a JSON file and initializes a Prompter instance.

Parameters:

  • fname (str) –

    Path to the JSON configuration file.

  • show (bool, default: False ) –

    Whether to display the generated prompt. Defaults to False.

Returns:

  • Prompter –

    An initialized Prompter instance.

Source code in lazyllm/components/prompter/prompter.py 
115
116
117
118
119
120
121
122
123
124
125
126
127
    @classmethod
    def from_file(cls, fname, *, show=False):
        """Loads prompt configuration from a JSON file and initializes a Prompter instance.

Args:
    fname (str): Path to the JSON configuration file.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        with open(fname) as fp:
            return cls.from_dict(json.load(fp), show=show)

empty() classmethod

Creates an empty Prompter instance.

Returns:

  • Prompter –

    A Prompter instance without any prompt configuration.

Source code in lazyllm/components/prompter/prompter.py 
129
130
131
132
133
134
135
136
    @classmethod
    def empty(cls):
        """Creates an empty Prompter instance.

Returns:
    Prompter: A Prompter instance without any prompt configuration.
"""
        return cls()

generate_prompt(input, history=None, tools=None, label=None, show=False)

Generates a formatted prompt string based on input and optional conversation history.

Parameters:

  • input (Union[str, Dict]) –

    User input. Can be a single string or a dictionary with multiple fields.

  • history (Optional[List[List[str]]], default: None ) –

    Multi-turn dialogue history, e.g., [['u1', 'a1'], ['u2', 'a2']].

  • tools (Optional[Any], default: None ) –

    Not supported. Must be None.

  • label (Optional[str], default: None ) –

    Optional label to append to the prompt, commonly used for training.

  • show (bool, default: False ) –

    Whether to print the generated prompt. Defaults to False.

Returns:

  • str –

    The final formatted prompt string.

Source code in lazyllm/components/prompter/prompter.py 
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
    def generate_prompt(self, input, history=None, tools=None, label=None, show=False):
        """Generates a formatted prompt string based on input and optional conversation history.

Args:
    input (Union[str, Dict]): User input. Can be a single string or a dictionary with multiple fields.
    history (Optional[List[List[str]]]): Multi-turn dialogue history, e.g., [['u1', 'a1'], ['u2', 'a2']].
    tools (Optional[Any]): Not supported. Must be None.
    label (Optional[str]): Optional label to append to the prompt, commonly used for training.
    show (bool): Whether to print the generated prompt. Defaults to False.

Returns:
    str: The final formatted prompt string.
"""
        if not self._is_empty():
            assert tools is None
            # datasets.formatting.formatting.LazyDict is used in transformers
            if not isinstance(input, collections.abc.Mapping):
                assert len(self._prompt_keys) == 1, (
                    f'invalid prompt `{self._prompt}` for <{type(input)}> input `{input}`')
                input = {self._prompt_keys[0]: input}
            try:
                if self.use_history and isinstance(history, list) and len(history) > 0:
                    assert isinstance(history[0], list), 'history must be list of list'
                    input[self._history_symbol] = self._eoa.join([self._eoh.join(h) for h in history])
                    input = self._chat_prompt.format(**input)
                else:
                    if self.use_history: input[self._history_symbol] = ''
                    input = self._prompt.format(**input)
            except Exception:
                raise RuntimeError(f'Generate prompt failed, and prompt is {self._prompt}; chat-prompt'
                                   f' is {self._chat_prompt}; input is {input}; history is {history}')
            if label: input += label
        if self._show or show: LOG.info(input)
        return input

get_response(response, input=None)

Extracts the actual model answer from the full response returned by an LLM.

Parameters:

  • response (str) –

    The full raw output from the model.

  • input (Optional[str], default: None ) –

    If the response starts with the input, that part will be removed.

Returns:

  • str –

    The cleaned model response.

Source code in lazyllm/components/prompter/prompter.py 
176
177
178
179
180
181
182
183
184
185
186
187
188
    def get_response(self, response, input=None):
        """Extracts the actual model answer from the full response returned by an LLM.

Args:
    response (str): The full raw output from the model.
    input (Optional[str]): If the response starts with the input, that part will be removed.

Returns:
    str: The cleaned model response.
"""
        if input and response.startswith(input):
            return response[len(input):]
        return response if self._response_split is None else response.split(self._response_split)[-1]

options: heading_level: 3 inherited_members: - generate_prompt - get_response members: false

lazyllm.components.prompter.EmptyPrompter

Bases: LazyLLMPrompterBase

An empty prompt generator that inherits from LazyLLMPrompterBase, and directly returns the original input.

This class performs no formatting and is useful for debugging, testing, or as a placeholder.

Examples:

>>> from lazyllm.components.prompter import EmptyPrompter

>>> prompter = EmptyPrompter()

>>> prompter.generate_prompt("Hello LazyLLM")
'Hello LazyLLM'

>>> prompter.generate_prompt({"query": "Tell me a joke"})
{'query': 'Tell me a joke'}

>>> # Even with additional parameters, the input is returned unchanged
>>> prompter.generate_prompt("No-op", history=[["Hi", "Hello"]], tools=[{"name": "search"}], label="debug")
'No-op'
Source code in lazyllm/components/prompter/builtinPrompt.py 
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
class EmptyPrompter(LazyLLMPrompterBase):
    """An empty prompt generator that inherits from `LazyLLMPrompterBase`, and directly returns the original input.

This class performs no formatting and is useful for debugging, testing, or as a placeholder.


Examples:
    >>> from lazyllm.components.prompter import EmptyPrompter

    >>> prompter = EmptyPrompter()

    >>> prompter.generate_prompt("Hello LazyLLM")
    'Hello LazyLLM'

    >>> prompter.generate_prompt({"query": "Tell me a joke"})
    {'query': 'Tell me a joke'}

    >>> # Even with additional parameters, the input is returned unchanged
    >>> prompter.generate_prompt("No-op", history=[["Hi", "Hello"]], tools=[{"name": "search"}], label="debug")
    'No-op'
    """

    def generate_prompt(self, input, history=None, tools=None, label=None, show=False):
        """A prompt passthrough implementation that inherits from `LazyLLMPrompterBase`.

This method directly returns the input without any formatting. Useful for debugging, testing, or placeholder use.

Args:
    input (Any): The input to be returned directly as the prompt.
    history (Option[List[List | Dict]]): Dialogue history, ignored. Defaults to None.
    tools (Option[List[Dict]]): Tool definitions, ignored. Defaults to None.
    label (Option[str]): Label, ignored. Defaults to None.
    show (bool): Whether to print the returned prompt. Defaults to False.
"""
        if self._show or show: LOG.info(input)
        return input

generate_prompt(input, history=None, tools=None, label=None, show=False)

A prompt passthrough implementation that inherits from LazyLLMPrompterBase.

This method directly returns the input without any formatting. Useful for debugging, testing, or placeholder use.

Parameters:

  • input (Any) –

    The input to be returned directly as the prompt.

  • history (Option[List[List | Dict]], default: None ) –

    Dialogue history, ignored. Defaults to None.

  • tools (Option[List[Dict]], default: None ) –

    Tool definitions, ignored. Defaults to None.

  • label (Option[str], default: None ) –

    Label, ignored. Defaults to None.

  • show (bool, default: False ) –

    Whether to print the returned prompt. Defaults to False.

Source code in lazyllm/components/prompter/builtinPrompt.py 
279
280
281
282
283
284
285
286
287
288
289
290
291
292
    def generate_prompt(self, input, history=None, tools=None, label=None, show=False):
        """A prompt passthrough implementation that inherits from `LazyLLMPrompterBase`.

This method directly returns the input without any formatting. Useful for debugging, testing, or placeholder use.

Args:
    input (Any): The input to be returned directly as the prompt.
    history (Option[List[List | Dict]]): Dialogue history, ignored. Defaults to None.
    tools (Option[List[Dict]]): Tool definitions, ignored. Defaults to None.
    label (Option[str]): Label, ignored. Defaults to None.
    show (bool): Whether to print the returned prompt. Defaults to False.
"""
        if self._show or show: LOG.info(input)
        return input

lazyllm.components.Prompter

Bases: object

Prompt generator class for LLM input formatting. Supports template-based prompting, history injection, and response extraction.

This class allows prompts to be defined via string templates, loaded from dicts, files, or predefined names. It supports history-aware formatting for multi-turn conversations and adapts to both mapping and string input types.

Parameters:

  • prompt (Optional[str], default: None ) –

    Prompt template string with format placeholders.

  • response_split (Optional[str], default: None ) –

    Optional delimiter to split model response and extract useful output.

  • chat_prompt (Optional[str], default: None ) –

    Chat template string, must contain a history placeholder.

  • history_symbol (str, default: 'llm_chat_history' ) –

    Name of the placeholder for historical messages, default is 'llm_chat_history'.

  • eoa (Optional[str], default: None ) –

    Delimiter between assistant/user in history items.

  • eoh (Optional[str], default: None ) –

    Delimiter between user-assistant pairs.

  • show (bool, default: False ) –

    Whether to print the final prompt when generating. Default is False.

Examples:

>>> from lazyllm import Prompter

>>> p = Prompter(prompt="Answer the following: {question}")
>>> p.generate_prompt("What is AI?")
'Answer the following: What is AI?'

>>> p.generate_prompt({"question": "Define machine learning"})
'Answer the following: Define machine learning'

>>> p = Prompter(
...     prompt="Instruction: {instruction}",
...     chat_prompt="Instruction: {instruction}\nHistory:\n{llm_chat_history}",
...     history_symbol="llm_chat_history",
...     eoa="</s>",
...     eoh="|"
... )
>>> p.generate_prompt(
...     input={"instruction": "Translate this."},
...     history=[["hello", "你好"], ["how are you", "你好吗"]]
... )
'Instruction: Translate this.\nHistory:\nhello|你好</s>how are you|你好吗'

>>> prompt_conf = {
...     "prompt": "Task: {task}",
...     "response_split": "---"
... }
>>> p = Prompter.from_dict(prompt_conf)
>>> p.generate_prompt("Summarize this article.")
'Task: Summarize this article.'

>>> full_output = "Task: Summarize this article.---This is the summary."
>>> p.get_response(full_output)
'This is the summary.'
Source code in lazyllm/components/prompter/prompter.py 
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
class Prompter(object):
    """Prompt generator class for LLM input formatting. Supports template-based prompting, history injection, and response extraction.

This class allows prompts to be defined via string templates, loaded from dicts, files, or predefined names.
It supports history-aware formatting for multi-turn conversations and adapts to both mapping and string input types.

Args:
    prompt (Optional[str]): Prompt template string with format placeholders.
    response_split (Optional[str]): Optional delimiter to split model response and extract useful output.
    chat_prompt (Optional[str]): Chat template string, must contain a history placeholder.
    history_symbol (str): Name of the placeholder for historical messages, default is 'llm_chat_history'.
    eoa (Optional[str]): Delimiter between assistant/user in history items.
    eoh (Optional[str]): Delimiter between user-assistant pairs.
    show (bool): Whether to print the final prompt when generating. Default is False.


Examples:
    >>> from lazyllm import Prompter

    >>> p = Prompter(prompt="Answer the following: {question}")
    >>> p.generate_prompt("What is AI?")
    'Answer the following: What is AI?'

    >>> p.generate_prompt({"question": "Define machine learning"})
    'Answer the following: Define machine learning'

    >>> p = Prompter(
    ...     prompt="Instruction: {instruction}",
    ...     chat_prompt="Instruction: {instruction}\\nHistory:\\n{llm_chat_history}",
    ...     history_symbol="llm_chat_history",
    ...     eoa="</s>",
    ...     eoh="|"
    ... )
    >>> p.generate_prompt(
    ...     input={"instruction": "Translate this."},
    ...     history=[["hello", "你好"], ["how are you", "你好吗"]]
    ... )
    'Instruction: Translate this.\\nHistory:\\nhello|你好</s>how are you|你好吗'

    >>> prompt_conf = {
    ...     "prompt": "Task: {task}",
    ...     "response_split": "---"
    ... }
    >>> p = Prompter.from_dict(prompt_conf)
    >>> p.generate_prompt("Summarize this article.")
    'Task: Summarize this article.'

    >>> full_output = "Task: Summarize this article.---This is the summary."
    >>> p.get_response(full_output)
    'This is the summary.'
    """
    def __init__(self, prompt=None, response_split=None, *, chat_prompt=None,
                 history_symbol='llm_chat_history', eoa=None, eoh=None, show=False):
        self._prompt, self._response_split = prompt, response_split
        self._chat_prompt = chat_prompt
        self._history_symbol, self._eoa, self._eoh = history_symbol, eoa, eoh
        self._show = show
        self._prompt_keys = list(set(re.findall(r'\{(\w+)\}', self._prompt))) if prompt else []
        if chat_prompt is not None:
            chat_keys = set(re.findall(r'\{(\w+)\}', self._chat_prompt))
            assert set(self._prompt_keys).issubset(chat_keys)
            assert chat_keys - set(self._prompt_keys) == set([self._history_symbol])
            self.use_history = True
        else:
            self.use_history = history_symbol in self._prompt_keys
            if self.use_history:
                self._prompt_keys.pop(self._prompt_keys.index(history_symbol))
                self._chat_prompt = self._prompt

    @classmethod
    def from_dict(cls, prompt, *, show=False):
        """Initializes a Prompter instance from a prompt configuration dictionary.

Args:
    prompt (Dict): A dictionary containing prompt-related configuration. Must include 'prompt' key.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        assert isinstance(prompt, dict)
        return cls(**prompt, show=show)

    @classmethod
    def from_template(cls, template_name, *, show=False):
        """Loads prompt configuration from a template name and initializes a Prompter instance.

Args:
    template_name (str): Name of the template. Must exist in the `templates` dictionary.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        return cls.from_dict(templates[template_name], show=show)

    @classmethod
    def from_file(cls, fname, *, show=False):
        """Loads prompt configuration from a JSON file and initializes a Prompter instance.

Args:
    fname (str): Path to the JSON configuration file.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        with open(fname) as fp:
            return cls.from_dict(json.load(fp), show=show)

    @classmethod
    def empty(cls):
        """Creates an empty Prompter instance.

Returns:
    Prompter: A Prompter instance without any prompt configuration.
"""
        return cls()

    def _is_empty(self):
        return self._prompt is None

    def generate_prompt(self, input, history=None, tools=None, label=None, show=False):
        """Generates a formatted prompt string based on input and optional conversation history.

Args:
    input (Union[str, Dict]): User input. Can be a single string or a dictionary with multiple fields.
    history (Optional[List[List[str]]]): Multi-turn dialogue history, e.g., [['u1', 'a1'], ['u2', 'a2']].
    tools (Optional[Any]): Not supported. Must be None.
    label (Optional[str]): Optional label to append to the prompt, commonly used for training.
    show (bool): Whether to print the generated prompt. Defaults to False.

Returns:
    str: The final formatted prompt string.
"""
        if not self._is_empty():
            assert tools is None
            # datasets.formatting.formatting.LazyDict is used in transformers
            if not isinstance(input, collections.abc.Mapping):
                assert len(self._prompt_keys) == 1, (
                    f'invalid prompt `{self._prompt}` for <{type(input)}> input `{input}`')
                input = {self._prompt_keys[0]: input}
            try:
                if self.use_history and isinstance(history, list) and len(history) > 0:
                    assert isinstance(history[0], list), 'history must be list of list'
                    input[self._history_symbol] = self._eoa.join([self._eoh.join(h) for h in history])
                    input = self._chat_prompt.format(**input)
                else:
                    if self.use_history: input[self._history_symbol] = ''
                    input = self._prompt.format(**input)
            except Exception:
                raise RuntimeError(f'Generate prompt failed, and prompt is {self._prompt}; chat-prompt'
                                   f' is {self._chat_prompt}; input is {input}; history is {history}')
            if label: input += label
        if self._show or show: LOG.info(input)
        return input

    def get_response(self, response, input=None):
        """Extracts the actual model answer from the full response returned by an LLM.

Args:
    response (str): The full raw output from the model.
    input (Optional[str]): If the response starts with the input, that part will be removed.

Returns:
    str: The cleaned model response.
"""
        if input and response.startswith(input):
            return response[len(input):]
        return response if self._response_split is None else response.split(self._response_split)[-1]

from_dict(prompt, *, show=False) classmethod

Initializes a Prompter instance from a prompt configuration dictionary.

Parameters:

  • prompt (Dict) –

    A dictionary containing prompt-related configuration. Must include 'prompt' key.

  • show (bool, default: False ) –

    Whether to display the generated prompt. Defaults to False.

Returns:

  • Prompter –

    An initialized Prompter instance.

Source code in lazyllm/components/prompter/prompter.py 
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
    @classmethod
    def from_dict(cls, prompt, *, show=False):
        """Initializes a Prompter instance from a prompt configuration dictionary.

Args:
    prompt (Dict): A dictionary containing prompt-related configuration. Must include 'prompt' key.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        assert isinstance(prompt, dict)
        return cls(**prompt, show=show)

from_template(template_name, *, show=False) classmethod

Loads prompt configuration from a template name and initializes a Prompter instance.

Parameters:

  • template_name (str) –

    Name of the template. Must exist in the templates dictionary.

  • show (bool, default: False ) –

    Whether to display the generated prompt. Defaults to False.

Returns:

  • Prompter –

    An initialized Prompter instance.

Source code in lazyllm/components/prompter/prompter.py 
102
103
104
105
106
107
108
109
110
111
112
113
    @classmethod
    def from_template(cls, template_name, *, show=False):
        """Loads prompt configuration from a template name and initializes a Prompter instance.

Args:
    template_name (str): Name of the template. Must exist in the `templates` dictionary.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        return cls.from_dict(templates[template_name], show=show)

from_file(fname, *, show=False) classmethod

Loads prompt configuration from a JSON file and initializes a Prompter instance.

Parameters:

  • fname (str) –

    Path to the JSON configuration file.

  • show (bool, default: False ) –

    Whether to display the generated prompt. Defaults to False.

Returns:

  • Prompter –

    An initialized Prompter instance.

Source code in lazyllm/components/prompter/prompter.py 
115
116
117
118
119
120
121
122
123
124
125
126
127
    @classmethod
    def from_file(cls, fname, *, show=False):
        """Loads prompt configuration from a JSON file and initializes a Prompter instance.

Args:
    fname (str): Path to the JSON configuration file.
    show (bool): Whether to display the generated prompt. Defaults to False.

Returns:
    Prompter: An initialized Prompter instance.
"""
        with open(fname) as fp:
            return cls.from_dict(json.load(fp), show=show)

empty() classmethod

Creates an empty Prompter instance.

Returns:

  • Prompter –

    A Prompter instance without any prompt configuration.

Source code in lazyllm/components/prompter/prompter.py 
129
130
131
132
133
134
135
136
    @classmethod
    def empty(cls):
        """Creates an empty Prompter instance.

Returns:
    Prompter: A Prompter instance without any prompt configuration.
"""
        return cls()

generate_prompt(input, history=None, tools=None, label=None, show=False)

Generates a formatted prompt string based on input and optional conversation history.

Parameters:

  • input (Union[str, Dict]) –

    User input. Can be a single string or a dictionary with multiple fields.

  • history (Optional[List[List[str]]], default: None ) –

    Multi-turn dialogue history, e.g., [['u1', 'a1'], ['u2', 'a2']].

  • tools (Optional[Any], default: None ) –

    Not supported. Must be None.

  • label (Optional[str], default: None ) –

    Optional label to append to the prompt, commonly used for training.

  • show (bool, default: False ) –

    Whether to print the generated prompt. Defaults to False.

Returns:

  • str –

    The final formatted prompt string.

Source code in lazyllm/components/prompter/prompter.py 
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
    def generate_prompt(self, input, history=None, tools=None, label=None, show=False):
        """Generates a formatted prompt string based on input and optional conversation history.

Args:
    input (Union[str, Dict]): User input. Can be a single string or a dictionary with multiple fields.
    history (Optional[List[List[str]]]): Multi-turn dialogue history, e.g., [['u1', 'a1'], ['u2', 'a2']].
    tools (Optional[Any]): Not supported. Must be None.
    label (Optional[str]): Optional label to append to the prompt, commonly used for training.
    show (bool): Whether to print the generated prompt. Defaults to False.

Returns:
    str: The final formatted prompt string.
"""
        if not self._is_empty():
            assert tools is None
            # datasets.formatting.formatting.LazyDict is used in transformers
            if not isinstance(input, collections.abc.Mapping):
                assert len(self._prompt_keys) == 1, (
                    f'invalid prompt `{self._prompt}` for <{type(input)}> input `{input}`')
                input = {self._prompt_keys[0]: input}
            try:
                if self.use_history and isinstance(history, list) and len(history) > 0:
                    assert isinstance(history[0], list), 'history must be list of list'
                    input[self._history_symbol] = self._eoa.join([self._eoh.join(h) for h in history])
                    input = self._chat_prompt.format(**input)
                else:
                    if self.use_history: input[self._history_symbol] = ''
                    input = self._prompt.format(**input)
            except Exception:
                raise RuntimeError(f'Generate prompt failed, and prompt is {self._prompt}; chat-prompt'
                                   f' is {self._chat_prompt}; input is {input}; history is {history}')
            if label: input += label
        if self._show or show: LOG.info(input)
        return input

get_response(response, input=None)

Extracts the actual model answer from the full response returned by an LLM.

Parameters:

  • response (str) –

    The full raw output from the model.

  • input (Optional[str], default: None ) –

    If the response starts with the input, that part will be removed.

Returns:

  • str –

    The cleaned model response.

Source code in lazyllm/components/prompter/prompter.py 
176
177
178
179
180
181
182
183
184
185
186
187
188
    def get_response(self, response, input=None):
        """Extracts the actual model answer from the full response returned by an LLM.

Args:
    response (str): The full raw output from the model.
    input (Optional[str]): If the response starts with the input, that part will be removed.

Returns:
    str: The cleaned model response.
"""
        if input and response.startswith(input):
            return response[len(input):]
        return response if self._response_split is None else response.split(self._response_split)[-1]

lazyllm.components.AlpacaPrompter

Bases: LazyLLMPrompterBase

Alpaca-style Prompter, supports tool calls, does not support historical dialogue.

Parameters:

  • instruction (Option[str], default: None ) –

    Task instructions for the large model, with at least one fillable slot (e.g. {instruction}). Or use a dictionary to specify the system and user instructions.

  • extra_keys (Option[List], default: None ) –

    Additional fields that will be filled with user input.

  • show (bool, default: False ) –

    Flag indicating whether to print the generated Prompt, default is False.

  • tools (Option[list], default: None ) –

    Tool-set which is provived for LLMs, default is None.

Examples:

>>> from lazyllm import AlpacaPrompter
>>> p = AlpacaPrompter('hello world {instruction}')
>>> p.generate_prompt('this is my input')
'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello world this is my input\n\n\n### Response:\n'
>>> p.generate_prompt('this is my input', return_dict=True)
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello world this is my input\n\n'}, {'role': 'user', 'content': ''}]}
>>>
>>> p = AlpacaPrompter('hello world {instruction}, {input}', extra_keys=['knowledge'])
>>> p.generate_prompt(dict(instruction='hello world', input='my input', knowledge='lazyllm'))
'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello world hello world, my input\n\nHere are some extra messages you can referred to:\n\n### knowledge:\nlazyllm\n\n\n### Response:\n'
>>> p.generate_prompt(dict(instruction='hello world', input='my input', knowledge='lazyllm'), return_dict=True)
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello world hello world, my input\n\nHere are some extra messages you can referred to:\n\n### knowledge:\nlazyllm\n\n'}, {'role': 'user', 'content': ''}]}
>>>
>>> p = AlpacaPrompter(dict(system="hello world", user="this is user instruction {input}"))
>>> p.generate_prompt(dict(input="my input"))
'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello word\n\n\n\nthis is user instruction my input### Response:\n'
>>> p.generate_prompt(dict(input="my input"), return_dict=True)
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\n\n ### Instruction:\nhello world'}, {'role': 'user', 'content': 'this is user instruction my input'}]}
Source code in lazyllm/components/prompter/alpacaPrompter.py 
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
class AlpacaPrompter(LazyLLMPrompterBase):
    """Alpaca-style Prompter, supports tool calls, does not support historical dialogue.


Args:
    instruction (Option[str]): Task instructions for the large model, with at least one fillable slot (e.g. ``{instruction}``). Or use a dictionary to specify the ``system`` and ``user`` instructions.
    extra_keys (Option[List]): Additional fields that will be filled with user input.
    show (bool): Flag indicating whether to print the generated Prompt, default is False.
    tools (Option[list]): Tool-set which is provived for LLMs, default is None.


Examples:
    >>> from lazyllm import AlpacaPrompter
    >>> p = AlpacaPrompter('hello world {instruction}')
    >>> p.generate_prompt('this is my input')
    'You are an AI-Agent developed by LazyLLM.\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\n\\n ### Instruction:\\nhello world this is my input\\n\\n\\n### Response:\\n'
    >>> p.generate_prompt('this is my input', return_dict=True)
    {'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\n\\n ### Instruction:\\nhello world this is my input\\n\\n'}, {'role': 'user', 'content': ''}]}
    >>>
    >>> p = AlpacaPrompter('hello world {instruction}, {input}', extra_keys=['knowledge'])
    >>> p.generate_prompt(dict(instruction='hello world', input='my input', knowledge='lazyllm'))
    'You are an AI-Agent developed by LazyLLM.\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\n\\n ### Instruction:\\nhello world hello world, my input\\n\\nHere are some extra messages you can referred to:\\n\\n### knowledge:\\nlazyllm\\n\\n\\n### Response:\\n'
    >>> p.generate_prompt(dict(instruction='hello world', input='my input', knowledge='lazyllm'), return_dict=True)
    {'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\n\\n ### Instruction:\\nhello world hello world, my input\\n\\nHere are some extra messages you can referred to:\\n\\n### knowledge:\\nlazyllm\\n\\n'}, {'role': 'user', 'content': ''}]}
    >>>
    >>> p = AlpacaPrompter(dict(system="hello world", user="this is user instruction {input}"))
    >>> p.generate_prompt(dict(input="my input"))
    'You are an AI-Agent developed by LazyLLM.\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\n\\n ### Instruction:\\nhello word\\n\\n\\n\\nthis is user instruction my input### Response:\\n'
    >>> p.generate_prompt(dict(input="my input"), return_dict=True)
    {'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\nBelow is an instruction that describes a task, paired with extra messages such as input that provides further context if possible. Write a response that appropriately completes the request.\\n\\n ### Instruction:\\nhello world'}, {'role': 'user', 'content': 'this is user instruction my input'}]}

    """
    def __init__(self, instruction: Union[None, str, Dict[str, str]] = None, extra_keys: Union[None, List[str]] = None,
                 show: bool = False, tools: Optional[List] = None):
        super(__class__, self).__init__(show, tools=tools)
        if isinstance(instruction, dict):
            splice_struction = instruction.get("system", "") + \
                AlpacaPrompter.ISA + instruction.get("user", "") + AlpacaPrompter.ISE
            instruction = splice_struction
        instruction_template = ("Below is an instruction that describes a task, paired with extra messages such as "
                                "input that provides further context if possible. Write a response that appropriately "
                                f"completes the request.\n\n### Instruction:\n{instruction if instruction else ''}"
                                "\n\n" + LazyLLMPrompterBase._get_extro_key_template(extra_keys))
        self._init_prompt("{system}\n{instruction}\n{tools}\n{user}### Response:\n",
                          instruction_template,
                          "### Response:\n")

    def _check_values(self, instruction, input, history, tools):
        assert not history, f"Chat history is not supported in {__class__}."
        assert not input, "All keys should in instruction or extra-keys"

generate_prompt(input=None, history=None, tools=None, label=None, *, show=False, return_dict=False)

Generate a corresponding Prompt based on user input.

Parameters:

  • input (Option[str | Dict], default: None ) –

    The input from the prompter, if it's a dict, it will be filled into the slots of the instruction; if it's a str, it will be used as input.

  • history (Option[List[List | Dict]], default: None ) –

    Historical conversation, can be [[u, s], [u, s]] or in openai's history format, defaults to None.

  • tools (Option[List[Dict]], default: None ) –

    A collection of tools that can be used, used when the large model performs FunctionCall, defaults to None.

  • label (Option[str], default: None ) –

    Label, used during fine-tuning or training, defaults to None.

  • show (bool, default: False ) –

    Flag indicating whether to print the generated Prompt, defaults to False.

  • return_dict (bool, default: False ) –

    Flag indicating whether to return a dict, generally set to True when using OnlineChatModule. If returning a dict, only the instruction will be filled. Defaults to False.

Source code in lazyllm/components/prompter/builtinPrompt.py 
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
    def generate_prompt(self, input: Union[str, List, Dict[str, str], None] = None,
                        history: List[Union[List[str], Dict[str, Any]]] = None,
                        tools: Union[List[Dict[str, Any]], None] = None,
                        label: Union[str, None] = None,
                        *, show: bool = False, return_dict: bool = False) -> Union[str, Dict]:
        """
Generate a corresponding Prompt based on user input.

Args:
    input (Option[str | Dict]): The input from the prompter, if it's a dict, it will be filled into the slots of the instruction; if it's a str, it will be used as input.
    history (Option[List[List | Dict]]): Historical conversation, can be ``[[u, s], [u, s]]`` or in openai's history format, defaults to None.
    tools (Option[List[Dict]]): A collection of tools that can be used, used when the large model performs FunctionCall, defaults to None.
    label (Option[str]): Label, used during fine-tuning or training, defaults to None.
    show (bool): Flag indicating whether to print the generated Prompt, defaults to False.
    return_dict (bool): Flag indicating whether to return a dict, generally set to True when using ``OnlineChatModule``. If returning a dict, only the ``instruction`` will be filled. Defaults to False.
"""
        input = copy.deepcopy(input)
        if self._pre_hook:
            input, history, tools, label = self._pre_hook(input, history, tools, label)
        instruction, input = self._get_instruction_and_input(input)
        history = self._get_histories(history, return_dict=return_dict)
        tools = self._get_tools(tools, return_dict=return_dict)
        self._check_values(instruction, input, history, tools)
        instruction, user_instruction = self._split_instruction(instruction)
        func = self._generate_prompt_dict_impl if return_dict else self._generate_prompt_impl
        result = func(instruction, input, user_instruction, history, tools, label)
        if self._show or show: LOG.info(result)
        return result

get_response(output, input=None)

Used to truncate the Prompt, keeping only valuable output.

Parameters:

  • output (str) –

    The output of the large model.

  • input (Option[str], default: None ) –

    The input of the large model. If this parameter is specified, any part of the output that includes the input will be completely truncated. Defaults to None.

Source code in lazyllm/components/prompter/builtinPrompt.py 
246
247
248
249
250
251
252
253
254
255
    def get_response(self, output: str, input: Union[str, None] = None) -> str:
        """Used to truncate the Prompt, keeping only valuable output.

Args:
        output (str): The output of the large model.
        input (Option[str]): The input of the large model. If this parameter is specified, any part of the output that includes the input will be completely truncated. Defaults to None.
"""
        if input and output.startswith(input):
            return output[len(input):]
        return output if getattr(self, "_split", None) is None else output.split(self._split)[-1]

lazyllm.components.ChatPrompter

Bases: LazyLLMPrompterBase

Prompt constructor for multi-turn dialogue, inherits from LazyLLMPrompterBase.

Supports tool calling, conversation history, and customizable instruction templates. Accepts instructions as either plain string or dict with separate system and user components, automatically merging them into a unified prompt template. Also supports injecting extra user-defined fields.

Parameters:

  • instruction (Option[str | Dict[str, str]], default: None ) –

    The prompt instruction template. Can be a string or a dict with system and user keys. If a dict is given, the components will be merged using special delimiters.

  • extra_keys (Option[List[str]], default: None ) –

    A list of additional keys that will be filled by user input to enrich the prompt context.

  • show (bool, default: False ) –

    Whether to print the generated prompt. Default is False.

  • tools (Option[List], default: None ) –

    A list of tools available to the model for function-calling tasks. Default is None.

  • history (Option[List[List[str]]], default: None ) –

    Dialogue history in the format [[user, assistant], ...]. Used to provide conversational memory. Default is None.

Examples:

>>> from lazyllm import ChatPrompter
  • Simple instruction string
>>> p = ChatPrompter('hello world')
>>> p.generate_prompt('this is my input')
'You are an AI-Agent developed by LazyLLM.hello world\nthis is my input\n'

>>> p.generate_prompt('this is my input', return_dict=True)
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nhello world'}, {'role': 'user', 'content': 'this is my input'}]}
  • Using extra_keys
>>> p = ChatPrompter('hello world {instruction}', extra_keys=['knowledge'])
>>> p.generate_prompt({
...     'instruction': 'this is my ins',
...     'input': 'this is my inp',
...     'knowledge': 'LazyLLM-Knowledge'
... })
'You are an AI-Agent developed by LazyLLM.hello world this is my ins\nHere are some extra messages you can referred to:\n\n### knowledge:\nLazyLLM-Knowledge\nthis is my inp\n'
  • With conversation history
>>> p.generate_prompt({
...     'instruction': 'this is my ins',
...     'input': 'this is my inp',
...     'knowledge': 'LazyLLM-Knowledge'
... }, history=[['s1', 'e1'], ['s2', 'e2']])
'You are an AI-Agent developed by LazyLLM.hello world this is my ins\nHere are some extra messages you can referred to:\n\n### knowledge:\nLazyLLM-Knowledge\ns1|e1\ns2|e2\nthis is my inp\n'
  • Using dict format for system/user instructions
>>> p = ChatPrompter(dict(system="hello world", user="this is user instruction {input}"))
>>> p.generate_prompt({'input': "my input", 'query': "this is user query"})
'You are an AI-Agent developed by LazyLLM.hello world\nthis is user instruction my input this is user query\n'

>>> p.generate_prompt({'input': "my input", 'query': "this is user query"}, return_dict=True)
{'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\nhello world'}, {'role': 'user', 'content': 'this is user instruction my input this is user query'}]}
Source code in lazyllm/components/prompter/chatPrompter.py 
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
class ChatPrompter(LazyLLMPrompterBase):
    """Prompt constructor for multi-turn dialogue, inherits from `LazyLLMPrompterBase`.

Supports tool calling, conversation history, and customizable instruction templates. Accepts instructions as either plain string or dict with separate `system` and `user` components, automatically merging them into a unified prompt template. Also supports injecting extra user-defined fields.

Args:
    instruction (Option[str | Dict[str, str]]): The prompt instruction template. Can be a string or a dict with `system` and `user` keys. If a dict is given, the components will be merged using special delimiters.
    extra_keys (Option[List[str]]): A list of additional keys that will be filled by user input to enrich the prompt context.
    show (bool): Whether to print the generated prompt. Default is False.
    tools (Option[List]): A list of tools available to the model for function-calling tasks. Default is None.
    history (Option[List[List[str]]]): Dialogue history in the format [[user, assistant], ...]. Used to provide conversational memory. Default is None.


Examples:
    >>> from lazyllm import ChatPrompter

    - Simple instruction string
    >>> p = ChatPrompter('hello world')
    >>> p.generate_prompt('this is my input')
    'You are an AI-Agent developed by LazyLLM.hello world\\nthis is my input\\n'

    >>> p.generate_prompt('this is my input', return_dict=True)
    {'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\nhello world'}, {'role': 'user', 'content': 'this is my input'}]}

    - Using extra_keys
    >>> p = ChatPrompter('hello world {instruction}', extra_keys=['knowledge'])
    >>> p.generate_prompt({
    ...     'instruction': 'this is my ins',
    ...     'input': 'this is my inp',
    ...     'knowledge': 'LazyLLM-Knowledge'
    ... })
    'You are an AI-Agent developed by LazyLLM.hello world this is my ins\\nHere are some extra messages you can referred to:\\n\\n### knowledge:\\nLazyLLM-Knowledge\\nthis is my inp\\n'

    - With conversation history
    >>> p.generate_prompt({
    ...     'instruction': 'this is my ins',
    ...     'input': 'this is my inp',
    ...     'knowledge': 'LazyLLM-Knowledge'
    ... }, history=[['s1', 'e1'], ['s2', 'e2']])
    'You are an AI-Agent developed by LazyLLM.hello world this is my ins\\nHere are some extra messages you can referred to:\\n\\n### knowledge:\\nLazyLLM-Knowledge\\ns1|e1\\ns2|e2\\nthis is my inp\\n'

    - Using dict format for system/user instructions
    >>> p = ChatPrompter(dict(system="hello world", user="this is user instruction {input}"))
    >>> p.generate_prompt({'input': "my input", 'query': "this is user query"})
    'You are an AI-Agent developed by LazyLLM.hello world\\nthis is user instruction my input this is user query\\n'

    >>> p.generate_prompt({'input': "my input", 'query': "this is user query"}, return_dict=True)
    {'messages': [{'role': 'system', 'content': 'You are an AI-Agent developed by LazyLLM.\\nhello world'}, {'role': 'user', 'content': 'this is user instruction my input this is user query'}]}
    """
    def __init__(self, instruction: Union[None, str, Dict[str, str]] = None, extra_keys: Union[None, List[str]] = None,
                 show: bool = False, tools: Optional[List] = None, history: Optional[List[List[str]]] = None):
        super(__class__, self).__init__(show, tools=tools, history=history)
        if isinstance(instruction, dict):
            splice_instruction = instruction.get("system", "") + \
                ChatPrompter.ISA + instruction.get("user", "") + ChatPrompter.ISE
            instruction = splice_instruction
        instruction_template = f'{instruction}\n{{extra_keys}}\n'.replace(
            '{extra_keys}', LazyLLMPrompterBase._get_extro_key_template(extra_keys)) if instruction else ""
        self._init_prompt("{sos}{system}{instruction}{tools}{eos}\n\n{history}\n{soh}\n{user}{input}\n{eoh}{soa}\n",
                          instruction_template)

    @property
    def _split(self): return f'{self._soa}\n' if self._soa else None

generate_prompt(input=None, history=None, tools=None, label=None, *, show=False, return_dict=False)

Generate a corresponding Prompt based on user input.

Parameters:

  • input (Option[str | Dict], default: None ) –

    The input from the prompter, if it's a dict, it will be filled into the slots of the instruction; if it's a str, it will be used as input.

  • history (Option[List[List | Dict]], default: None ) –

    Historical conversation, can be [[u, s], [u, s]] or in openai's history format, defaults to None.

  • tools (Option[List[Dict]], default: None ) –

    A collection of tools that can be used, used when the large model performs FunctionCall, defaults to None.

  • label (Option[str], default: None ) –

    Label, used during fine-tuning or training, defaults to None.

  • show (bool, default: False ) –

    Flag indicating whether to print the generated Prompt, defaults to False.

  • return_dict (bool, default: False ) –

    Flag indicating whether to return a dict, generally set to True when using OnlineChatModule. If returning a dict, only the instruction will be filled. Defaults to False.

Source code in lazyllm/components/prompter/builtinPrompt.py 
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
    def generate_prompt(self, input: Union[str, List, Dict[str, str], None] = None,
                        history: List[Union[List[str], Dict[str, Any]]] = None,
                        tools: Union[List[Dict[str, Any]], None] = None,
                        label: Union[str, None] = None,
                        *, show: bool = False, return_dict: bool = False) -> Union[str, Dict]:
        """
Generate a corresponding Prompt based on user input.

Args:
    input (Option[str | Dict]): The input from the prompter, if it's a dict, it will be filled into the slots of the instruction; if it's a str, it will be used as input.
    history (Option[List[List | Dict]]): Historical conversation, can be ``[[u, s], [u, s]]`` or in openai's history format, defaults to None.
    tools (Option[List[Dict]]): A collection of tools that can be used, used when the large model performs FunctionCall, defaults to None.
    label (Option[str]): Label, used during fine-tuning or training, defaults to None.
    show (bool): Flag indicating whether to print the generated Prompt, defaults to False.
    return_dict (bool): Flag indicating whether to return a dict, generally set to True when using ``OnlineChatModule``. If returning a dict, only the ``instruction`` will be filled. Defaults to False.
"""
        input = copy.deepcopy(input)
        if self._pre_hook:
            input, history, tools, label = self._pre_hook(input, history, tools, label)
        instruction, input = self._get_instruction_and_input(input)
        history = self._get_histories(history, return_dict=return_dict)
        tools = self._get_tools(tools, return_dict=return_dict)
        self._check_values(instruction, input, history, tools)
        instruction, user_instruction = self._split_instruction(instruction)
        func = self._generate_prompt_dict_impl if return_dict else self._generate_prompt_impl
        result = func(instruction, input, user_instruction, history, tools, label)
        if self._show or show: LOG.info(result)
        return result

get_response(output, input=None)

Used to truncate the Prompt, keeping only valuable output.

Parameters:

  • output (str) –

    The output of the large model.

  • input (Option[str], default: None ) –

    The input of the large model. If this parameter is specified, any part of the output that includes the input will be completely truncated. Defaults to None.

Source code in lazyllm/components/prompter/builtinPrompt.py 
246
247
248
249
250
251
252
253
254
255
    def get_response(self, output: str, input: Union[str, None] = None) -> str:
        """Used to truncate the Prompt, keeping only valuable output.

Args:
        output (str): The output of the large model.
        input (Option[str]): The input of the large model. If this parameter is specified, any part of the output that includes the input will be completely truncated. Defaults to None.
"""
        if input and output.startswith(input):
            return output[len(input):]
        return output if getattr(self, "_split", None) is None else output.split(self._split)[-1]

MultiModal

Text to Image

lazyllm.components.StableDiffusionDeploy

Bases: LazyLLMDeployBase

Source code in lazyllm/components/deploy/stable_diffusion/stable_diffusion3.py 
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
class StableDiffusionDeploy(LazyLLMDeployBase):
    message_format = None
    keys_name_handle = None
    default_headers = {'Content-Type': 'application/json'}

    def __init__(self, launcher: Optional[LazyLLMLaunchersBase] = None,
                 log_path: Optional[str] = None, trust_remote_code: bool = True, port: Optional[int] = None):
        super().__init__(launcher=launcher)
        self._log_path = log_path
        self._trust_remote_code = trust_remote_code
        self._port = port

    def __call__(self, finetuned_model=None, base_model=None):
        if not finetuned_model:
            finetuned_model = base_model
        elif not os.path.exists(finetuned_model) or \
            not any(file.endswith(('.bin', '.safetensors'))
                    for _, _, filenames in os.walk(finetuned_model) for file in filenames):
            LOG.warning(f"Note! That finetuned_model({finetuned_model}) is an invalid path, "
                        f"base_model({base_model}) will be used")
            finetuned_model = base_model
        return lazyllm.deploy.RelayServer(port=self._port, func=_StableDiffusion3(finetuned_model),
                                          launcher=self._launcher, log_path=self._log_path, cls='stable_diffusion')()

Visual Question Answering

Reference LMDeploy, which supports the Visual Question Answering model.

Text to Sound

lazyllm.components.TTSDeploy

Source code in lazyllm/components/deploy/text_to_speech/__init__.py 
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
class TTSDeploy:

    def __new__(cls, name, **kwarg):
        return cls.get_deploy_cls(name)(**kwarg)

    @classmethod
    def get_deploy_cls(cls, name):
        name = name.lower()
        if name == 'bark':
            return BarkDeploy
        elif name in ('chattts', 'chattts-new'):
            return ChatTTSDeploy
        elif name.startswith('musicgen'):
            return MusicGenDeploy
        else:
            raise RuntimeError(f"Not support model: {name}")

lazyllm.components.ChatTTSDeploy

Bases: TTSBase

ChatTTS Model Deployment Class. This class is used to deploy the ChatTTS model to a specified server for network invocation.

__init__(self, launcher=None) Constructor, initializes the deployment class.

Parameters:

  • launcher (launcher, default: None ) –

    An instance of the launcher used to start the remote service.

__call__(self, finetuned_model=None, base_model=None) Deploys the model and returns the remote service address.

Parameters:

  • finetuned_model (str) –

    If provided, this model will be used for deployment; if not provided or the path is invalid, base_model will be used.

  • base_model (str) –

    The default model, which will be used for deployment if finetuned_model is invalid.

  • Return (str) –

    The URL address of the remote service.

Notes
  • Input for infer: str. The text corresponding to the audio to be generated.
  • Return of infer: The string encoded from the generated file paths, starting with the encoding flag "", followed by the serialized dictionary. The key files in the dictionary stores a list, with elements being the paths of the generated audio files.
  • Supported models: ChatTTS

Examples:

>>> from lazyllm import launchers, UrlModule
>>> from lazyllm.components import ChatTTSDeploy
>>> deployer = ChatTTSDeploy(launchers.remote())
>>> url = deployer(base_model='ChatTTS')
>>> model = UrlModule(url=url)
>>> res = model('Hello World!')
>>> print(res)
... <lazyllm-query>{"query": "", "files": ["path/to/chattts/sound_xxx.wav"]}
Source code in lazyllm/components/deploy/text_to_speech/chattts.py 
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
class ChatTTSDeploy(TTSBase):
    """ChatTTS Model Deployment Class. This class is used to deploy the ChatTTS model to a specified server for network invocation.

`__init__(self, launcher=None)`
Constructor, initializes the deployment class.

Args:
    launcher (lazyllm.launcher): An instance of the launcher used to start the remote service.

`__call__(self, finetuned_model=None, base_model=None)`
Deploys the model and returns the remote service address.

Args:
    finetuned_model (str): If provided, this model will be used for deployment; if not provided or the path is invalid, `base_model` will be used.
    base_model (str): The default model, which will be used for deployment if `finetuned_model` is invalid.
    Return (str): The URL address of the remote service.

Notes:
    - Input for infer: `str`.  The text corresponding to the audio to be generated.
    - Return of infer: The string encoded from the generated file paths, starting with the encoding flag "<lazyllm-query>", followed by the serialized dictionary. The key `files` in the dictionary stores a list, with elements being the paths of the generated audio files.
    - Supported models: [ChatTTS](https://huggingface.co/2Noise/ChatTTS)


Examples:
    >>> from lazyllm import launchers, UrlModule
    >>> from lazyllm.components import ChatTTSDeploy
    >>> deployer = ChatTTSDeploy(launchers.remote())
    >>> url = deployer(base_model='ChatTTS')
    >>> model = UrlModule(url=url)
    >>> res = model('Hello World!')
    >>> print(res)
    ... <lazyllm-query>{"query": "", "files": ["path/to/chattts/sound_xxx.wav"]}
    """
    keys_name_handle = {
        'inputs': 'inputs',
    }
    message_format = {
        'inputs': 'Who are you ?',
        'refinetext': {
            'prompt': "[oral_2][laugh_0][break_6]",
            'top_P': 0.7,
            'top_K': 20,
            'temperature': 0.7,
            'repetition_penalty': 1.0,
            'max_new_token': 384,
            'min_new_token': 0,
            'show_tqdm': True,
            'ensure_non_empty': True,
        },
        'infercode': {
            'prompt': "[speed_5]",
            'spk_emb': None,
            'temperature': 0.3,
            'repetition_penalty': 1.05,
            'max_new_token': 2048,
        }

    }
    default_headers = {'Content-Type': 'application/json'}
    func = _ChatTTSModule

lazyllm.components.BarkDeploy

Bases: TTSBase

Bark Model Deployment Class. This class is used to deploy the Bark model to a specified server for network invocation.

__init__(self, launcher=None) Constructor, initializes the deployment class.

Parameters:

  • launcher (launcher, default: None ) –

    An instance of the launcher used to start the remote service.

__call__(self, finetuned_model=None, base_model=None) Deploys the model and returns the remote service address.

Parameters:

  • finetuned_model (str) –

    If provided, this model will be used for deployment; if not provided or the path is invalid, base_model will be used.

  • base_model (str) –

    The default model, which will be used for deployment if finetuned_model is invalid.

  • Return (str) –

    The URL address of the remote service.

Notes
  • Input for infer: str. The text corresponding to the audio to be generated.
  • Return of infer: The string encoded from the generated file paths, starting with the encoding flag "", followed by the serialized dictionary. The key files in the dictionary stores a list, with elements being the paths of the generated audio files.
  • Supported models: bark

Examples:

>>> from lazyllm import launchers, UrlModule
>>> from lazyllm.components import BarkDeploy
>>> deployer = BarkDeploy(launchers.remote())
>>> url = deployer(base_model='bark')
>>> model = UrlModule(url=url)
>>> res = model('Hello World!')
>>> print(res)
... <lazyllm-query>{"query": "", "files": ["path/to/bark/sound_xxx.wav"]}
Source code in lazyllm/components/deploy/text_to_speech/bark.py 
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
class BarkDeploy(TTSBase):
    """Bark Model Deployment Class. This class is used to deploy the Bark model to a specified server for network invocation.

`__init__(self, launcher=None)`
Constructor, initializes the deployment class.

Args:
    launcher (lazyllm.launcher): An instance of the launcher used to start the remote service.

`__call__(self, finetuned_model=None, base_model=None)`
Deploys the model and returns the remote service address.

Args:
    finetuned_model (str): If provided, this model will be used for deployment; if not provided or the path is invalid, `base_model` will be used.
    base_model (str): The default model, which will be used for deployment if `finetuned_model` is invalid.
    Return (str): The URL address of the remote service.

Notes:
    - Input for infer: `str`.  The text corresponding to the audio to be generated.
    - Return of infer: The string encoded from the generated file paths, starting with the encoding flag "<lazyllm-query>", followed by the serialized dictionary. The key `files` in the dictionary stores a list, with elements being the paths of the generated audio files.
    - Supported models: [bark](https://huggingface.co/suno/bark)


Examples:
    >>> from lazyllm import launchers, UrlModule
    >>> from lazyllm.components import BarkDeploy
    >>> deployer = BarkDeploy(launchers.remote())
    >>> url = deployer(base_model='bark')
    >>> model = UrlModule(url=url)
    >>> res = model('Hello World!')
    >>> print(res)
    ... <lazyllm-query>{"query": "", "files": ["path/to/bark/sound_xxx.wav"]}
    """
    keys_name_handle = {
        'inputs': 'inputs',
    }
    message_format = {
        'inputs': 'Who are you ?',
        'voice_preset': None,
    }
    default_headers = {'Content-Type': 'application/json'}

    func = _Bark

lazyllm.components.MusicGenDeploy

Bases: TTSBase

MusicGen Model Deployment Class. This class is used to deploy the MusicGen model to a specified server for network invocation.

__init__(self, launcher=None) Constructor, initializes the deployment class.

Parameters:

  • launcher (launcher, default: None ) –

    An instance of the launcher used to start the remote service.

__call__(self, finetuned_model=None, base_model=None) Deploys the model and returns the remote service address.

Parameters:

  • finetuned_model (str) –

    If provided, this model will be used for deployment; if not provided or the path is invalid, base_model will be used.

  • base_model (str) –

    The default model, which will be used for deployment if finetuned_model is invalid.

  • Return (str) –

    The URL address of the remote service.

Notes
  • Input for infer: str. The text corresponding to the audio to be generated.
  • Return of infer: The string encoded from the generated file paths, starting with the encoding flag "", followed by the serialized dictionary. The key files in the dictionary stores a list, with elements being the paths of the generated audio files.
  • Supported models: musicgen-small

Examples:

>>> from lazyllm import launchers, UrlModule
>>> from lazyllm.components import MusicGenDeploy
>>> deployer = MusicGenDeploy(launchers.remote())
>>> url = deployer(base_model='musicgen-small')
>>> model = UrlModule(url=url)
>>> model('Symphony with flute as the main melody')
... <lazyllm-query>{"query": "", "files": ["path/to/musicgen/sound_xxx.wav"]}
Source code in lazyllm/components/deploy/text_to_speech/musicgen.py 
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
class MusicGenDeploy(TTSBase):
    """MusicGen Model Deployment Class. This class is used to deploy the MusicGen model to a specified server for network invocation.

`__init__(self, launcher=None)`
Constructor, initializes the deployment class.

Args:
    launcher (lazyllm.launcher): An instance of the launcher used to start the remote service.

`__call__(self, finetuned_model=None, base_model=None)`
Deploys the model and returns the remote service address.

Args:
    finetuned_model (str): If provided, this model will be used for deployment; if not provided or the path is invalid, `base_model` will be used.
    base_model (str): The default model, which will be used for deployment if `finetuned_model` is invalid.
    Return (str): The URL address of the remote service.

Notes:
    - Input for infer: `str`.  The text corresponding to the audio to be generated.
    - Return of infer: The string encoded from the generated file paths, starting with the encoding flag "<lazyllm-query>", followed by the serialized dictionary. The key `files` in the dictionary stores a list, with elements being the paths of the generated audio files.
    - Supported models: [musicgen-small](https://huggingface.co/facebook/musicgen-small)


Examples:
    >>> from lazyllm import launchers, UrlModule
    >>> from lazyllm.components import MusicGenDeploy
    >>> deployer = MusicGenDeploy(launchers.remote())
    >>> url = deployer(base_model='musicgen-small')
    >>> model = UrlModule(url=url)
    >>> model('Symphony with flute as the main melody')
    ... <lazyllm-query>{"query": "", "files": ["path/to/musicgen/sound_xxx.wav"]}
    """
    message_format = None
    keys_name_handle = None
    default_headers = {'Content-Type': 'application/json'}
    func = _MusicGen

Speech to Text

lazyllm.components.SenseVoiceDeploy

Bases: LazyLLMDeployBase

SenseVoice Model Deployment Class. This class is used to deploy the SenseVoice model to a specified server for network invocation.

__init__(self, launcher=None) Constructor, initializes the deployment class.

Parameters:

  • launcher (launcher, default: None ) –

    An instance of the launcher used to start the remote service.

__call__(self, finetuned_model=None, base_model=None) Deploys the model and returns the remote service address.

Parameters:

  • finetuned_model (str) –

    If provided, this model will be used for deployment; if not provided or the path is invalid, base_model will be used.

  • base_model (str) –

    The default model, which will be used for deployment if finetuned_model is invalid.

  • Return (str) –

    The URL address of the remote service.

Notes
  • Input for infer: str. The audio path or link.
  • Return of infer: str. The recognized content.
  • Supported models: SenseVoiceSmall

Examples:

>>> import os
>>> import lazyllm
>>> from lazyllm import launchers, UrlModule
>>> from lazyllm.components import SenseVoiceDeploy
>>> deployer = SenseVoiceDeploy(launchers.remote())
>>> url = deployer(base_model='SenseVoiceSmall')
>>> model = UrlModule(url=url)
>>> model('path/to/audio') # support format: .mp3, .wav
... xxxxxxxxxxxxxxxx
Source code in lazyllm/components/deploy/speech_to_text/sense_voice.py 
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
class SenseVoiceDeploy(LazyLLMDeployBase):
    """SenseVoice Model Deployment Class. This class is used to deploy the SenseVoice model to a specified server for network invocation.

`__init__(self, launcher=None)`
Constructor, initializes the deployment class.

Args:
    launcher (lazyllm.launcher): An instance of the launcher used to start the remote service.

`__call__(self, finetuned_model=None, base_model=None)`
Deploys the model and returns the remote service address.

Args:
    finetuned_model (str): If provided, this model will be used for deployment; if not provided or the path is invalid, `base_model` will be used.
    base_model (str): The default model, which will be used for deployment if `finetuned_model` is invalid.
    Return (str): The URL address of the remote service.

Notes:
    - Input for infer: `str`. The audio path or link.
    - Return of infer: `str`. The recognized content.
    - Supported models: [SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall)


Examples:
    >>> import os
    >>> import lazyllm
    >>> from lazyllm import launchers, UrlModule
    >>> from lazyllm.components import SenseVoiceDeploy
    >>> deployer = SenseVoiceDeploy(launchers.remote())
    >>> url = deployer(base_model='SenseVoiceSmall')
    >>> model = UrlModule(url=url)
    >>> model('path/to/audio') # support format: .mp3, .wav
    ... xxxxxxxxxxxxxxxx
    """
    keys_name_handle = {
        'inputs': 'inputs',
        'audio': 'audio',
    }
    message_format = {
        'inputs': 'Who are you ?',
        'audio': None,
    }
    default_headers = {'Content-Type': 'application/json'}

    def __init__(self, launcher: Optional[LazyLLMLaunchersBase] = None,
                 log_path: Optional[str] = None, trust_remote_code: bool = True, port: Optional[int] = None):
        super().__init__(launcher=launcher)
        self._log_path = log_path
        self._trust_remote_code = trust_remote_code
        self._port = port

    def __call__(self, finetuned_model=None, base_model=None):
        if not finetuned_model:
            finetuned_model = base_model
        elif not os.path.exists(finetuned_model) or \
            not any(file.endswith(('.pt', '.bin', '.safetensors'))
                    for _, _, filenames in os.walk(finetuned_model) for file in filenames):
            LOG.warning(f"Note! That finetuned_model({finetuned_model}) is an invalid path, "
                        f"base_model({base_model}) will be used")
            finetuned_model = base_model
        return lazyllm.deploy.RelayServer(port=self._port, func=SenseVoice(finetuned_model), launcher=self._launcher,
                                          log_path=self._log_path, cls='sensevoice')()

lazyllm.components.deploy.speech_to_text.sense_voice.SenseVoice

Bases: object

SenseVoice(base_path, source=None, init=False)

A speech-to-text wrapper using FunASR models for lazy initialization and audio transcription. This class supports automatic model downloading, safe initialization, and inference from audio paths or URLs.

Parameters: - base_path (str): Path or model identifier to download the STT model. - source (str, optional): Model source name; defaults to lazyllm.config['model_source']. - init (bool): Whether to initialize the model immediately on creation.

Attributes: - base_path (str): Final resolved path of the model after download. - model: Loaded FunASR model instance. - init_flag: A lazy flag used to ensure model is only loaded once.

  • call(string: str | dict) -> str: Transcribes the input audio file or URL to text. Accepts base64-encoded content, file paths, or URLs.
  • load_stt(): Loads the FunASR speech-to-text model and related VAD (Voice Activity Detection).
  • rebuild(base_path, init): Rebuilds the class instance (used for serialization).
  • reduce(): Supports pickling by ensuring proper lazy-loading on deserialization.
Source code in lazyllm/components/deploy/speech_to_text/sense_voice.py 
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
class SenseVoice(object):
    """SenseVoice(base_path, source=None, init=False)

A speech-to-text wrapper using FunASR models for lazy initialization and audio transcription.
This class supports automatic model downloading, safe initialization, and inference from audio paths or URLs.

Parameters:
- base_path (str): Path or model identifier to download the STT model.
- source (str, optional): Model source name; defaults to `lazyllm.config['model_source']`.
- init (bool): Whether to initialize the model immediately on creation.

Attributes:
- base_path (str): Final resolved path of the model after download.
- model: Loaded FunASR model instance.
- init_flag: A lazy flag used to ensure model is only loaded once.

Methods:
- __call__(string: str | dict) -> str:
    Transcribes the input audio file or URL to text. Accepts base64-encoded content, file paths, or URLs.
- load_stt():
    Loads the FunASR speech-to-text model and related VAD (Voice Activity Detection).
- rebuild(base_path, init):
    Rebuilds the class instance (used for serialization).
- __reduce__():
    Supports pickling by ensuring proper lazy-loading on deserialization.
"""
    def __init__(self, base_path, source=None, init=False):
        source = lazyllm.config['model_source'] if not source else source
        self.base_path = ModelManager(source).download(base_path) or ''
        self.model = None
        self.init_flag = lazyllm.once_flag()
        if init:
            lazyllm.call_once(self.init_flag, self.load_stt)

    def load_stt(self):
        """load_stt()

Loads the speech-to-text model using FunASR with optional support for Huawei NPU via `torch_npu`.

The method initializes the model with the following characteristics:
- Uses `fsmn-vad` for voice activity detection with long utterance support.
- Sets maximum single segment time to 30 seconds.
- Selects `cuda:0` as the default inference device.

The model is stored in `self.model` and will be used to transcribe audio input.

Note:
If `torch_npu` is available in the environment, the function attempts to load it for potential Huawei Ascend acceleration.
"""
        if importlib.util.find_spec("torch_npu") is not None:
            import torch_npu  # noqa F401
            from torch_npu.contrib import transfer_to_npu  # noqa F401

        self.model = funasr.AutoModel(
            model=self.base_path,
            trust_remote_code=False,
            vad_model="fsmn-vad",
            vad_kwargs={"max_single_segment_time": 30000},
            device="cuda:0",
        )

    def __call__(self, string):
        lazyllm.call_once(self.init_flag, self.load_stt)
        if isinstance(string, dict):
            if string['audio']:
                string = string['audio'][-1] if isinstance(string['audio'], list) else string['audio']
            else:
                string = string['inputs']
        assert isinstance(string, str)
        string = string.strip()
        try:
            string = _base64_to_file(string) if _is_base64_with_mime(string) else string
        except Exception as e:
            LOG.error(f"Error processing base64 encoding: {e}")
            return f"Error processing base64 encoding {e}"
        if not string.endswith(supported_formats):
            return f"Only {', '.join(supported_formats)} formats in the form of file paths or URLs are supported."
        if not is_valid_path(string) and not is_valid_url(string):
            return f"This {string} is not a valid URL or file path. Please check."
        res = self.model.generate(
            input=string,
            cache={},
            language="auto",  # "zn", "en", "yue", "ja", "ko", "nospeech"
            use_itn=True,
            batch_size_s=60,
            merge_vad=True,
            merge_length_s=15,
        )
        text = funasr.utils.postprocess_utils.rich_transcription_postprocess(res[0]["text"])
        return text

    @classmethod
    def rebuild(cls, base_path, init):
        """rebuild(base_path: str, init: bool) -> SenseVoice

Class method used to reconstruct a `SenseVoice` instance during deserialization (e.g., when using `cloudpickle`).

Parameters:
- base_path (str): Path to the speech-to-text model.
- init (bool): Whether to immediately initialize and load the model upon creation.

Returns:
- A new instance of `SenseVoice` with the specified configuration.

Note:
This method is internally used to support model serialization and multiprocessing compatibility.
"""
        return cls(base_path, init=init)

    def __reduce__(self):
        init = bool(os.getenv('LAZYLLM_ON_CLOUDPICKLE', None) == 'ON' or self.init_flag)
        return SenseVoice.rebuild, (self.base_path, init)
load_stt()

load_stt()

Loads the speech-to-text model using FunASR with optional support for Huawei NPU via torch_npu.

The method initializes the model with the following characteristics: - Uses fsmn-vad for voice activity detection with long utterance support. - Sets maximum single segment time to 30 seconds. - Selects cuda:0 as the default inference device.

The model is stored in self.model and will be used to transcribe audio input.

Note: If torch_npu is available in the environment, the function attempts to load it for potential Huawei Ascend acceleration.

Source code in lazyllm/components/deploy/speech_to_text/sense_voice.py 
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
    def load_stt(self):
        """load_stt()

Loads the speech-to-text model using FunASR with optional support for Huawei NPU via `torch_npu`.

The method initializes the model with the following characteristics:
- Uses `fsmn-vad` for voice activity detection with long utterance support.
- Sets maximum single segment time to 30 seconds.
- Selects `cuda:0` as the default inference device.

The model is stored in `self.model` and will be used to transcribe audio input.

Note:
If `torch_npu` is available in the environment, the function attempts to load it for potential Huawei Ascend acceleration.
"""
        if importlib.util.find_spec("torch_npu") is not None:
            import torch_npu  # noqa F401
            from torch_npu.contrib import transfer_to_npu  # noqa F401

        self.model = funasr.AutoModel(
            model=self.base_path,
            trust_remote_code=False,
            vad_model="fsmn-vad",
            vad_kwargs={"max_single_segment_time": 30000},
            device="cuda:0",
        )
rebuild(base_path, init) classmethod

rebuild(base_path: str, init: bool) -> SenseVoice

Class method used to reconstruct a SenseVoice instance during deserialization (e.g., when using cloudpickle).

Parameters: - base_path (str): Path to the speech-to-text model. - init (bool): Whether to immediately initialize and load the model upon creation.

Returns: - A new instance of SenseVoice with the specified configuration.

Note: This method is internally used to support model serialization and multiprocessing compatibility.

Source code in lazyllm/components/deploy/speech_to_text/sense_voice.py 
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
    @classmethod
    def rebuild(cls, base_path, init):
        """rebuild(base_path: str, init: bool) -> SenseVoice

Class method used to reconstruct a `SenseVoice` instance during deserialization (e.g., when using `cloudpickle`).

Parameters:
- base_path (str): Path to the speech-to-text model.
- init (bool): Whether to immediately initialize and load the model upon creation.

Returns:
- A new instance of `SenseVoice` with the specified configuration.

Note:
This method is internally used to support model serialization and multiprocessing compatibility.
"""
        return cls(base_path, init=init)

ModelManager

lazyllm.components.ModelManager

ModelManager is a utility class provided by LazyLLM for developers to automatically download models. Currently, it supports search for models from local directories, as well as automatically downloading model from huggingface or modelscope. Before using ModelManager, the following environment variables need to be set:

  • LAZYLLM_MODEL_SOURCE: The source for model downloads, which can be set to huggingface or modelscope .
  • LAZYLLM_MODEL_SOURCE_TOKEN: The token provided by huggingface or modelscope for private model download.
  • LAZYLLM_MODEL_PATH: A colon-separated : list of local absolute paths for model search.
  • LAZYLLM_MODEL_CACHE_DIR: Directory for downloaded models.

Other Parameters:

  • model_source (str) –

    The source for model downloads, currently only supports huggingface or modelscope . If necessary, ModelManager downloads model data from the source. If not provided, LAZYLLM_MODEL_SOURCE environment variable would be used, and if LAZYLLM_MODEL_SOURCE is not set, ModelManager will not download any model.

  • token (str) –

    The token provided by huggingface or modelscope . If the token is present, ModelManager uses the token to download model. If not provided, LAZYLLM_MODEL_SOURCE_TOKEN environment variable would be used. and if LAZYLLM_MODEL_SOURCE_TOKEN is not set, ModelManager will not download private models, only public ones.

  • model_path (str) –

    A colon-separated list of absolute paths. Before actually start to download model, ModelManager trys to find the target model in the directories in this list. If not provided, LAZYLLM_MODEL_PATH environment variable would be used, and LAZYLLM_MODEL_PATH is not set, ModelManager skips looking for models from model_path.

  • cache_dir (str) –

    An absolute path of a directory to save downloaded models. If not provided, LAZYLLM_MODEL_CACHE_DIR environment variable would be used, and if LAZYLLM_MODEL_PATH is not set, the default value is ~/.lazyllm/model.

 ModelManager.download(model) -> str

Download models from model_source. The function first searches for the target model in directories listed in the model_path parameter of ModelManager class. If not found, it searches under cache_dir. If still not found, it downloads the model from model_source and stores it under cache_dir.

Parameters:

  • model (str) –

    The name of the target model. The function uses this name to download the model from model_source.

Examples:

>>> from lazyllm.components import ModelManager
>>> downloader = ModelManager(model_source='modelscope')
>>> downloader.download('chatglm3-6b')
Source code in lazyllm/components/utils/downloader/model_downloader.py 
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
class ModelManager():
    """ModelManager is a utility class provided by LazyLLM for developers to automatically download models.
Currently, it supports search for models from local directories, as well as automatically downloading model from
huggingface or modelscope. Before using ModelManager, the following environment variables need to be set:

- LAZYLLM_MODEL_SOURCE: The source for model downloads, which can be set to ``huggingface`` or ``modelscope`` .
- LAZYLLM_MODEL_SOURCE_TOKEN: The token provided by ``huggingface`` or ``modelscope`` for private model download.
- LAZYLLM_MODEL_PATH: A colon-separated ``:`` list of local absolute paths for model search.
- LAZYLLM_MODEL_CACHE_DIR: Directory for downloaded models.

Keyword Args: 
    model_source (str, optional): The source for model downloads, currently only supports ``huggingface`` or ``modelscope`` .
        If necessary, ModelManager downloads model data from the source. If not provided, LAZYLLM_MODEL_SOURCE
        environment variable would be used, and if LAZYLLM_MODEL_SOURCE is not set, ModelManager will not download
        any model.
    token (str, optional): The token provided by ``huggingface`` or ``modelscope`` . If the token is present, ModelManager uses
        the token to download model. If not provided, LAZYLLM_MODEL_SOURCE_TOKEN environment variable would be used.
        and if LAZYLLM_MODEL_SOURCE_TOKEN is not set, ModelManager will not download private models, only public ones.
    model_path (str, optional): A colon-separated list of absolute paths. Before actually start to download model,
        ModelManager trys to find the target model in the directories in this list. If not provided,
        LAZYLLM_MODEL_PATH environment variable would be used, and LAZYLLM_MODEL_PATH is not set, ModelManager skips
        looking for models from model_path.
    cache_dir (str, optional): An absolute path of a directory to save downloaded models. If not provided,
        LAZYLLM_MODEL_CACHE_DIR environment variable would be used, and if LAZYLLM_MODEL_PATH is not set, the default
        value is ~/.lazyllm/model.

<span style="font-size: 20px;">&ensp;**`ModelManager.download(model) -> str`**</span>

Download models from model_source. The function first searches for the target model in directories listed in the
model_path parameter of ModelManager class. If not found, it searches under cache_dir. If still not found,
it downloads the model from model_source and stores it under cache_dir.

Args:
    model (str): The name of the target model. The function uses this name to download the model from model_source.
    To further simplify use of the function, LazyLLM provides a mapping dict from abbreviated model names to original
    names on the download source for popular models, such as ``Llama-3-8B`` , ``GLM3-6B`` or ``Qwen1.5-7B``. For more details,
    please refer to the file ``lazyllm/module/utils/downloader/model_mapping.py`` . The model argument can be either
    an abbreviated name or one from the download source.


Examples:
    >>> from lazyllm.components import ModelManager
    >>> downloader = ModelManager(model_source='modelscope')
    >>> downloader.download('chatglm3-6b')
    """
    def __init__(self, model_source, token=lazyllm.config['model_source_token'],
                 cache_dir=lazyllm.config['model_cache_dir'], model_path=lazyllm.config['model_path']):
        self.model_source = model_source or lazyllm.config['model_source']
        self.token = token or None
        self.cache_dir = cache_dir
        self.model_paths = model_path.split(":") if len(model_path) > 0 else []
        if self.model_source == 'huggingface':
            self.hub_downloader = _HuggingfaceDownloader(token=self.token)
        else:
            self.hub_downloader = _ModelscopeDownloader(token=self.token)
            if self.model_source != 'modelscope':
                lazyllm.LOG.error("Only support Huggingface and Modelscope currently. "
                                  f"Unsupported model source: {self.model_source}. Forcing use of Modelscope.")

    @staticmethod
    @functools.lru_cache
    def get_model_type(model) -> str:
        assert isinstance(model, str) and len(model) > 0, f'model name should be a non-empty string, get {model}'
        __class__._try_add_mapping(model)
        for name, info in model_name_mapping.items():
            if 'type' not in info: continue

            model_name_set = {name.casefold()}
            for source in info['source']:
                model_name_set.add(info['source'][source].split('/')[-1].casefold())

            if model.split(os.sep)[-1].casefold() in model_name_set:
                return info['type']
        return 'llm'

    @staticmethod
    @functools.lru_cache
    def _get_model_name(model) -> str:
        search_string = os.path.basename(model)
        __class__._try_add_mapping(search_string)
        for model_name, sources in model_name_mapping.items():
            if model_name.lower() == search_string.lower() or any(
                    os.path.basename(source_file).lower() == search_string.lower()
                    for source_file in sources["source"].values()):
                return model_name
        return ""

    @staticmethod
    @functools.lru_cache
    def get_model_prompt_keys(model) -> dict:
        model_name = __class__._get_model_name(model)
        __class__._try_add_mapping(model_name)
        if model_name and "prompt_keys" in model_name_mapping[model_name.lower()]:
            return model_name_mapping[model_name.lower()]["prompt_keys"]
        else:
            return dict()

    @staticmethod
    def validate_model_path(model_path):
        extensions = {'.pt', '.bin', '.safetensors'}
        for _, _, files in os.walk(model_path):
            for file in files:
                if any(file.endswith(ext) for ext in extensions):
                    return True
        return False

    @staticmethod
    def _try_add_mapping(model):
        model_base = os.path.basename(model)
        model = model_base.lower()
        if model in model_name_mapping.keys():
            return
        matched_model_prefix = next((key for key in model_provider if model.startswith(key)), None)
        if matched_model_prefix:
            matching_keys = [key for key in model_groups.keys() if key in model]
            if matching_keys:
                matched_groups = max(matching_keys, key=len)
                model_name_mapping[model] = {
                    "prompt_keys": model_groups[matched_groups]["prompt_keys"],
                    "source": {k: v + '/' + model_base for k, v in model_provider[matched_model_prefix].items()}
                }

    def download(self, model='', call_back=None):
        assert isinstance(model, str), "model name should be a string."
        if len(model) == 0 or model[0] in (os.sep, '.', '~') or os.path.isabs(model): return model
        if (model_at_path := self._model_exists_at_path(model)): return model_at_path
        if self.model_source == '' or self.model_source not in ('huggingface', 'modelscope'):
            lazyllm.LOG.error("model automatic downloads only support Huggingface and Modelscope currently.")
            return model

        self._try_add_mapping(model)
        if model_name_mapping.get(model.lower(), {}).get('download_by_other'): return model

        if model.lower() in model_name_mapping.keys() and \
                self.model_source in model_name_mapping[model.lower()]['source'].keys():
            full_model_dir = os.path.join(self.cache_dir, model)

            mapped_model_name = model_name_mapping[model.lower()]['source'][self.model_source]
            model_save_dir = self._do_download(mapped_model_name, call_back)
            if model_save_dir:
                # The code safely creates a symbolic link by removing any existing target.
                if os.path.exists(full_model_dir):
                    os.remove(full_model_dir)
                if os.path.islink(full_model_dir):
                    os.unlink(full_model_dir)
                os.symlink(model_save_dir, full_model_dir, target_is_directory=True)
                return full_model_dir
            return model_save_dir  # return False
        else:
            model_name_for_download = model

            if '/' not in model_name_for_download:
                # Try to figure out a possible model provider
                matched_model_prefix = next((key for key in model_provider if model.lower().startswith(key)), None)
                if matched_model_prefix and self.model_source in model_provider[matched_model_prefix]:
                    model_name_for_download = model_provider[matched_model_prefix][self.model_source] + '/' + model

            model_save_dir = self._do_download(model_name_for_download, call_back)
            return model_save_dir

    def _validate_token(self):
        return self.hub_downloader.verify_hub_token()

    def _validate_model_id(self, model_id):
        return self.hub_downloader._verify_model_id(model_id)

    def _model_exists_at_path(self, model_name):
        if len(self.model_paths) == 0:
            return None
        model_dirs = []

        # For short model name, get all possible names from the mapping.
        if model_name.lower() in model_name_mapping.keys():
            for source in ('huggingface', 'modelscope'):
                if source in model_name_mapping[model_name.lower()]['source'].keys():
                    model_dirs.append(model_name_mapping[model_name.lower()]['source'][source].replace('/', os.sep))
        model_dirs.append(model_name.replace('/', os.sep))

        for model_path in self.model_paths:
            if len(model_path) == 0: continue
            if model_path[0] != os.sep:
                lazyllm.LOG.warning(f"skipping path {model_path} as only absolute paths is accepted.")
                continue
            for model_dir in model_dirs:
                full_model_dir = os.path.join(model_path, model_dir)
                if self._is_model_valid(full_model_dir):
                    return full_model_dir
        return None

    def _is_model_valid(self, model_dir):
        if not os.path.isdir(model_dir):
            return False
        return any((True for _ in os.scandir(model_dir)))

    def _do_download(self, model='', call_back=None):
        model_dir = model.replace('/', os.sep)
        full_model_dir = os.path.join(self.cache_dir, self.model_source, model_dir)

        try:
            return self.hub_downloader.download(model, full_model_dir, call_back)
        # Use `BaseException` to capture `KeyboardInterrupt` and normal `Exceptioin`.
        except BaseException as e:  # noqa B036
            lazyllm.LOG.warning(f"Download encountered an error: {e}")
            if not self.token and 'Permission denied' not in str(e):
                lazyllm.LOG.warning('Token is empty, which may prevent private models from being downloaded, '
                                    'as indicated by "the model does not exist." Please set the token with the '
                                    'environment variable LAZYLLM_MODEL_SOURCE_TOKEN to download private models.')
            if os.path.isdir(full_model_dir):
                shutil.rmtree(full_model_dir)
                lazyllm.LOG.warning(f"{full_model_dir} removed due to exceptions.")
        return False

Formatter

lazyllm.components.formatter.LazyLLMFormatterBase

This class is the base class of the formatter. The formatter is the formatter of the model output result. Users can customize the formatter or use the formatter provided by LazyLLM. Main methods: _parse_formatter: parse the index content. _load: Parse the str object, and the part containing Python objects is parsed out, such as list, dict and other objects. _parse_py_data_by_formatter: format the python object according to the custom formatter and index. format: format the passed content. If the content is a string type, convert the string into a python object first, and then format it. If the content is a python object, format it directly.

Examples:

>>> from lazyllm.components.formatter import FormatterBase
>>> class MyFormatter(FormatterBase):
...     def __init__(self, formatter: str = None):
...         self._formatter = formatter
...         if self._formatter:
...             self._parse_formatter()
...         else:
...             self._slices = None
...     def _parse_formatter(self):
...         slice_str = self._formatter.strip()[1:-1]
...         slices = []
...         parts = slice_str.split(":")
...         start = int(parts[0]) if parts[0] else None
...         end = int(parts[1]) if len(parts) > 1 and parts[1] else None
...         step = int(parts[2]) if len(parts) > 2 and parts[2] else None
...         slices.append(slice(start, end, step))
...         self._slices = slices
...     def _load(self, data):
...         return [int(x) for x in data.strip('[]').split(',')]
...     def _parse_py_data_by_formatter(self, data):
...         if self._slices is not None:
...             result = []
...             for s in self._slices:
...                 if isinstance(s, slice):
...                     result.extend(data[s])
...                 else:
...                     result.append(data[int(s)])
...             return result
...         else:
...             return data
...
>>> fmt = MyFormatter("[1:3]")
>>> res = fmt.format("[1,2,3,4,5]")
>>> print(res)
[2, 3]
Source code in lazyllm/components/formatter/formatterbase.py 
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
class LazyLLMFormatterBase(metaclass=LazyLLMRegisterMetaClass):
    """This class is the base class of the formatter. The formatter is the formatter of the model output result. Users can customize the formatter or use the formatter provided by LazyLLM.
Main methods: _parse_formatter: parse the index content. _load: Parse the str object, and the part containing Python objects is parsed out, such as list, dict and other objects. _parse_py_data_by_formatter: format the python object according to the custom formatter and index. format: format the passed content. If the content is a string type, convert the string into a python object first, and then format it. If the content is a python object, format it directly.


Examples:
    >>> from lazyllm.components.formatter import FormatterBase
    >>> class MyFormatter(FormatterBase):
    ...     def __init__(self, formatter: str = None):
    ...         self._formatter = formatter
    ...         if self._formatter:
    ...             self._parse_formatter()
    ...         else:
    ...             self._slices = None
    ...     def _parse_formatter(self):
    ...         slice_str = self._formatter.strip()[1:-1]
    ...         slices = []
    ...         parts = slice_str.split(":")
    ...         start = int(parts[0]) if parts[0] else None
    ...         end = int(parts[1]) if len(parts) > 1 and parts[1] else None
    ...         step = int(parts[2]) if len(parts) > 2 and parts[2] else None
    ...         slices.append(slice(start, end, step))
    ...         self._slices = slices
    ...     def _load(self, data):
    ...         return [int(x) for x in data.strip('[]').split(',')]
    ...     def _parse_py_data_by_formatter(self, data):
    ...         if self._slices is not None:
    ...             result = []
    ...             for s in self._slices:
    ...                 if isinstance(s, slice):
    ...                     result.extend(data[s])
    ...                 else:
    ...                     result.append(data[int(s)])
    ...             return result
    ...         else:
    ...             return data
    ...
    >>> fmt = MyFormatter("[1:3]")
    >>> res = fmt.format("[1,2,3,4,5]")
    >>> print(res)
    [2, 3]
    """
    def _load(self, msg: str):
        return msg

    def _parse_py_data_by_formatter(self, py_data):
        raise NotImplementedError("This data parse function is not implemented.")

    def format(self, msg):
        if isinstance(msg, str): msg = self._load(msg)
        return self._parse_py_data_by_formatter(msg)

    def __call__(self, *msg):
        return self.format(msg[0] if len(msg) == 1 else package(msg))

    def __or__(self, other):
        if not isinstance(other, LazyLLMFormatterBase):
            return NotImplemented
        return PipelineFormatter(other.__ror__(self))

    def __ror__(self, f: Callable) -> Pipeline:
        if isinstance(f, Pipeline):
            if not f._capture:
                _ = Finalizer(lambda: setattr(f, '_capture', True), lambda: setattr(f, '_capture', False))
            f._add(str(uuid.uuid4().hex) if len(f._item_names) else None, self)
            return f
        return Pipeline(f, self)

lazyllm.components.formatter.formatterbase.JsonLikeFormatter

Bases: LazyLLMFormatterBase

This class is used to extract subfields from nested structures (like dicts, lists, tuples) using a JSON-like indexing syntax.

The behavior is driven by a formatter string similar to Python-style slicing and dictionary access:

  • [0] fetches the first item
  • [0][{key}] accesses the key field in the first item
  • [0,1][{a,b}] fetches the a and b fields from the first and second items
  • [::2] does slicing with a step of 2
  • *[0][{x}] means return a wrapped/structured result

Parameters:

  • formatter (str, default: None ) –

    A format string that controls how to slice and extract the structure. If None, the input will be returned directly.

Examples:

>>> from lazyllm.components.formatter.formatterbase import JsonLikeFormatter
>>> formatter = JsonLikeFormatter("[{a,b}]")
Source code in lazyllm/components/formatter/formatterbase.py 
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
class JsonLikeFormatter(LazyLLMFormatterBase):
    """This class is used to extract subfields from nested structures (like dicts, lists, tuples) using a JSON-like indexing syntax.

The behavior is driven by a formatter string similar to Python-style slicing and dictionary access:

- `[0]` fetches the first item
- `[0][{key}]` accesses the `key` field in the first item
- `[0,1][{a,b}]` fetches the `a` and `b` fields from the first and second items
- `[::2]` does slicing with a step of 2
- `*[0][{x}]` means return a wrapped/structured result

Args:
    formatter (str, optional): A format string that controls how to slice and extract the structure. If None, the input will be returned directly.


Examples:
    >>> from lazyllm.components.formatter.formatterbase import JsonLikeFormatter
    >>> formatter = JsonLikeFormatter("[{a,b}]")
    """
    class _ListIdxes(tuple): pass
    class _DictKeys(tuple): pass

    def __init__(self, formatter: Optional[str] = None):
        if formatter and formatter.startswith('*['):
            self._return_package = True
            self._formatter = formatter.strip('*')
        else:
            self._return_package = False
            self._formatter = formatter

        if self._formatter:
            assert '*' not in self._formatter, '`*` can only be used before `[` in the beginning'
            self._formatter = self._formatter.strip().replace('{', '[{').replace('}', '}]')
            self._parse_formatter()
        else:
            self._slices = None

    def _parse_formatter(self):
        # Remove the surrounding brackets
        assert self._formatter.startswith('[') and self._formatter.endswith(']')
        slice_str = self._formatter.strip()[1:-1]
        dimensions = slice_str.split("][")
        slices = []

        for dim in dimensions:
            if '{' in dim:
                slices.append(__class__._DictKeys(d.strip() for d in dim[1:-1].split(',') if d.strip()))
            elif ":" in dim:
                assert ',' not in dim, '[a, b:c] is not supported'
                parts = dim.split(":")
                start = int(parts[0]) if _is_number(parts[0]) else None
                end = int(parts[1]) if len(parts) > 1 and _is_number(parts[1]) else None
                step = int(parts[2]) if len(parts) > 2 and _is_number(parts[2]) else None
                slices.append(slice(start, end, step))
            elif ',' in dim:
                slices.append(__class__._ListIdxes(d.strip() for d in dim.split(',') if d.strip()))
            else:
                slices.append(dim.strip())
        self._slices = slices

    def _parse_py_data_by_formatter(self, data, *, slices=None):  # noqa C901
        def _impl(data, slice):
            if isinstance(data, (tuple, list)) and isinstance(slice, str):
                return data[int(slice)]
            if isinstance(slice, __class__._ListIdxes):
                if isinstance(data, dict): return [data[k] for k in slice]
                elif isinstance(data, (tuple, list)): return type(data)(data[int(k)] for k in slice)
                else: raise RuntimeError('Only tuple/list/dict is supported for [a,b,c]')
            if isinstance(slice, __class__._DictKeys):
                assert isinstance(data, dict)
                if len(slice) == 1 and slice[0] == ':': return data
                return {k: data[k] for k in slice}
            return data[slice]

        if slices is None: slices = self._slices
        if not slices: return data
        curr_slice = slices[0]
        if isinstance(curr_slice, slice):
            if isinstance(data, dict):
                assert curr_slice.start is None and curr_slice.stop is None and curr_slice.step is None, (
                    'Only {:} and [:] is supported in dict slice')
                curr_slice = __class__._ListIdxes(data.keys())
            elif isinstance(data, (tuple, list)):
                return type(data)(self._parse_py_data_by_formatter(d, slices=slices[1:])
                                  for d in _impl(data, curr_slice))
        if isinstance(curr_slice, __class__._DictKeys):
            return {k: self._parse_py_data_by_formatter(v, slices=slices[1:])
                    for k, v in _impl(data, curr_slice).items()}
        elif isinstance(curr_slice, __class__._ListIdxes):
            tp = package if self._return_package else list if isinstance(data, dict) else type(data)
            return tp(self._parse_py_data_by_formatter(r, slices=slices[1:]) for r in _impl(data, curr_slice))
        else: return self._parse_py_data_by_formatter(_impl(data, curr_slice), slices=slices[1:])

lazyllm.components.formatter.formatterbase.PythonFormatter

Bases: JsonLikeFormatter

Reserved formatter class for supporting Python-style data extraction syntax. To be developed.

Currently inherits from JsonLikeFormatter with no additional behavior.

Source code in lazyllm/components/formatter/formatterbase.py 
196
197
198
199
200
201
class PythonFormatter(JsonLikeFormatter):
    """Reserved formatter class for supporting Python-style data extraction syntax. To be developed.

Currently inherits from JsonLikeFormatter with no additional behavior.
"""
    pass

lazyllm.components.formatter.FileFormatter

Bases: LazyLLMFormatterBase

A formatter that transforms query strings with document context between structured formats.

Supports three modes: - "decode": Decodes structured query strings into dictionaries with query and files. - "encode": Encodes a dictionary with query and files into a structured query string. - "merge": Merges multiple structured query strings into one.

Parameters:

  • formatter (str, default: 'decode' ) –

    The operation mode. Must be one of "decode", "encode", or "merge". Defaults to "decode".

Examples:

>>> from lazyllm.components.formatter import FileFormatter

>>> # Decode mode
>>> fmt = FileFormatter('decode')
Source code in lazyllm/components/formatter/formatterbase.py 
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
class FileFormatter(LazyLLMFormatterBase):
    """A formatter that transforms query strings with document context between structured formats.

Supports three modes:
- "decode": Decodes structured query strings into dictionaries with `query` and `files`.
- "encode": Encodes a dictionary with `query` and `files` into a structured query string.
- "merge": Merges multiple structured query strings into one.

Args:
    formatter (str): The operation mode. Must be one of "decode", "encode", or "merge". Defaults to "decode".


Examples:
    >>> from lazyllm.components.formatter import FileFormatter

    >>> # Decode mode
    >>> fmt = FileFormatter('decode')
    """

    def __init__(self, formatter: str = 'decode'):
        self._mode = formatter.strip().lower()
        assert self._mode in ('decode', 'encode', 'merge')

    def _parse_py_data_by_formatter(self, py_data):
        if self._mode == 'merge':
            if isinstance(py_data, str):
                return py_data
            assert isinstance(py_data, package)
            return lazyllm_merge_query(*py_data)

        if isinstance(py_data, package):
            res = []
            for i_data in py_data:
                res.append(self._parse_py_data_by_formatter(i_data))
            return package(res)
        elif isinstance(py_data, (str, dict)):
            return self._decode_one_data(py_data)
        else:
            return py_data

    def _decode_one_data(self, py_data):
        if self._mode == 'decode':
            if isinstance(py_data, str):
                return decode_query_with_filepaths(py_data)
            else:
                return py_data
        else:
            if isinstance(py_data, dict) and 'query' in py_data and 'files' in py_data:
                return encode_query_with_filepaths(**py_data)
            else:
                return py_data

lazyllm.components.formatter.YamlFormatter

Bases: JsonLikeFormatter

A formatter for extracting structured information from YAML-formatted strings.

Inherits from JsonLikeFormatter. Uses the internal method to parse YAML strings into Python objects, and then applies JSON-like formatting rules to extract desired fields.

Suitable for handling nested YAML content with formatter-based field selection.

Examples:

>>> from lazyllm.components.formatter import YamlFormatter
>>> formatter = YamlFormatter("{name,age}")
>>> msg = """ 
... name: Alice
... age: 30
... city: London
... """
>>> formatter(msg)
{'name': 'Alice', 'age': 30}
Source code in lazyllm/components/formatter/yamlformatter.py 
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
class YamlFormatter(JsonLikeFormatter):
    """A formatter for extracting structured information from YAML-formatted strings.

Inherits from JsonLikeFormatter. Uses the internal method to parse YAML strings into Python objects, and then applies JSON-like formatting rules to extract desired fields.

Suitable for handling nested YAML content with formatter-based field selection.


Examples:
    >>> from lazyllm.components.formatter import YamlFormatter
    >>> formatter = YamlFormatter("{name,age}")
    >>> msg = \"\"\" 
    ... name: Alice
    ... age: 30
    ... city: London
    ... \"\"\"
    >>> formatter(msg)
    {'name': 'Alice', 'age': 30}
    """
    def _load(self, msg: str):
        try:
            return yaml.load(msg, Loader=yaml.SafeLoader)
        except Exception as e:
            lazyllm.LOG.info(f"Error: {e}")
            return ""

lazyllm.components.formatter.encode_query_with_filepaths(query=None, files=None)

Encodes a query string together with associated file paths into a structured string format with context.

If file paths are provided, the query and file list will be wrapped into a JSON object prefixed with __lazyllm_docs__. Otherwise, it returns the original query string.

Parameters:

  • query (str, default: None ) –

    The user query string. Defaults to an empty string.

  • files (str or List[str], default: None ) –

    File path(s) associated with the query. Can be a single string or a list of strings.

Returns:

  • str ( str ) –

    A structured encoded query string or the raw query.

Raises:

  • AssertionError –

    If files is not a string or list of strings.

Examples:

>>> from lazyllm.components.formatter import encode_query_with_filepaths

>>> # Encode a query along with associated documentation files
>>> encode_query_with_filepaths("Generate questions based on the document", files=["a.md"])
'<lazyllm-query>{"query": "Generate questions based on the document", "files": ["a.md"]}'
Source code in lazyllm/components/formatter/formatterbase.py 
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
def encode_query_with_filepaths(query: str = None, files: Union[str, List[str]] = None) -> str:
    """Encodes a query string together with associated file paths into a structured string format with context.

If file paths are provided, the query and file list will be wrapped into a JSON object prefixed with ``__lazyllm_docs__``. Otherwise, it returns the original query string.

Args:
    query (str): The user query string. Defaults to an empty string.
    files (str or List[str]): File path(s) associated with the query. Can be a single string or a list of strings.

Returns:
    str: A structured encoded query string or the raw query.

Raises:
    AssertionError: If `files` is not a string or list of strings.


Examples:
    >>> from lazyllm.components.formatter import encode_query_with_filepaths

    >>> # Encode a query along with associated documentation files
    >>> encode_query_with_filepaths("Generate questions based on the document", files=["a.md"])
    '<lazyllm-query>{"query": "Generate questions based on the document", "files": ["a.md"]}'
    """
    query = query if query else ''
    if files:
        if isinstance(files, str): files = [files]
        assert isinstance(files, list), "files must be a list."
        assert all(isinstance(item, str) for item in files), "All items in files must be strings"
        return LAZYLLM_QUERY_PREFIX + json.dumps({'query': query, 'files': files})
    else:
        return query

lazyllm.components.formatter.decode_query_with_filepaths(query_files)

Decodes a structured query string into a dictionary containing the original query and file paths.

If the input string starts with the special prefix __lazyllm_docs__, it attempts to parse the JSON content; otherwise, it returns the raw query string as-is.

Parameters:

  • query_files (str) –

    The encoded query string that may include both query and file paths.

Returns:

  • Union[dict, str] –

    Union[dict, str]: A dictionary containing 'query' and 'files' if structured, otherwise the original query string.

Raises:

  • AssertionError –

    If the input is not a string.

  • ValueError –

    If the string is prefixed but JSON decoding fails.

Examples:

>>> from lazyllm.components.formatter import decode_query_with_filepaths

>>> # Decode a structured query with files
>>> decode_query_with_filepaths('<lazyllm-query>{"query": "Summarize the content", "files": ["doc.md"]}')
{'query': 'Summarize the content', 'files': ['doc.md']}

>>> # Decode a plain string without files
>>> decode_query_with_filepaths("This is just a simple question")
'This is just a simple question'
Source code in lazyllm/components/formatter/formatterbase.py 
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
def decode_query_with_filepaths(query_files: str) -> Union[dict, str]:
    """Decodes a structured query string into a dictionary containing the original query and file paths.

If the input string starts with the special prefix ``__lazyllm_docs__``, it attempts to parse the JSON content; otherwise, it returns the raw query string as-is.

Args:
    query_files (str): The encoded query string that may include both query and file paths.

Returns:
    Union[dict, str]: A dictionary containing 'query' and 'files' if structured, otherwise the original query string.

Raises:
    AssertionError: If the input is not a string.
    ValueError: If the string is prefixed but JSON decoding fails.


Examples:
    >>> from lazyllm.components.formatter import decode_query_with_filepaths

    >>> # Decode a structured query with files
    >>> decode_query_with_filepaths('<lazyllm-query>{"query": "Summarize the content", "files": ["doc.md"]}')
    {'query': 'Summarize the content', 'files': ['doc.md']}

    >>> # Decode a plain string without files
    >>> decode_query_with_filepaths("This is just a simple question")
    'This is just a simple question'
    """
    assert isinstance(query_files, str), "query_files must be a str."
    query_files = query_files.strip()
    if query_files.startswith(LAZYLLM_QUERY_PREFIX):
        try:
            obj = json.loads(query_files[len(LAZYLLM_QUERY_PREFIX):])
            return obj
        except json.JSONDecodeError as e:
            raise ValueError(f"JSON parsing failed: {e}")
    else:
        return query_files

lazyllm.components.formatter.lazyllm_merge_query(*args)

Merges multiple query strings (potentially with associated file paths) into a single structured query string.

Each argument can be a plain query string or a structured query created by encode_query_with_filepaths. The function decodes each input, concatenates all query texts, and merges the associated file paths. The final result is re-encoded into a single query string with unified context.

Parameters:

  • *args (str, default: () ) –

    Multiple query strings. Each can be either plain text or an encoded structured query with files.

Returns:

  • str ( str ) –

    A single structured query string containing the merged query and file paths.

Examples:

>>> from lazyllm.components.formatter import encode_query_with_filepaths, lazyllm_merge_query

>>> # Merge two structured queries with English content and associated files
>>> q1 = encode_query_with_filepaths("Please summarize document one", files=["doc1.md"])
>>> q2 = encode_query_with_filepaths("Add details from document two", files=["doc2.md"])
>>> lazyllm_merge_query(q1, q2)
'<lazyllm-query>{"query": "Please summarize document oneAdd details from document two", "files": ["doc1.md", "doc2.md"]}'

>>> # Merge plain English text queries without documents
>>> lazyllm_merge_query("What is AI?", "Explain deep learning.")
'What is AI?Explain deep learning.'
Source code in lazyllm/components/formatter/formatterbase.py 
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
def lazyllm_merge_query(*args: str) -> str:
    """Merges multiple query strings (potentially with associated file paths) into a single structured query string.

Each argument can be a plain query string or a structured query created by ``encode_query_with_filepaths``. The function decodes each input, concatenates all query texts, and merges the associated file paths. The final result is re-encoded into a single query string with unified context.

Args:
    *args (str): Multiple query strings. Each can be either plain text or an encoded structured query with files.

Returns:
    str: A single structured query string containing the merged query and file paths.


Examples:
    >>> from lazyllm.components.formatter import encode_query_with_filepaths, lazyllm_merge_query

    >>> # Merge two structured queries with English content and associated files
    >>> q1 = encode_query_with_filepaths("Please summarize document one", files=["doc1.md"])
    >>> q2 = encode_query_with_filepaths("Add details from document two", files=["doc2.md"])
    >>> lazyllm_merge_query(q1, q2)
    '<lazyllm-query>{"query": "Please summarize document oneAdd details from document two", "files": ["doc1.md", "doc2.md"]}'

    >>> # Merge plain English text queries without documents
    >>> lazyllm_merge_query("What is AI?", "Explain deep learning.")
    'What is AI?Explain deep learning.'
    """
    if len(args) == 1:
        return args[0]
    for item in args:
        assert isinstance(item, str), "Merge object must be str!"
    querys = ''
    files = []
    for item in args:
        decode = decode_query_with_filepaths(item)
        if isinstance(decode, dict):
            querys += decode['query']
            files.extend(decode['files'])
        else:
            querys += decode
    return encode_query_with_filepaths(querys, files)

lazyllm.components.formatter.formatterbase.JsonLikeFormatter

Bases: LazyLLMFormatterBase

This class is used to extract subfields from nested structures (like dicts, lists, tuples) using a JSON-like indexing syntax.

The behavior is driven by a formatter string similar to Python-style slicing and dictionary access:

  • [0] fetches the first item
  • [0][{key}] accesses the key field in the first item
  • [0,1][{a,b}] fetches the a and b fields from the first and second items
  • [::2] does slicing with a step of 2
  • *[0][{x}] means return a wrapped/structured result

Parameters:

  • formatter (str, default: None ) –

    A format string that controls how to slice and extract the structure. If None, the input will be returned directly.

Examples:

>>> from lazyllm.components.formatter.formatterbase import JsonLikeFormatter
>>> formatter = JsonLikeFormatter("[{a,b}]")
Source code in lazyllm/components/formatter/formatterbase.py 
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
class JsonLikeFormatter(LazyLLMFormatterBase):
    """This class is used to extract subfields from nested structures (like dicts, lists, tuples) using a JSON-like indexing syntax.

The behavior is driven by a formatter string similar to Python-style slicing and dictionary access:

- `[0]` fetches the first item
- `[0][{key}]` accesses the `key` field in the first item
- `[0,1][{a,b}]` fetches the `a` and `b` fields from the first and second items
- `[::2]` does slicing with a step of 2
- `*[0][{x}]` means return a wrapped/structured result

Args:
    formatter (str, optional): A format string that controls how to slice and extract the structure. If None, the input will be returned directly.


Examples:
    >>> from lazyllm.components.formatter.formatterbase import JsonLikeFormatter
    >>> formatter = JsonLikeFormatter("[{a,b}]")
    """
    class _ListIdxes(tuple): pass
    class _DictKeys(tuple): pass

    def __init__(self, formatter: Optional[str] = None):
        if formatter and formatter.startswith('*['):
            self._return_package = True
            self._formatter = formatter.strip('*')
        else:
            self._return_package = False
            self._formatter = formatter

        if self._formatter:
            assert '*' not in self._formatter, '`*` can only be used before `[` in the beginning'
            self._formatter = self._formatter.strip().replace('{', '[{').replace('}', '}]')
            self._parse_formatter()
        else:
            self._slices = None

    def _parse_formatter(self):
        # Remove the surrounding brackets
        assert self._formatter.startswith('[') and self._formatter.endswith(']')
        slice_str = self._formatter.strip()[1:-1]
        dimensions = slice_str.split("][")
        slices = []

        for dim in dimensions:
            if '{' in dim:
                slices.append(__class__._DictKeys(d.strip() for d in dim[1:-1].split(',') if d.strip()))
            elif ":" in dim:
                assert ',' not in dim, '[a, b:c] is not supported'
                parts = dim.split(":")
                start = int(parts[0]) if _is_number(parts[0]) else None
                end = int(parts[1]) if len(parts) > 1 and _is_number(parts[1]) else None
                step = int(parts[2]) if len(parts) > 2 and _is_number(parts[2]) else None
                slices.append(slice(start, end, step))
            elif ',' in dim:
                slices.append(__class__._ListIdxes(d.strip() for d in dim.split(',') if d.strip()))
            else:
                slices.append(dim.strip())
        self._slices = slices

    def _parse_py_data_by_formatter(self, data, *, slices=None):  # noqa C901
        def _impl(data, slice):
            if isinstance(data, (tuple, list)) and isinstance(slice, str):
                return data[int(slice)]
            if isinstance(slice, __class__._ListIdxes):
                if isinstance(data, dict): return [data[k] for k in slice]
                elif isinstance(data, (tuple, list)): return type(data)(data[int(k)] for k in slice)
                else: raise RuntimeError('Only tuple/list/dict is supported for [a,b,c]')
            if isinstance(slice, __class__._DictKeys):
                assert isinstance(data, dict)
                if len(slice) == 1 and slice[0] == ':': return data
                return {k: data[k] for k in slice}
            return data[slice]

        if slices is None: slices = self._slices
        if not slices: return data
        curr_slice = slices[0]
        if isinstance(curr_slice, slice):
            if isinstance(data, dict):
                assert curr_slice.start is None and curr_slice.stop is None and curr_slice.step is None, (
                    'Only {:} and [:] is supported in dict slice')
                curr_slice = __class__._ListIdxes(data.keys())
            elif isinstance(data, (tuple, list)):
                return type(data)(self._parse_py_data_by_formatter(d, slices=slices[1:])
                                  for d in _impl(data, curr_slice))
        if isinstance(curr_slice, __class__._DictKeys):
            return {k: self._parse_py_data_by_formatter(v, slices=slices[1:])
                    for k, v in _impl(data, curr_slice).items()}
        elif isinstance(curr_slice, __class__._ListIdxes):
            tp = package if self._return_package else list if isinstance(data, dict) else type(data)
            return tp(self._parse_py_data_by_formatter(r, slices=slices[1:]) for r in _impl(data, curr_slice))
        else: return self._parse_py_data_by_formatter(_impl(data, curr_slice), slices=slices[1:])

lazyllm.components.formatter.formatterbase.PythonFormatter

Bases: JsonLikeFormatter

Reserved formatter class for supporting Python-style data extraction syntax. To be developed.

Currently inherits from JsonLikeFormatter with no additional behavior.

Source code in lazyllm/components/formatter/formatterbase.py 
196
197
198
199
200
201
class PythonFormatter(JsonLikeFormatter):
    """Reserved formatter class for supporting Python-style data extraction syntax. To be developed.

Currently inherits from JsonLikeFormatter with no additional behavior.
"""
    pass

lazyllm.components.formatter.FileFormatter

Bases: LazyLLMFormatterBase

A formatter that transforms query strings with document context between structured formats.

Supports three modes: - "decode": Decodes structured query strings into dictionaries with query and files. - "encode": Encodes a dictionary with query and files into a structured query string. - "merge": Merges multiple structured query strings into one.

Parameters:

  • formatter (str, default: 'decode' ) –

    The operation mode. Must be one of "decode", "encode", or "merge". Defaults to "decode".

Examples:

>>> from lazyllm.components.formatter import FileFormatter

>>> # Decode mode
>>> fmt = FileFormatter('decode')
Source code in lazyllm/components/formatter/formatterbase.py 
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
class FileFormatter(LazyLLMFormatterBase):
    """A formatter that transforms query strings with document context between structured formats.

Supports three modes:
- "decode": Decodes structured query strings into dictionaries with `query` and `files`.
- "encode": Encodes a dictionary with `query` and `files` into a structured query string.
- "merge": Merges multiple structured query strings into one.

Args:
    formatter (str): The operation mode. Must be one of "decode", "encode", or "merge". Defaults to "decode".


Examples:
    >>> from lazyllm.components.formatter import FileFormatter

    >>> # Decode mode
    >>> fmt = FileFormatter('decode')
    """

    def __init__(self, formatter: str = 'decode'):
        self._mode = formatter.strip().lower()
        assert self._mode in ('decode', 'encode', 'merge')

    def _parse_py_data_by_formatter(self, py_data):
        if self._mode == 'merge':
            if isinstance(py_data, str):
                return py_data
            assert isinstance(py_data, package)
            return lazyllm_merge_query(*py_data)

        if isinstance(py_data, package):
            res = []
            for i_data in py_data:
                res.append(self._parse_py_data_by_formatter(i_data))
            return package(res)
        elif isinstance(py_data, (str, dict)):
            return self._decode_one_data(py_data)
        else:
            return py_data

    def _decode_one_data(self, py_data):
        if self._mode == 'decode':
            if isinstance(py_data, str):
                return decode_query_with_filepaths(py_data)
            else:
                return py_data
        else:
            if isinstance(py_data, dict) and 'query' in py_data and 'files' in py_data:
                return encode_query_with_filepaths(**py_data)
            else:
                return py_data

lazyllm.components.formatter.YamlFormatter

Bases: JsonLikeFormatter

A formatter for extracting structured information from YAML-formatted strings.

Inherits from JsonLikeFormatter. Uses the internal method to parse YAML strings into Python objects, and then applies JSON-like formatting rules to extract desired fields.

Suitable for handling nested YAML content with formatter-based field selection.

Examples:

>>> from lazyllm.components.formatter import YamlFormatter
>>> formatter = YamlFormatter("{name,age}")
>>> msg = """ 
... name: Alice
... age: 30
... city: London
... """
>>> formatter(msg)
{'name': 'Alice', 'age': 30}
Source code in lazyllm/components/formatter/yamlformatter.py 
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
class YamlFormatter(JsonLikeFormatter):
    """A formatter for extracting structured information from YAML-formatted strings.

Inherits from JsonLikeFormatter. Uses the internal method to parse YAML strings into Python objects, and then applies JSON-like formatting rules to extract desired fields.

Suitable for handling nested YAML content with formatter-based field selection.


Examples:
    >>> from lazyllm.components.formatter import YamlFormatter
    >>> formatter = YamlFormatter("{name,age}")
    >>> msg = \"\"\" 
    ... name: Alice
    ... age: 30
    ... city: London
    ... \"\"\"
    >>> formatter(msg)
    {'name': 'Alice', 'age': 30}
    """
    def _load(self, msg: str):
        try:
            return yaml.load(msg, Loader=yaml.SafeLoader)
        except Exception as e:
            lazyllm.LOG.info(f"Error: {e}")
            return ""

lazyllm.components.formatter.encode_query_with_filepaths(query=None, files=None)

Encodes a query string together with associated file paths into a structured string format with context.

If file paths are provided, the query and file list will be wrapped into a JSON object prefixed with __lazyllm_docs__. Otherwise, it returns the original query string.

Parameters:

  • query (str, default: None ) –

    The user query string. Defaults to an empty string.

  • files (str or List[str], default: None ) –

    File path(s) associated with the query. Can be a single string or a list of strings.

Returns:

  • str ( str ) –

    A structured encoded query string or the raw query.

Raises:

  • AssertionError –

    If files is not a string or list of strings.

Examples:

>>> from lazyllm.components.formatter import encode_query_with_filepaths

>>> # Encode a query along with associated documentation files
>>> encode_query_with_filepaths("Generate questions based on the document", files=["a.md"])
'<lazyllm-query>{"query": "Generate questions based on the document", "files": ["a.md"]}'
Source code in lazyllm/components/formatter/formatterbase.py 
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
def encode_query_with_filepaths(query: str = None, files: Union[str, List[str]] = None) -> str:
    """Encodes a query string together with associated file paths into a structured string format with context.

If file paths are provided, the query and file list will be wrapped into a JSON object prefixed with ``__lazyllm_docs__``. Otherwise, it returns the original query string.

Args:
    query (str): The user query string. Defaults to an empty string.
    files (str or List[str]): File path(s) associated with the query. Can be a single string or a list of strings.

Returns:
    str: A structured encoded query string or the raw query.

Raises:
    AssertionError: If `files` is not a string or list of strings.


Examples:
    >>> from lazyllm.components.formatter import encode_query_with_filepaths

    >>> # Encode a query along with associated documentation files
    >>> encode_query_with_filepaths("Generate questions based on the document", files=["a.md"])
    '<lazyllm-query>{"query": "Generate questions based on the document", "files": ["a.md"]}'
    """
    query = query if query else ''
    if files:
        if isinstance(files, str): files = [files]
        assert isinstance(files, list), "files must be a list."
        assert all(isinstance(item, str) for item in files), "All items in files must be strings"
        return LAZYLLM_QUERY_PREFIX + json.dumps({'query': query, 'files': files})
    else:
        return query

lazyllm.components.formatter.decode_query_with_filepaths(query_files)

Decodes a structured query string into a dictionary containing the original query and file paths.

If the input string starts with the special prefix __lazyllm_docs__, it attempts to parse the JSON content; otherwise, it returns the raw query string as-is.

Parameters:

  • query_files (str) –

    The encoded query string that may include both query and file paths.

Returns:

  • Union[dict, str] –

    Union[dict, str]: A dictionary containing 'query' and 'files' if structured, otherwise the original query string.

Raises:

  • AssertionError –

    If the input is not a string.

  • ValueError –

    If the string is prefixed but JSON decoding fails.

Examples:

>>> from lazyllm.components.formatter import decode_query_with_filepaths

>>> # Decode a structured query with files
>>> decode_query_with_filepaths('<lazyllm-query>{"query": "Summarize the content", "files": ["doc.md"]}')
{'query': 'Summarize the content', 'files': ['doc.md']}

>>> # Decode a plain string without files
>>> decode_query_with_filepaths("This is just a simple question")
'This is just a simple question'
Source code in lazyllm/components/formatter/formatterbase.py 
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
def decode_query_with_filepaths(query_files: str) -> Union[dict, str]:
    """Decodes a structured query string into a dictionary containing the original query and file paths.

If the input string starts with the special prefix ``__lazyllm_docs__``, it attempts to parse the JSON content; otherwise, it returns the raw query string as-is.

Args:
    query_files (str): The encoded query string that may include both query and file paths.

Returns:
    Union[dict, str]: A dictionary containing 'query' and 'files' if structured, otherwise the original query string.

Raises:
    AssertionError: If the input is not a string.
    ValueError: If the string is prefixed but JSON decoding fails.


Examples:
    >>> from lazyllm.components.formatter import decode_query_with_filepaths

    >>> # Decode a structured query with files
    >>> decode_query_with_filepaths('<lazyllm-query>{"query": "Summarize the content", "files": ["doc.md"]}')
    {'query': 'Summarize the content', 'files': ['doc.md']}

    >>> # Decode a plain string without files
    >>> decode_query_with_filepaths("This is just a simple question")
    'This is just a simple question'
    """
    assert isinstance(query_files, str), "query_files must be a str."
    query_files = query_files.strip()
    if query_files.startswith(LAZYLLM_QUERY_PREFIX):
        try:
            obj = json.loads(query_files[len(LAZYLLM_QUERY_PREFIX):])
            return obj
        except json.JSONDecodeError as e:
            raise ValueError(f"JSON parsing failed: {e}")
    else:
        return query_files

lazyllm.components.formatter.lazyllm_merge_query(*args)

Merges multiple query strings (potentially with associated file paths) into a single structured query string.

Each argument can be a plain query string or a structured query created by encode_query_with_filepaths. The function decodes each input, concatenates all query texts, and merges the associated file paths. The final result is re-encoded into a single query string with unified context.

Parameters:

  • *args (str, default: () ) –

    Multiple query strings. Each can be either plain text or an encoded structured query with files.

Returns:

  • str ( str ) –

    A single structured query string containing the merged query and file paths.

Examples:

>>> from lazyllm.components.formatter import encode_query_with_filepaths, lazyllm_merge_query

>>> # Merge two structured queries with English content and associated files
>>> q1 = encode_query_with_filepaths("Please summarize document one", files=["doc1.md"])
>>> q2 = encode_query_with_filepaths("Add details from document two", files=["doc2.md"])
>>> lazyllm_merge_query(q1, q2)
'<lazyllm-query>{"query": "Please summarize document oneAdd details from document two", "files": ["doc1.md", "doc2.md"]}'

>>> # Merge plain English text queries without documents
>>> lazyllm_merge_query("What is AI?", "Explain deep learning.")
'What is AI?Explain deep learning.'
Source code in lazyllm/components/formatter/formatterbase.py 
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
def lazyllm_merge_query(*args: str) -> str:
    """Merges multiple query strings (potentially with associated file paths) into a single structured query string.

Each argument can be a plain query string or a structured query created by ``encode_query_with_filepaths``. The function decodes each input, concatenates all query texts, and merges the associated file paths. The final result is re-encoded into a single query string with unified context.

Args:
    *args (str): Multiple query strings. Each can be either plain text or an encoded structured query with files.

Returns:
    str: A single structured query string containing the merged query and file paths.


Examples:
    >>> from lazyllm.components.formatter import encode_query_with_filepaths, lazyllm_merge_query

    >>> # Merge two structured queries with English content and associated files
    >>> q1 = encode_query_with_filepaths("Please summarize document one", files=["doc1.md"])
    >>> q2 = encode_query_with_filepaths("Add details from document two", files=["doc2.md"])
    >>> lazyllm_merge_query(q1, q2)
    '<lazyllm-query>{"query": "Please summarize document oneAdd details from document two", "files": ["doc1.md", "doc2.md"]}'

    >>> # Merge plain English text queries without documents
    >>> lazyllm_merge_query("What is AI?", "Explain deep learning.")
    'What is AI?Explain deep learning.'
    """
    if len(args) == 1:
        return args[0]
    for item in args:
        assert isinstance(item, str), "Merge object must be str!"
    querys = ''
    files = []
    for item in args:
        decode = decode_query_with_filepaths(item)
        if isinstance(decode, dict):
            querys += decode['query']
            files.extend(decode['files'])
        else:
            querys += decode
    return encode_query_with_filepaths(querys, files)

lazyllm.components.JsonFormatter

Bases: JsonLikeFormatter

This class is a JSON formatter, that is, the user wants the model to output content is JSON format, and can also select a field in the output content by indexing.

Examples:

>>> import lazyllm
>>> from lazyllm.components import JsonFormatter
>>> toc_prompt='''
... You are now an intelligent assistant. Your task is to understand the user's input and convert the outline into a list of nested dictionaries. Each dictionary contains a `title` and a `describe`, where the `title` should clearly indicate the level using Markdown format, and the `describe` is a description and writing guide for that section.
... 
... Please generate the corresponding list of nested dictionaries based on the following user input:
... 
... Example output:
... [
...     {
...         "title": "# Level 1 Title",
...         "describe": "Please provide a detailed description of the content under this title, offering background information and core viewpoints."
...     },
...     {
...         "title": "## Level 2 Title",
...         "describe": "Please provide a detailed description of the content under this title, giving specific details and examples to support the viewpoints of the Level 1 title."
...     },
...     {
...         "title": "### Level 3 Title",
...         "describe": "Please provide a detailed description of the content under this title, deeply analyzing and providing more details and data support."
...     }
... ]
... User input is as follows:
... '''
>>> query = "Please help me write an article about the application of artificial intelligence in the medical field."
>>> m = lazyllm.TrainableModule("internlm2-chat-20b").prompt(toc_prompt).start()
>>> ret = m(query, max_new_tokens=2048)
>>> print(f"ret: {ret!r}")  # the model output without specifying a formatter
'Based on your user input, here is the corresponding list of nested dictionaries:
[
    {
        "title": "# Application of Artificial Intelligence in the Medical Field",
        "describe": "Please provide a detailed description of the application of artificial intelligence in the medical field, including its benefits, challenges, and future prospects."
    },
    {
        "title": "## AI in Medical Diagnosis",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical diagnosis, including specific examples of AI-based diagnostic tools and their impact on patient outcomes."
    },
    {
        "title": "### AI in Medical Imaging",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical imaging, including the advantages of AI-based image analysis and its applications in various medical specialties."
    },
    {
        "title": "### AI in Drug Discovery and Development",
        "describe": "Please provide a detailed description of how artificial intelligence is used in drug discovery and development, including the role of AI in identifying potential drug candidates and streamlining the drug development process."
    },
    {
        "title": "## AI in Medical Research",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical research, including its applications in genomics, epidemiology, and clinical trials."
    },
    {
        "title": "### AI in Genomics and Precision Medicine",
        "describe": "Please provide a detailed description of how artificial intelligence is used in genomics and precision medicine, including the role of AI in analyzing large-scale genomic data and tailoring treatments to individual patients."
    },
    {
        "title": "### AI in Epidemiology and Public Health",
        "describe": "Please provide a detailed description of how artificial intelligence is used in epidemiology and public health, including its applications in disease surveillance, outbreak prediction, and resource allocation."
    },
    {
        "title": "### AI in Clinical Trials",
        "describe": "Please provide a detailed description of how artificial intelligence is used in clinical trials, including its role in patient recruitment, trial design, and data analysis."
    },
    {
        "title": "## AI in Medical Practice",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical practice, including its applications in patient monitoring, personalized medicine, and telemedicine."
    },
    {
        "title": "### AI in Patient Monitoring",
        "describe": "Please provide a detailed description of how artificial intelligence is used in patient monitoring, including its role in real-time monitoring of vital signs and early detection of health issues."
    },
    {
        "title": "### AI in Personalized Medicine",
        "describe": "Please provide a detailed description of how artificial intelligence is used in personalized medicine, including its role in analyzing patient data to tailor treatments and predict outcomes."
    },
    {
        "title": "### AI in Telemedicine",
        "describe": "Please provide a detailed description of how artificial intelligence is used in telemedicine, including its applications in remote consultations, virtual diagnoses, and digital health records."
    },
    {
        "title": "## AI in Medical Ethics and Policy",
        "describe": "Please provide a detailed description of the ethical and policy considerations surrounding the use of artificial intelligence in the medical field, including issues related to data privacy, bias, and accountability."
    }
]'
>>> m = lazyllm.TrainableModule("internlm2-chat-20b").formatter(JsonFormatter("[:][title]")).prompt(toc_prompt).start()
>>> ret = m(query, max_new_tokens=2048)
>>> print(f"ret: {ret}")  # the model output of the specified formaater
['# Application of Artificial Intelligence in the Medical Field', '## AI in Medical Diagnosis', '### AI in Medical Imaging', '### AI in Drug Discovery and Development', '## AI in Medical Research', '### AI in Genomics and Precision Medicine', '### AI in Epidemiology and Public Health', '### AI in Clinical Trials', '## AI in Medical Practice', '### AI in Patient Monitoring', '### AI in Personalized Medicine', '### AI in Telemedicine', '## AI in Medical Ethics and Policy']
Source code in lazyllm/components/formatter/jsonformatter.py 
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
class JsonFormatter(JsonLikeFormatter):
    """This class is a JSON formatter, that is, the user wants the model to output content is JSON format, and can also select a field in the output content by indexing.


Examples:
    >>> import lazyllm
    >>> from lazyllm.components import JsonFormatter
    >>> toc_prompt='''
    ... You are now an intelligent assistant. Your task is to understand the user's input and convert the outline into a list of nested dictionaries. Each dictionary contains a `title` and a `describe`, where the `title` should clearly indicate the level using Markdown format, and the `describe` is a description and writing guide for that section.
    ... 
    ... Please generate the corresponding list of nested dictionaries based on the following user input:
    ... 
    ... Example output:
    ... [
    ...     {
    ...         "title": "# Level 1 Title",
    ...         "describe": "Please provide a detailed description of the content under this title, offering background information and core viewpoints."
    ...     },
    ...     {
    ...         "title": "## Level 2 Title",
    ...         "describe": "Please provide a detailed description of the content under this title, giving specific details and examples to support the viewpoints of the Level 1 title."
    ...     },
    ...     {
    ...         "title": "### Level 3 Title",
    ...         "describe": "Please provide a detailed description of the content under this title, deeply analyzing and providing more details and data support."
    ...     }
    ... ]
    ... User input is as follows:
    ... '''
    >>> query = "Please help me write an article about the application of artificial intelligence in the medical field."
    >>> m = lazyllm.TrainableModule("internlm2-chat-20b").prompt(toc_prompt).start()
    >>> ret = m(query, max_new_tokens=2048)
    >>> print(f"ret: {ret!r}")  # the model output without specifying a formatter
    'Based on your user input, here is the corresponding list of nested dictionaries:
    [
        {
            "title": "# Application of Artificial Intelligence in the Medical Field",
            "describe": "Please provide a detailed description of the application of artificial intelligence in the medical field, including its benefits, challenges, and future prospects."
        },
        {
            "title": "## AI in Medical Diagnosis",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical diagnosis, including specific examples of AI-based diagnostic tools and their impact on patient outcomes."
        },
        {
            "title": "### AI in Medical Imaging",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical imaging, including the advantages of AI-based image analysis and its applications in various medical specialties."
        },
        {
            "title": "### AI in Drug Discovery and Development",
            "describe": "Please provide a detailed description of how artificial intelligence is used in drug discovery and development, including the role of AI in identifying potential drug candidates and streamlining the drug development process."
        },
        {
            "title": "## AI in Medical Research",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical research, including its applications in genomics, epidemiology, and clinical trials."
        },
        {
            "title": "### AI in Genomics and Precision Medicine",
            "describe": "Please provide a detailed description of how artificial intelligence is used in genomics and precision medicine, including the role of AI in analyzing large-scale genomic data and tailoring treatments to individual patients."
        },
        {
            "title": "### AI in Epidemiology and Public Health",
            "describe": "Please provide a detailed description of how artificial intelligence is used in epidemiology and public health, including its applications in disease surveillance, outbreak prediction, and resource allocation."
        },
        {
            "title": "### AI in Clinical Trials",
            "describe": "Please provide a detailed description of how artificial intelligence is used in clinical trials, including its role in patient recruitment, trial design, and data analysis."
        },
        {
            "title": "## AI in Medical Practice",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical practice, including its applications in patient monitoring, personalized medicine, and telemedicine."
        },
        {
            "title": "### AI in Patient Monitoring",
            "describe": "Please provide a detailed description of how artificial intelligence is used in patient monitoring, including its role in real-time monitoring of vital signs and early detection of health issues."
        },
        {
            "title": "### AI in Personalized Medicine",
            "describe": "Please provide a detailed description of how artificial intelligence is used in personalized medicine, including its role in analyzing patient data to tailor treatments and predict outcomes."
        },
        {
            "title": "### AI in Telemedicine",
            "describe": "Please provide a detailed description of how artificial intelligence is used in telemedicine, including its applications in remote consultations, virtual diagnoses, and digital health records."
        },
        {
            "title": "## AI in Medical Ethics and Policy",
            "describe": "Please provide a detailed description of the ethical and policy considerations surrounding the use of artificial intelligence in the medical field, including issues related to data privacy, bias, and accountability."
        }
    ]'
    >>> m = lazyllm.TrainableModule("internlm2-chat-20b").formatter(JsonFormatter("[:][title]")).prompt(toc_prompt).start()
    >>> ret = m(query, max_new_tokens=2048)
    >>> print(f"ret: {ret}")  # the model output of the specified formaater
    ['# Application of Artificial Intelligence in the Medical Field', '## AI in Medical Diagnosis', '### AI in Medical Imaging', '### AI in Drug Discovery and Development', '## AI in Medical Research', '### AI in Genomics and Precision Medicine', '### AI in Epidemiology and Public Health', '### AI in Clinical Trials', '## AI in Medical Practice', '### AI in Patient Monitoring', '### AI in Personalized Medicine', '### AI in Telemedicine', '## AI in Medical Ethics and Policy']
    """
    def _extract_json_from_string(self, mixed_str: str):
        json_objects = []
        brace_level = 0
        current_json = ""
        in_string = False

        for char in mixed_str:
            if char == '"' and (len(current_json) == 0 or current_json[-1] != '\\'):
                in_string = not in_string

            if not in_string:
                if char in '{[':
                    if brace_level == 0:
                        current_json = ""
                    brace_level += 1
                elif char in '}]':
                    brace_level -= 1

            if brace_level > 0 or (brace_level == 0 and char in '}]'):
                current_json += char

            if brace_level == 0 and current_json:
                try:
                    json.loads(current_json)
                    json_objects.append(current_json)
                    current_json = ""
                except json.JSONDecodeError:
                    continue

        return json_objects

    def _load(self, msg: str):
        # Convert str to json format
        assert msg.count("{") == msg.count("}"), f"{msg} is not a valid json string."
        try:
            json_strs = self._extract_json_from_string(msg)
            if len(json_strs) == 0:
                raise TypeError(f"{msg} is not a valid json string.")
            res = []
            for json_str in json_strs:
                res.append(json.loads(json_str))
            return res if len(res) > 1 else res[0]
        except Exception as e:
            lazyllm.LOG.info(f"Error: {e}")
            return ""

lazyllm.components.EmptyFormatter

Bases: LazyLLMFormatterBase

This type is the system default formatter. When the user does not specify anything or does not want to format the model output, this type is selected. The model output will be in the same format.

Examples:

>>> import lazyllm
>>> from lazyllm.components import EmptyFormatter
>>> toc_prompt='''
... You are now an intelligent assistant. Your task is to understand the user's input and convert the outline into a list of nested dictionaries. Each dictionary contains a `title` and a `describe`, where the `title` should clearly indicate the level using Markdown format, and the `describe` is a description and writing guide for that section.
... 
... Please generate the corresponding list of nested dictionaries based on the following user input:
... 
... Example output:
... [
...     {
...         "title": "# Level 1 Title",
...         "describe": "Please provide a detailed description of the content under this title, offering background information and core viewpoints."
...     },
...     {
...         "title": "## Level 2 Title",
...         "describe": "Please provide a detailed description of the content under this title, giving specific details and examples to support the viewpoints of the Level 1 title."
...     },
...     {
...         "title": "### Level 3 Title",
...         "describe": "Please provide a detailed description of the content under this title, deeply analyzing and providing more details and data support."
...     }
... ]
... User input is as follows:
... '''
>>> query = "Please help me write an article about the application of artificial intelligence in the medical field."
>>> m = lazyllm.TrainableModule("internlm2-chat-20b").prompt(toc_prompt).start()  # the model output without specifying a formatter
>>> ret = m(query, max_new_tokens=2048)
>>> print(f"ret: {ret!r}")
'Based on your user input, here is the corresponding list of nested dictionaries:
[
    {
        "title": "# Application of Artificial Intelligence in the Medical Field",
        "describe": "Please provide a detailed description of the application of artificial intelligence in the medical field, including its benefits, challenges, and future prospects."
    },
    {
        "title": "## AI in Medical Diagnosis",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical diagnosis, including specific examples of AI-based diagnostic tools and their impact on patient outcomes."
    },
    {
        "title": "### AI in Medical Imaging",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical imaging, including the advantages of AI-based image analysis and its applications in various medical specialties."
    },
    {
        "title": "### AI in Drug Discovery and Development",
        "describe": "Please provide a detailed description of how artificial intelligence is used in drug discovery and development, including the role of AI in identifying potential drug candidates and streamlining the drug development process."
    },
    {
        "title": "## AI in Medical Research",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical research, including its applications in genomics, epidemiology, and clinical trials."
    },
    {
        "title": "### AI in Genomics and Precision Medicine",
        "describe": "Please provide a detailed description of how artificial intelligence is used in genomics and precision medicine, including the role of AI in analyzing large-scale genomic data and tailoring treatments to individual patients."
    },
    {
        "title": "### AI in Epidemiology and Public Health",
        "describe": "Please provide a detailed description of how artificial intelligence is used in epidemiology and public health, including its applications in disease surveillance, outbreak prediction, and resource allocation."
    },
    {
        "title": "### AI in Clinical Trials",
        "describe": "Please provide a detailed description of how artificial intelligence is used in clinical trials, including its role in patient recruitment, trial design, and data analysis."
    },
    {
        "title": "## AI in Medical Practice",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical practice, including its applications in patient monitoring, personalized medicine, and telemedicine."
    },
    {
        "title": "### AI in Patient Monitoring",
        "describe": "Please provide a detailed description of how artificial intelligence is used in patient monitoring, including its role in real-time monitoring of vital signs and early detection of health issues."
    },
    {
        "title": "### AI in Personalized Medicine",
        "describe": "Please provide a detailed description of how artificial intelligence is used in personalized medicine, including its role in analyzing patient data to tailor treatments and predict outcomes."
    },
    {
        "title": "### AI in Telemedicine",
        "describe": "Please provide a detailed description of how artificial intelligence is used in telemedicine, including its applications in remote consultations, virtual diagnoses, and digital health records."
    },
    {
        "title": "## AI in Medical Ethics and Policy",
        "describe": "Please provide a detailed description of the ethical and policy considerations surrounding the use of artificial intelligence in the medical field, including issues related to data privacy, bias, and accountability."
    }
]'
>>> m = lazyllm.TrainableModule("internlm2-chat-20b").formatter(EmptyFormatter()).prompt(toc_prompt).start()  # the model output of the specified formatter
>>> ret = m(query, max_new_tokens=2048)
>>> print(f"ret: {ret!r}")
'Based on your user input, here is the corresponding list of nested dictionaries:
[
    {
        "title": "# Application of Artificial Intelligence in the Medical Field",
        "describe": "Please provide a detailed description of the application of artificial intelligence in the medical field, including its benefits, challenges, and future prospects."
    },
    {
        "title": "## AI in Medical Diagnosis",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical diagnosis, including specific examples of AI-based diagnostic tools and their impact on patient outcomes."
    },
    {
        "title": "### AI in Medical Imaging",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical imaging, including the advantages of AI-based image analysis and its applications in various medical specialties."
    },
    {
        "title": "### AI in Drug Discovery and Development",
        "describe": "Please provide a detailed description of how artificial intelligence is used in drug discovery and development, including the role of AI in identifying potential drug candidates and streamlining the drug development process."
    },
    {
        "title": "## AI in Medical Research",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical research, including its applications in genomics, epidemiology, and clinical trials."
    },
    {
        "title": "### AI in Genomics and Precision Medicine",
        "describe": "Please provide a detailed description of how artificial intelligence is used in genomics and precision medicine, including the role of AI in analyzing large-scale genomic data and tailoring treatments to individual patients."
    },
    {
        "title": "### AI in Epidemiology and Public Health",
        "describe": "Please provide a detailed description of how artificial intelligence is used in epidemiology and public health, including its applications in disease surveillance, outbreak prediction, and resource allocation."
    },
    {
        "title": "### AI in Clinical Trials",
        "describe": "Please provide a detailed description of how artificial intelligence is used in clinical trials, including its role in patient recruitment, trial design, and data analysis."
    },
    {
        "title": "## AI in Medical Practice",
        "describe": "Please provide a detailed description of how artificial intelligence is used in medical practice, including its applications in patient monitoring, personalized medicine, and telemedicine."
    },
    {
        "title": "### AI in Patient Monitoring",
        "describe": "Please provide a detailed description of how artificial intelligence is used in patient monitoring, including its role in real-time monitoring of vital signs and early detection of health issues."
    },
    {
        "title": "### AI in Personalized Medicine",
        "describe": "Please provide a detailed description of how artificial intelligence is used in personalized medicine, including its role in analyzing patient data to tailor treatments and predict outcomes."
    },
    {
        "title": "### AI in Telemedicine",
        "describe": "Please provide a detailed description of how artificial intelligence is used in telemedicine, including its applications in remote consultations, virtual diagnoses, and digital health records."
    },
    {
        "title": "## AI in Medical Ethics and Policy",
        "describe": "Please provide a detailed description of the ethical and policy considerations surrounding the use of artificial intelligence in the medical field, including issues related to data privacy, bias, and accountability."
    }
]'
Source code in lazyllm/components/formatter/formatterbase.py 
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
class EmptyFormatter(LazyLLMFormatterBase):
    """This type is the system default formatter. When the user does not specify anything or does not want to format the model output, this type is selected. The model output will be in the same format.


Examples:
    >>> import lazyllm
    >>> from lazyllm.components import EmptyFormatter
    >>> toc_prompt='''
    ... You are now an intelligent assistant. Your task is to understand the user's input and convert the outline into a list of nested dictionaries. Each dictionary contains a `title` and a `describe`, where the `title` should clearly indicate the level using Markdown format, and the `describe` is a description and writing guide for that section.
    ... 
    ... Please generate the corresponding list of nested dictionaries based on the following user input:
    ... 
    ... Example output:
    ... [
    ...     {
    ...         "title": "# Level 1 Title",
    ...         "describe": "Please provide a detailed description of the content under this title, offering background information and core viewpoints."
    ...     },
    ...     {
    ...         "title": "## Level 2 Title",
    ...         "describe": "Please provide a detailed description of the content under this title, giving specific details and examples to support the viewpoints of the Level 1 title."
    ...     },
    ...     {
    ...         "title": "### Level 3 Title",
    ...         "describe": "Please provide a detailed description of the content under this title, deeply analyzing and providing more details and data support."
    ...     }
    ... ]
    ... User input is as follows:
    ... '''
    >>> query = "Please help me write an article about the application of artificial intelligence in the medical field."
    >>> m = lazyllm.TrainableModule("internlm2-chat-20b").prompt(toc_prompt).start()  # the model output without specifying a formatter
    >>> ret = m(query, max_new_tokens=2048)
    >>> print(f"ret: {ret!r}")
    'Based on your user input, here is the corresponding list of nested dictionaries:
    [
        {
            "title": "# Application of Artificial Intelligence in the Medical Field",
            "describe": "Please provide a detailed description of the application of artificial intelligence in the medical field, including its benefits, challenges, and future prospects."
        },
        {
            "title": "## AI in Medical Diagnosis",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical diagnosis, including specific examples of AI-based diagnostic tools and their impact on patient outcomes."
        },
        {
            "title": "### AI in Medical Imaging",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical imaging, including the advantages of AI-based image analysis and its applications in various medical specialties."
        },
        {
            "title": "### AI in Drug Discovery and Development",
            "describe": "Please provide a detailed description of how artificial intelligence is used in drug discovery and development, including the role of AI in identifying potential drug candidates and streamlining the drug development process."
        },
        {
            "title": "## AI in Medical Research",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical research, including its applications in genomics, epidemiology, and clinical trials."
        },
        {
            "title": "### AI in Genomics and Precision Medicine",
            "describe": "Please provide a detailed description of how artificial intelligence is used in genomics and precision medicine, including the role of AI in analyzing large-scale genomic data and tailoring treatments to individual patients."
        },
        {
            "title": "### AI in Epidemiology and Public Health",
            "describe": "Please provide a detailed description of how artificial intelligence is used in epidemiology and public health, including its applications in disease surveillance, outbreak prediction, and resource allocation."
        },
        {
            "title": "### AI in Clinical Trials",
            "describe": "Please provide a detailed description of how artificial intelligence is used in clinical trials, including its role in patient recruitment, trial design, and data analysis."
        },
        {
            "title": "## AI in Medical Practice",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical practice, including its applications in patient monitoring, personalized medicine, and telemedicine."
        },
        {
            "title": "### AI in Patient Monitoring",
            "describe": "Please provide a detailed description of how artificial intelligence is used in patient monitoring, including its role in real-time monitoring of vital signs and early detection of health issues."
        },
        {
            "title": "### AI in Personalized Medicine",
            "describe": "Please provide a detailed description of how artificial intelligence is used in personalized medicine, including its role in analyzing patient data to tailor treatments and predict outcomes."
        },
        {
            "title": "### AI in Telemedicine",
            "describe": "Please provide a detailed description of how artificial intelligence is used in telemedicine, including its applications in remote consultations, virtual diagnoses, and digital health records."
        },
        {
            "title": "## AI in Medical Ethics and Policy",
            "describe": "Please provide a detailed description of the ethical and policy considerations surrounding the use of artificial intelligence in the medical field, including issues related to data privacy, bias, and accountability."
        }
    ]'
    >>> m = lazyllm.TrainableModule("internlm2-chat-20b").formatter(EmptyFormatter()).prompt(toc_prompt).start()  # the model output of the specified formatter
    >>> ret = m(query, max_new_tokens=2048)
    >>> print(f"ret: {ret!r}")
    'Based on your user input, here is the corresponding list of nested dictionaries:
    [
        {
            "title": "# Application of Artificial Intelligence in the Medical Field",
            "describe": "Please provide a detailed description of the application of artificial intelligence in the medical field, including its benefits, challenges, and future prospects."
        },
        {
            "title": "## AI in Medical Diagnosis",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical diagnosis, including specific examples of AI-based diagnostic tools and their impact on patient outcomes."
        },
        {
            "title": "### AI in Medical Imaging",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical imaging, including the advantages of AI-based image analysis and its applications in various medical specialties."
        },
        {
            "title": "### AI in Drug Discovery and Development",
            "describe": "Please provide a detailed description of how artificial intelligence is used in drug discovery and development, including the role of AI in identifying potential drug candidates and streamlining the drug development process."
        },
        {
            "title": "## AI in Medical Research",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical research, including its applications in genomics, epidemiology, and clinical trials."
        },
        {
            "title": "### AI in Genomics and Precision Medicine",
            "describe": "Please provide a detailed description of how artificial intelligence is used in genomics and precision medicine, including the role of AI in analyzing large-scale genomic data and tailoring treatments to individual patients."
        },
        {
            "title": "### AI in Epidemiology and Public Health",
            "describe": "Please provide a detailed description of how artificial intelligence is used in epidemiology and public health, including its applications in disease surveillance, outbreak prediction, and resource allocation."
        },
        {
            "title": "### AI in Clinical Trials",
            "describe": "Please provide a detailed description of how artificial intelligence is used in clinical trials, including its role in patient recruitment, trial design, and data analysis."
        },
        {
            "title": "## AI in Medical Practice",
            "describe": "Please provide a detailed description of how artificial intelligence is used in medical practice, including its applications in patient monitoring, personalized medicine, and telemedicine."
        },
        {
            "title": "### AI in Patient Monitoring",
            "describe": "Please provide a detailed description of how artificial intelligence is used in patient monitoring, including its role in real-time monitoring of vital signs and early detection of health issues."
        },
        {
            "title": "### AI in Personalized Medicine",
            "describe": "Please provide a detailed description of how artificial intelligence is used in personalized medicine, including its role in analyzing patient data to tailor treatments and predict outcomes."
        },
        {
            "title": "### AI in Telemedicine",
            "describe": "Please provide a detailed description of how artificial intelligence is used in telemedicine, including its applications in remote consultations, virtual diagnoses, and digital health records."
        },
        {
            "title": "## AI in Medical Ethics and Policy",
            "describe": "Please provide a detailed description of the ethical and policy considerations surrounding the use of artificial intelligence in the medical field, including issues related to data privacy, bias, and accountability."
        }
    ]'
    """
    def _parse_py_data_by_formatter(self, msg: str):
        return msg

ComponentBase

lazyllm.components.core.ComponentBase

Bases: object

Base class for components, providing a unified interface and basic implementation to facilitate creation of various components.
Components execute tasks via a specified launcher and support custom task execution logic.

Parameters:

  • launcher (LazyLLMLaunchersBase or type, default: empty() ) –

    Launcher instance or launcher class used by the component, defaults to empty launcher.

Examples:

>>> from lazyllm.components.core import ComponentBase
>>> class MyComponent(ComponentBase):
...     def apply(self, x):
...         return x * 2
>>> comp = MyComponent()
>>> comp.name = "ExampleComponent"
>>> print(comp.name)
ExampleComponent
>>> result = comp(10)
>>> print(result)
20
>>> print(comp.apply(5))
10
Source code in lazyllm/components/core.py 
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
class ComponentBase(object, metaclass=LazyLLMRegisterMetaClass):
    """Base class for components, providing a unified interface and basic implementation to facilitate creation of various components.  
Components execute tasks via a specified launcher and support custom task execution logic.

Args:
    launcher (LazyLLMLaunchersBase or type, optional): Launcher instance or launcher class used by the component, defaults to empty launcher.


Examples:
    >>> from lazyllm.components.core import ComponentBase
    >>> class MyComponent(ComponentBase):
    ...     def apply(self, x):
    ...         return x * 2
    >>> comp = MyComponent()
    >>> comp.name = "ExampleComponent"
    >>> print(comp.name)
    ExampleComponent
    >>> result = comp(10)
    >>> print(result)
    20
    >>> print(comp.apply(5))
    10
    """
    def __init__(self, *, launcher=launchers.empty()):  # noqa B008
        self._llm_name = None
        self.job = ReadOnlyWrapper()
        if isinstance(launcher, LazyLLMLaunchersBase):
            self._launcher = launcher
        elif isinstance(launcher, type) and issubclass(launcher, LazyLLMLaunchersBase):
            self._launcher = launcher()
        else:
            raise RuntimeError('Invalid launcher given:', launcher)

    def apply():
        """Core execution method of the component, to be implemented by subclasses.  
Defines the specific business logic or task execution steps of the component.

**Note:**  
If this method is overridden by the subclass, it will be called when the component is invoked.
"""
        raise NotImplementedError('please implement function \'apply\'')

    def cmd(self, *args, **kw) -> Union[str, tuple, list]:
        """Generates the execution command of the component, to be implemented by subclasses.  
The returned command can be a string, tuple, or list, representing the instruction to execute the task.

**Note:**  
If the `apply` method is not overridden, this command will be used to create a job for the launcher to run.
"""
        raise NotImplementedError('please implement function \'cmd\'')

    @property
    def name(self): return self._llm_name
    @name.setter
    def name(self, name): self._llm_name = name

    @property
    def launcher(self): return self._launcher

    def _get_job_with_cmd(self, *args, **kw):
        cmd = self.cmd(*args, **kw)
        cmd = cmd if isinstance(cmd, LazyLLMCMD) else LazyLLMCMD(cmd)
        return self._launcher.makejob(cmd=cmd)

    def _overwrote(self, f):
        return getattr(self.__class__, f) is not getattr(__class__, f) or \
            getattr(self.__class__, '__reg_overwrite__', None) == f

    def __call__(self, *args, **kw):
        if self._overwrote('apply'):
            assert not self._overwrote('cmd'), (
                'Cannot overwrite \'cmd\' and \'apply\' in the same class')
            assert isinstance(self._launcher, launchers.Empty), 'Please use EmptyLauncher instead.'
            return self._launcher.launch(self.apply, *args, **kw)
        else:
            job = self._get_job_with_cmd(*args, **kw)
            self.job.set(job)
            return self._launcher.launch(job)

    def __repr__(self):
        return lazyllm.make_repr('lazyllm.llm.' + self.__class__._lazy_llm_group,
                                 self.__class__.__name__, name=self.name)

apply()

Core execution method of the component, to be implemented by subclasses.
Defines the specific business logic or task execution steps of the component.

Note:
If this method is overridden by the subclass, it will be called when the component is invoked.

Source code in lazyllm/components/core.py 
40
41
42
43
44
45
46
47
    def apply():
        """Core execution method of the component, to be implemented by subclasses.  
Defines the specific business logic or task execution steps of the component.

**Note:**  
If this method is overridden by the subclass, it will be called when the component is invoked.
"""
        raise NotImplementedError('please implement function \'apply\'')

cmd(*args, **kw)

Generates the execution command of the component, to be implemented by subclasses.
The returned command can be a string, tuple, or list, representing the instruction to execute the task.

Note:
If the apply method is not overridden, this command will be used to create a job for the launcher to run.

Source code in lazyllm/components/core.py 
49
50
51
52
53
54
55
56
    def cmd(self, *args, **kw) -> Union[str, tuple, list]:
        """Generates the execution command of the component, to be implemented by subclasses.  
The returned command can be a string, tuple, or list, representing the instruction to execute the task.

**Note:**  
If the `apply` method is not overridden, this command will be used to create a job for the launcher to run.
"""
        raise NotImplementedError('please implement function \'cmd\'')