How to combine vad, asr, punc and nnlm models inside pipeline

[LauraGPT]

You could combine the VAD, ASR, NNLM and PUNC models freely in pipeline. The following is the demo of Paraformer-large.

The pipeline which only contains ASR model：

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
)

The pipeline which contains NNLM and ASR models：

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    lm_model='damo/speech_transformer_lm_zh-cn-common-vocab8404-pytorch',
    lm_weight=0.15,
    beam_size=10,
)

The pipeline which contains VAD and ASR models：

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
)

The pipeline which contains VAD, ASR and PUNC models：

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
    punc_model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',
)

The pipeline which contains VAD, ASR, PUNC and NNLM models：

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
    punc_model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',
    lm_model='damo/speech_transformer_lm_zh-cn-common-vocab8404-pytorch',
    lm_weight=0.15,
    beam_size=10,
)

[LauraGPT]

Notes：

For the Paraformer-large-long, the pipeline defaults to combine VAD, ASR and PUNC models. For example, the following pipeline decodes with VAD, ASR and PUNC models.

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
)

It equals to

    inference_pipeline = pipeline(
        task=Tasks.auto_speech_recognition,
        model='damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
        vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
        punc_model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',
    )

If you want to disable the PUNC, just to set punc_model='':

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    punc_model='',
)

[mattzheng]

报错：TypeError: 'NoneType' object is not callable

[LumenForgeX]

对于读取本地模型，如
inference_pipeline = pipeline(
task=Tasks.auto_speech_recognition,
model='/root/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
vad_model='/root/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
punc_model='/root/damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',）
只有model会生效，vad_model和punc_model不生效，仍读取hub模型。

[LauraGPT]

Yeah, it is a bug. We would fix it.

[hildazzz]

长达7分钟的干净语音文件，使用：

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
    vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
    punc_model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',
)

没有输出time_stamp，这是为什么呢？sentences的输出也是空的。我希望得到时间戳和对应的文本内容（带有标点符号），谢谢。

[LauraGPT]

You could generate timestamp by:

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch',
)

[hildazzz]

RuntimeError: AutomaticSpeechRecognitionPipeline: cuDNN version incompatibility: PyTorch was compiled against (8, 5, 0) but found runtime version (8, 2, 1). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN.Looks like your LD_LIBRARY_PATH contains incompatible version of cudnnPlease either remove it from the path or install cudnn (8, 5, 0)

按照教程使用，报错如上，请问是哪里有问题？

[LumenForgeX]

You can use this command in you terminal-----> unset LD_LIBRARY_PATH

[LauraGPT]

Update for current FunASR 1.3.x: the 2023 modelscope.pipeline(...) examples above describe the legacy API. For the current Python pipeline, use funasr.AutoModel.

Current VAD + ASR + punctuation pipeline

Model IDs and complete local directories are both supported. When using local models, each argument must point to its own complete model directory containing config.yaml, model.pt, and the associated tokenizer/frontend files:

from funasr import AutoModel

model = AutoModel(
    model="/models/paraformer-zh",
    vad_model="/models/fsmn-vad",
    punc_model="/models/ct-punc",
    device="cpu",
    disable_update=True,
)

result = model.generate(
    input="/audio/example.wav",
    batch_size_s=300,
    return_raw_text=True,
    sentence_timestamp=True,
)[0]

print(result["text"])
print(result["timestamp"])
print(result["sentence_info"])

The current AutoModel builds vad_model and punc_model independently, so three valid local paths do not fall back to Hub models. The local-path loader also skips downloading when the supplied directory exists. See the current model construction and local-path resolution.

I verified this with FunASR 1.3.14 / PyTorch 2.11 on CPU while all network proxies pointed to an unavailable local port. The main ASR, VAD, and punctuation model_path values resolved to the three supplied directories, and the official sample produced:

text: 欢迎大家来到么哒社区进行体验。
timestamp count: 14
first timestamp: [1070, 1310]
last timestamp: [4130, 4365]
sentence_info: [{"text": "欢迎大家来到么哒社区进行体验。", "start": 1070, "end": 4365, ...}]

Timestamp field names

The current output field is timestamp, not the legacy time_stamp. It contains character/word [start_ms, end_ms] pairs in milliseconds.

sentence_info is not enabled by default. With the same loaded models:

• default call: 14 timestamp entries and 0 sentence_info entries;
• sentence_timestamp=True: the same 14 timestamps plus one sentence from 1,070 to 4,365 ms.

Sentence segmentation needs punc_model, because punctuation boundaries are used to group the token timestamps. This behavior is implemented in the current VAD/ASR/punctuation merge and sentence-timestamp path. With vad_model, long audio is segmented internally and each ASR timestamp is shifted back to the original audio timeline; manual slicing is not required.

NNLM / language-model decoding

Do not copy the old lm_model=..., lm_weight=... example into current AutoModel. The current constructor creates the main model plus optional VAD, punctuation, and speaker models; it does not construct a separate lm_model scorer. The current Paraformer Python beam-search initialization also leaves the N-gram scorer unset, so the legacy Python NNLM shallow-fusion example is not a supported current path.

If you need language-model decoding today, use the C++/ONNX 2-pass runtime, which documents an N-gram WFST model through --lm-dir, together with --model-dir, --online-model-dir, --vad-dir, and --punc-dir: runtime command and parameter contract.

The current non-streaming Python example is also maintained in the FunASR tutorial.