how to use vad, asr and punc model by pipeline

[LauraGPT]

how to split wav by vad model

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks
from modelscope.utils.logger import get_logger
import logging
logger = get_logger(log_level=logging.CRITICAL)
logger.setLevel(logging.CRITICAL)

import soundfile

waveform, sample_rate = soundfile.read("/home/zhifu.gzf/.cache/modelscope/hub/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/example/vad_example.wav")
# fs = 16000

# vad pipeline
inference_pipeline_vad = pipeline(
    task=Tasks.voice_activity_detection,
    model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
    model_revision=None,
)
# asr pipeline
inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch')
# punc pipeline
inference_pipeline_punc = pipeline(
    task=Tasks.punctuation,
    model='damo/punc_ct-transformer_zh-cn-common-vad_realtime-vocab272727',
    model_revision=None,
)

segments_result = inference_pipeline_vad(audio_in=waveform)
param_punc_dict = {"cache": []}
for i, segments in enumerate(segments_result["text"]):
    beg_idx = segments[0] * sample_rate/1000
    end_idx = segments[1] * sample_rate/1000
    waveform_slice = waveform[int(beg_idx):int(end_idx)]
    result_segments = inference_pipeline(audio_in=waveform_slice)
    if result_segments != []:
        result_segments_withpunc = inference_pipeline_punc(text_in=result_segments['text'], param_dict=param_punc_dict)
        print(result_segments_withpunc['text'])
    else:
        print(result_segments)
    

[AnitaSherry]

how to split wav by vad model

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks
from modelscope.utils.logger import get_logger
import logging
logger = get_logger(log_level=logging.CRITICAL)
logger.setLevel(logging.CRITICAL)

import soundfile

waveform, sample_rate = soundfile.read("/home/zhifu.gzf/.cache/modelscope/hub/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/example/vad_example.wav")
# fs = 16000

inference_pipeline_vad = pipeline(
    task=Tasks.voice_activity_detection,
    model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
    model_revision=None,
)
inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch')

segments_result = inference_pipeline_vad(audio_in=waveform)
for i, segments in enumerate(segments_result["text"]):
    beg_idx = segments[0] * sample_rate/1000
    end_idx = segments[1] * sample_rate/1000
    waveform_slice = waveform[int(beg_idx):int(end_idx)]
    result_segments = inference_pipeline(audio_in=waveform_slice)
    print(result_segments)
    

for i, segments in enumerate(segments_result["text"]):
这一行里，segments_result["text"]是字符串，应该加一个eval()

[LauraGPT]

how to split wav by vad model

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks
from modelscope.utils.logger import get_logger
import logging
logger = get_logger(log_level=logging.CRITICAL)
logger.setLevel(logging.CRITICAL)

import soundfile

waveform, sample_rate = soundfile.read("/home/zhifu.gzf/.cache/modelscope/hub/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/example/vad_example.wav")
# fs = 16000

inference_pipeline_vad = pipeline(
    task=Tasks.voice_activity_detection,
    model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',
    model_revision=None,
)
inference_pipeline = pipeline(
    task=Tasks.auto_speech_recognition,
    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch')

segments_result = inference_pipeline_vad(audio_in=waveform)
for i, segments in enumerate(segments_result["text"]):
    beg_idx = segments[0] * sample_rate/1000
    end_idx = segments[1] * sample_rate/1000
    waveform_slice = waveform[int(beg_idx):int(end_idx)]
    result_segments = inference_pipeline(audio_in=waveform_slice)
    print(result_segments)
    

for i, segments in enumerate(segments_result["text"]): 这一行里，segments_result["text"]是字符串，应该加一个eval()

I have changed the code, Please git pull and reinstall

[wwfcnu]

inference_pipeline(audio_in=waveform_slice)支持batch输入吗

[LauraGPT]

Update for current FunASR (main at 1d8080a, verified on 2026-07-15).

You no longer need three separate ModelScope pipelines, manual VAD slicing, or eval(). Current FunASR's AutoModel performs the complete offline chain:

1. VAD finds speech segments.
2. Paraformer decodes the detected segments.
3. Segment-local timestamps are shifted back onto the original audio timeline.
4. The merged transcript is passed through the punctuation model.

Current VAD + ASR + punctuation example

from funasr import AutoModel

model = AutoModel(
    model="paraformer-zh",
    vad_model="fsmn-vad",
    vad_kwargs={"max_single_segment_time": 60000},
    punc_model="ct-punc",
    device="cpu",  # use "cuda" when available
    disable_update=True,
)

wav_file = f"{model.model_path}/example/asr_example.wav"
results = model.generate(
    input=wav_file,
    batch_size_s=300,
    return_raw_text=True,
    sentence_timestamp=True,
)

result = results[0]
print(result["text"])
print(result["raw_text"])
print(result["timestamp"])
print(result["sentence_info"])

The return value is already structured Python data. Never run eval() on VAD or ASR output.

With the bundled sample and the environment listed below, the verified result was:

text: 欢迎大家来到么哒社区进行体验。
raw_text: 欢 迎 大 家 来 到 么 哒 社 区 进 行 体 验
timestamp count: 14
first timestamp: [1070, 1310]
last timestamp: [4130, 4365]
sentence_info: start=1070, end=4365, text=欢迎大家来到么哒社区进行体验。

timestamp contains character/word boundaries in milliseconds on the original input-audio timeline, after FunASR has restored each VAD segment's offset. sentence_info.start and .end are also milliseconds.

Batch input

The current Python API accepts a list of file paths or audio arrays and returns one result dictionary per input:

files = ["meeting-a.wav", "meeting-b.wav"]
results = model.generate(
    input=files,
    batch_size_s=300,
    return_raw_text=True,
    sentence_timestamp=True,
)

assert len(results) == len(files)
for path, result in zip(files, results):
    print(path, result["text"])

I tested the same bundled file once as a single input and twice as input=[wav_file, wav_file]. The batch returned two independent dictionaries, and both matched the single-input punctuation text, raw text, and all 14 timestamps exactly.

batch_size_s is the duration budget used to schedule VAD-produced speech segments; it is not the number of input files and it does not split one file into that many pieces. On CPU, segment decoding may run sequentially even though list input is supported.

If you explicitly need VAD-only output

VAD-only results are also structured lists now:

import soundfile as sf

vad = AutoModel(model="fsmn-vad", device="cpu", disable_update=True)
waveform, sample_rate = sf.read(wav_file)
segments = vad.generate(input=wav_file)[0]["value"]

for begin_ms, end_ms in segments:
    begin_sample = int(begin_ms * sample_rate / 1000)
    end_sample = int(end_ms * sample_rate / 1000)
    waveform_slice = waveform[begin_sample:end_sample]

Again, no eval() is needed. For ordinary long-file transcription, prefer the integrated vad_model / punc_model constructor because it also handles segment ordering, timestamp offset restoration, and final punctuation (current implementation). The maintained example is in the FunASR tutorial.

The reproduction used Python 3.12.3, FunASR 1.3.14, ModelScope 1.38.1, PyTorch 2.11.0 CPU, and the current paraformer-zh, fsmn-vad, and ct-punc aliases.