How to use VAD decoding in FunASR

[lingyunfly]

There are two decoding mode for VAD model(offline、 online)，for more details, please refer to VAD.

If you want to decode in offline mode

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

if __name__ == '__main__':
    audio_in = 'https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav'
    output_dir = None
    inference_pipline = pipeline(
        task=Tasks.voice_activity_detection,
        model="damo/speech_fsmn_vad_zh-cn-16k-common-pytorch",
        model_revision='v1.2.0',
        output_dir=output_dir,
        batch_size=1,
    )
    segments_result = inference_pipline(audio_in=audio_in)
    print(segments_result)

If you want to decode in online mode, you should set the mode to online. The step means the package size send each time. The is_final means whether it is the last package of input data. When a new request is started, you should set the param_dict = {'in_cache': dict()}

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks
from modelscope.utils.logger import get_logger
import logging
logger = get_logger(log_level=logging.CRITICAL)
logger.setLevel(logging.CRITICAL)
import soundfile
import os


if __name__ == '__main__':
    os.environ["MODELSCOPE_CACHE"] = "./"
    output_dir = None
    inference_pipline = pipeline(
        task=Tasks.voice_activity_detection,
        model="damo/speech_fsmn_vad_zh-cn-16k-common-pytorch",
        model_revision='v1.2.0',
        output_dir=output_dir,
        batch_size=1,
        mode='online',
    )
    model_dir = os.path.join(os.environ["MODELSCOPE_CACHE"], "damo/speech_fsmn_vad_zh-cn-16k-common-pytorch")
    wav_file = os.path.join(model_dir, "example/vad_example.wav")
    speech, sample_rate = soundfile.read(wav_file)
    speech_length = speech.shape[0]
    
    sample_offset = 0
    step = 160 * 10
    param_dict = {'in_cache': dict()}
    for sample_offset in range(0, speech_length, min(step, speech_length - sample_offset)):
        if sample_offset + step >= speech_length - 1:
            step = speech_length - sample_offset
            is_final = True
        else:
            is_final = False
        param_dict['is_final'] = is_final
        segments_result = inference_pipline(audio_in=speech[sample_offset: sample_offset + step],
                                            param_dict=param_dict)
        print(segments_result)

[walkeronwater]

Hi, could you please give a short example to refresh timestamp cache that calling an online-mode VAD returns?

"When a new request is started, you should set the param_dict = {'in_cache': dict()}"
-- I've tried this method but got accumulated timestamps that never refreshed.

[LauraGPT]

Update for current FunASR (main at 1d8080a, verified on 2026-07-15).

The important distinction is:

• Within one streaming request, reuse the same cache. Timestamps are absolute milliseconds from the start of that request, so they are expected to increase across chunks.
• Between independent requests, use a new cache dictionary. Do not share one cache across recordings, users, or WebSocket connections.

The original post uses the historical ModelScope pipeline API. The current native FunASR API is AutoModel.

Non-streaming VAD

from funasr import AutoModel

model = AutoModel(model="fsmn-vad", device="cpu", disable_update=True)
wav_file = f"{model.model_path}/example/vad_example.wav"

result = model.generate(input=wav_file)[0]
print(result["value"])

Each pair is [start_ms, end_ms] relative to the beginning of the input audio.

Streaming VAD with request-scoped cache

import soundfile as sf
from funasr import AutoModel

CHUNK_MS = 200
model = AutoModel(model="fsmn-vad", device="cpu", disable_update=True)
wav_file = f"{model.model_path}/example/vad_example.wav"
speech, sample_rate = sf.read(wav_file)


def decode_one_request(speech, sample_rate):
    # A new dictionary for every independent audio/request/connection.
    cache = {}
    stride = int(CHUNK_MS * sample_rate / 1000)
    total = (len(speech) - 1) // stride + 1
    events = []

    for index in range(total):
        chunk = speech[index * stride : (index + 1) * stride]
        is_final = index == total - 1
        result = model.generate(
            input=chunk,
            cache=cache,
            is_final=is_final,
            chunk_size=CHUNK_MS,
        )[0]["value"]
        if result:
            events.extend(result)

    return events


first = decode_one_request(speech, sample_rate)
second = decode_one_request(speech, sample_rate)
assert first == second
print(first)

Streaming output uses endpoint events:

• [begin_ms, -1] means a speech start was detected.
• [-1, end_ms] means the matching speech end was detected.
• [begin_ms, end_ms] may be returned when both boundaries are available together.
• [] means no new boundary was emitted for that chunk.

For the bundled 70,470.625 ms example, both independent runs returned the same events:

[[70, -1], [-1, 26270], [26770, -1], [-1, 37600],
 [38210, -1], [-1, 59550], [59830, -1], [-1, 70450]]

Why timestamps can appear to "never refresh"

is_final=True is part of the request boundary, not only a hint for printing the last result. In the current implementation it flushes the last VAD state and then reinitializes the cache (source).

I reproduced the accumulated-timestamp symptom by intentionally omitting is_final=True for the first copy of the official audio and then feeding a second copy through the same cache. The unfinished cache retained 7,036 frames, and the second copy emitted timestamps from 92,610 through 140,920 ms instead of restarting near zero. Calling cache.clear() before that second request restored the original 70 through 70,450 ms events exactly.

Therefore:

1. Keep one cache for all chunks of the same stream.
2. Always send the last chunk with is_final=True on a normal close.
3. If a client disconnects before a final chunk, discard that cache (or call cache.clear()) before handling another request.
4. Give concurrent clients separate cache dictionaries; never keep one global VAD cache.

Do not clear the cache after every chunk: that would reset the acoustic/model history and break streaming endpoint detection. The current official streaming example is also available in the FunASR tutorial.

The reproduction above used Python 3.12.3, FunASR 1.3.14, ModelScope 1.38.1, PyTorch 2.11.0 CPU, and the official iic/speech_fsmn_vad_zh-cn-16k-common-pytorch checkpoint and bundled sample.