asr

Transcribe mixed audio into timed captions. Wire the ASR node into sinks that support subtitles (segment, rtmp_push, RTC sinks, etc.).

Example

{
  "name": "captions",
  "type": "asr",
  "inputs": ["mix_audio"],
  "config": {
    "language": "en-US"
  }
}

Config

Field	Type	Default	Description
language	string	managed default	Optional BCP-47 recognition hint, such as en-US or zh-CN. When omitted, the managed service selects its language strategy.

Output format

ASR does not produce video or audio streams. It emits ASR text events as avflow.asrText DataEvents. Topic-based sinks use type as the topic and send data; sinks without topics send the whole { type, data } object.

Field	Type	Description
text	string	Transcribed text for this utterance
startMs	number	Start time (ms from job start)
endMs	number	End time (ms from job start)
speaker	string	Optional display label for the speaker
identity	string	Optional participant / source identity
isFinal	boolean	true when the utterance is committed; false for interim partials

{
  "type": "avflow.asrText",
  "data": {
    "text": "Welcome to the show.",
    "startMs": 12400,
    "endMs": 15800,
    "speaker": "Alice",
    "identity": "user_123",
    "isFinal": true
  }
}

When audio passes through an audio_mixer, the speaker is resolved from the active-speaker timeline. With per-track ASR (no mixer), it comes from the participant identity.

The managed ASR service may emit interim results (isFinal: false) while a phrase is still being spoken. Downstream sinks decide whether to forward or keep only finals — see below.

Downstream delivery

Wire ASR into a sink’s inputs alongside your audio/video mixers. The caption fields stay the same across sinks; the transport envelope differs:

Sink	How captions are delivered
segment	HLS WebVTT subtitle track (.vtt); only isFinal: true cues are written. Use segment caption.showSpeaker for voice tags and segment caption.textSplit for shorter display cues.
livekit	Reliable publishData with topic avflow.asrText; body is the data object.
jitsi	Endpoint message { "type": "avflow.asrText", "data": { ... } }.
daily	App message { "type": "avflow.asrText", "data": { ... } }.
agora	Data-stream message { "type": "avflow.asrText", "data": { ... } }.
rtmp_push	H.26x user_data_unregistered SEI or AV1 Metadata OBU.
srt_push	H.26x user_data_unregistered SEI or AV1 Metadata OBU.
whip	H.26x user_data_unregistered SEI or AV1 Metadata OBU before RTP packetization.

For these in-band sinks, VP8/VP9 discard text metadata and a video input is required. AVFlow does not create an implicit carrier; use an explicit video_generator when needed. segment is the only sink that supports captions without video. image does not accept caption input.

H.26x consumers identify user_data_unregistered payloads by UUID 4156464c-4f57-5445-5854-4d4554410001 (AVFLOWTEXTMETA). AV1 uses METADATA_TYPE_ITUT_T35, country code 0xFF, extension 0x00, identifier AV, then the same UUID and JSON body. The UUID is fixed for all avflow in-band text metadata; the JSON body is an envelope: { "type": "avflow.asrText" | "avflow.voiceAgentText" | "avflow.audioLevels" | "avflow.videoMixerLayout", "data": { ... } }. For ASR, data uses the caption event fields shown above (text, startMs, endMs, speaker, identity, isFinal).

The metadata survives container remuxing when the encoded video is copied. CDN or transcoder products that fully decode and re-encode video must be configured to preserve unregistered SEI/ITU-T T.35 metadata; arbitrary transcoders are not required by the codec specifications to copy it.

Wiring

room_src ──► audio_mixer ──► segment
                         └──► asr ──┘

Pricing

$0.012/min per minute of ASR-processed audio (asrProcessedMillis). Long silence removed by AVFlow VAD is not billed; short preroll and hangover around speech are included to avoid clipped words. Node pricing.