README.md

mediadecode-ffmpeg

FFmpeg adapter for the mediadecode abstraction layer, built on top of ffmpeg-next.

Implements mediadecode's VideoAdapter / AudioAdapter / SubtitleAdapter traits and the matching push-style *StreamDecoder traits. Frame payloads are zero-copy refcounted views over FFmpeg's AVBufferRef via the [FfmpegBuffer] type — receiving a frame does not memcpy the pixel data.

FfmpegVideoStreamDecoder mirrors the send_packet / receive_frame shape of ffmpeg::decoder::Video, auto-probes the host's HW backends, and falls through to a software decoder when none open. Audio and subtitles use parallel FfmpegAudioStreamDecoder / FfmpegSubtitleStreamDecoder types.

Backends

FfmpegVideoStreamDecoder::open walks this probe order, opening the first backend that accepts the stream:

Target	Probe order
macOS / iOS / tvOS	VideoToolbox → software
Linux	VAAPI → CUDA → software
Windows	D3D11VA → CUDA → software
other	software

Output frames are CPU-side, downloaded with av_hwframe_transfer_data (NV12 for 8-bit, P010/P012/P016/P210/P212/P216/P410/P412/P416 for 10/12/16-bit). Pixel-format conversion is intentionally out of scope — downstream colconv handles it.

If every HW backend opens but later fails at decode time and the software backend is also unavailable, the error surfaces as VideoDecodeError::Decode(Error::AllBackendsFailed(p)) carrying any packets the decoder had already accepted from the demuxer (accessible via p.unconsumed_packets() / p.into_unconsumed_packets()) — so non-seekable callers (live streams, pipes, network sources) can replay them through their own software decoder without re-demuxing.

Usage

use ffmpeg_next as ffmpeg;
use ffmpeg::{format, media};
use mediadecode::{Timebase, decoder::VideoStreamDecoder};
use mediadecode_ffmpeg::{
  Error as FfmpegError, FfmpegVideoStreamDecoder, VideoDecodeError,
  empty_video_frame, video_packet_from_ffmpeg,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
  ffmpeg::init()?;

  let path = std::env::args().nth(1).expect("usage: <input-file>");
  let mut input = format::input(&path)?;
  let stream = input.streams().best(media::Type::Video).unwrap();
  let stream_index = stream.index();
  let time_base = Timebase::new(
    stream.time_base().numerator() as u32,
    std::num::NonZeroU32::new(stream.time_base().denominator() as u32).unwrap(),
  );

  // Probes HW backends in order, falls back to software.
  let mut decoder = match FfmpegVideoStreamDecoder::open(stream.parameters(), time_base) {
    Ok(d) => d,
    Err(FfmpegError::AllBackendsFailed(p)) => {
      // No backend at all could open this stream — including software.
      // `unconsumed_packets` is empty at open-time. Caller decides.
      let _unconsumed_packets = p.into_unconsumed_packets();
      return Ok(());
    }
    Err(e) => return Err(e.into()),
  };

  let mut frame = empty_video_frame();
  for (s, av_packet) in input.packets() {
    if s.index() != stream_index { continue; }
    let Some(pkt) = video_packet_from_ffmpeg(&av_packet) else { continue };

    match decoder.send_packet(&pkt) {
      Ok(()) => {}
      Err(VideoDecodeError::Decode(FfmpegError::AllBackendsFailed(p))) => {
        // Runtime exhaustion: rescued packets are the bytes the decoder
        // already consumed from `input`. Replay them through your own
        // software decoder before the current packet so non-seekable
        // sources recover cleanly.
        let _unconsumed_packets = p.into_unconsumed_packets();
        return Ok(());
      }
      Err(e) => return Err(e.into()),
    }
    while decoder.receive_frame(&mut frame).is_ok() {
      // frame.pixel_format(), frame.width(), frame.height(),
      // frame.planes() — zero-copy views over AVBufferRef.
    }
  }
  decoder.send_eof()?;
  while decoder.receive_frame(&mut frame).is_ok() { /* drain */ }
  Ok(())
}

Audio and subtitle decoding share the shape — see examples/decode_via_trait.rs and tests/audio_subtitle_via_trait.rs for end-to-end demuxer-driven runs that cover all three streams.

Public surface map

• Decoders: FfmpegVideoStreamDecoder, FfmpegAudioStreamDecoder, FfmpegSubtitleStreamDecoder. Plus their error types: VideoDecodeError, AudioDecodeError, SubtitleDecodeError.
• Type aliases: VideoPacket, AudioPacket, SubtitlePacket, VideoFrame, AudioFrame, SubtitleFrame — the mediadecode generic types pre-parameterized with this crate's adapter / buffer / extras, so you don't have to spell them out.
• Buffer: FfmpegBuffer — refcounted view over an AVBufferRef with safe constructors (empty, from_packet, try_* panic-free counterparts).
• Boundary helpers: video_packet_from_ffmpeg, audio_packet_from_ffmpeg, subtitle_packet_from_ffmpeg — convert a borrowed ffmpeg::Packet into the matching mediadecode packet without copying the compressed payload.
• Empty-frame builders: empty_video_frame, empty_audio_frame, empty_subtitle_frame — well-formed destinations for receive_frame.

Running tests and benches

The integration test and benchmark expect a real video file. Set MEDIADECODE_SAMPLE_VIDEO to enable them:

MEDIADECODE_SAMPLE_VIDEO=/path/to/clip.mp4 cargo test
MEDIADECODE_SAMPLE_VIDEO=/path/to/clip.mp4 cargo test --test hw_smoke -- --ignored
MEDIADECODE_SAMPLE_VIDEO=/path/to/clip.mp4 cargo bench

Without the env var the integration tests skip with a notice; unit tests run unconditionally.

Build requirements

• A system FFmpeg ≥ 5.1 linkable via pkg-config (we reference AV_PIX_FMT_P212LE / AV_PIX_FMT_P412LE, which were added in 5.1). Tested against 8.1. Verify with ffmpeg -hwaccels that your build has the backends you expect compiled in (e.g. videotoolbox on macOS, vaapi / cuda on Linux, d3d11va / cuda on Windows).
• Rust ≥ 1.95, edition 2024.

License

mediadecode-ffmpeg is under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE, LICENSE-MIT for details.

Copyright (c) 2026 FinDIT Studio authors.