Encoding and muxing¶

Use MediaSink to open an output file, attach encoders, and write a valid container. Muxing means writing encoded streams into a container file. This is the low-level path underneath Transcoder: you control the encoders, you feed the frames, and you decide when streams open and close.

If you only need file in, file out, use Transcoder.transcode first. Use MediaSink when you need to drive encoders by hand: generated frames, a custom pipeline, or several encoders fed from different sources.

The shape of an output file¶

A MediaSink is a muxer over an open output file. The workflow is always the same three phases, in order:

Open the sink and add every encoder (and every copy stream). The muxer's header freezes the stream list, so nothing can be added once the first frame is written.
Drive each encoder by draining a Flow<Frame> through it. The encoder pushes frames in, pulls packets out, and hands them to the muxer.
Close the sink. This flushes the encoders, writes the container trailer, and releases native resources.

import io.github.yuroyami.kiteffmpeg.MediaSink
import io.github.yuroyami.kiteffmpeg.VideoEncoderSpec
import io.github.yuroyami.kiteffmpeg.CodecId
import io.github.yuroyami.kiteffmpeg.Rational

MediaSink.open("output.mp4").use { sink ->
    val video = sink.addVideoEncoder(
        VideoEncoderSpec(
            codec = CodecId.Mpeg4,
            width = 1280, height = 720,
            frameRate = Rational(30, 1),
            bitrateBps = 4_000_000,
        )
    )
    video.drive(frames)          // frames: Flow<Frame>
}                                // close() writes the trailer

The output format is inferred from the file extension. .mp4, .mkv, .mov and the rest are chosen for you; you never name a muxer.

Add streams before the first frame

addVideoEncoder, addAudioEncoder, addCopyStream, and setMetadata all have to be called before any frame or packet reaches the muxer. The header is written once, on the first write, and it carries the full stream list and the metadata tags. After that the set is frozen.

Adding a video encoder¶

addVideoEncoder(VideoEncoderSpec) configures and opens an encoder, returning a VideoEncoder handle. The spec carries the geometry, the codec, and a free-form options map:

val spec = VideoEncoderSpec(
    codec = CodecId.Mpeg4,
    width = 1920, height = 1080,
    pixelFormat = PixelFormat.Yuv420p,       // default
    frameRate = Rational(30, 1),
    bitrateBps = 6_000_000,                  // default 4_000_000
    options = mapOf("bf" to "2"),            // up to two B-frames between references
)
val encoder = sink.addVideoEncoder(spec)

VideoEncoderSpec fields:

Field	Type	Default	Notes
`codec`	`CodecId`	required	The format: `Mpeg4`, `H264`, `Hevc`, and so on. Which encoder writes it is `encoder`, below.
`width` / `height`	`Int`	required	Output frame size. Match your filter output.
`pixelFormat`	`PixelFormat`	`Yuv420p`	Most codecs want `Yuv420p`.
`frameRate`	`Rational`	required	Exact fraction, e.g. `Rational(30000, 1001)` for 29.97.
`bitrateBps`	`Long`	`4_000_000`	Target bitrate. Ignored when you set `crf`.
`keyframeIntervalFrames`	`Int`	`frameRate × 2`	GOP length. Computed from the frame rate unless you override it.
`options`	`Map<String,String>`	empty	Encoder options set with `av_opt_set`. A key that repeats a field, such as `b` or `pix_fmt`, is refused.
`encoder`	`EncoderId?`	`null`	The encoder that writes `codec`, such as `EncoderId.H264VideoToolbox`. `null` takes the one FFmpeg picks for the format.
`color`	`ColorInfo?`	`null`	The color the output declares: primaries, transfer, matrix, range and chroma location. Every field that is not unspecified is written.
`sampleAspectRatio`	`Rational?`	`null`	The shape of one pixel, for anamorphic video. `null` writes none, which players read as square.
`hdr`	`HdrMetadata?`	`null`	The mastering display and the content light level of HDR video.

A ColorInfo read from a stream or a frame fills what the source did not declare with a guess, such as BT.709 for high definition. Pass its withoutGuesses() to write only what the source declared:

val spec = VideoEncoderSpec(
    codec = CodecId.Hevc,
    encoder = EncoderId.HevcVideoToolbox,
    width = 3840, height = 2160,
    pixelFormat = PixelFormat.P010le,
    frameRate = Rational(24000, 1001),
    color = source.primaryVideo!!.video!!.color.withoutGuesses(),
    hdr = source.primaryVideo!!.video!!.hdr,
)

A raw option that sets the same thing as one of these fields is refused once the field is set, for example color_trc beside color or aspect beside sampleAspectRatio.

Per-encoder options¶

The options map is set on the encoder with av_opt_set. The keys are the FFmpeg option names, so an option that ffmpeg -h encoder=mpeg4 lists for the encoder you use is valid. A key that sets the same thing as a typed field is refused with an FFmpegException that names both: b, bit_rate, g, gop_size, pix_fmt, width, height, video_size, time_base, framerate and r for video, and b, bit_rate, ar, sample_rate, ac, channels, sample_fmt and time_base for audio:

VideoEncoderSpec(
    codec = CodecId.Mpeg4,
    width = 1280, height = 720,
    frameRate = Rational(30, 1),
    options = mapOf(
        "bf" to "2",             // up to two B-frames between references
        "mpeg_quant" to "1",     // MPEG quantisers instead of H.263 ones
    ),
)

crf overrides bitrate

With crf, libx264 and libx265 run in constant-quality mode and ignore bitrateBps. Those two encoders exist only in a GPL FFmpeg, which no published artifact carries, so this applies only to a build you make yourself.

Driving the encoder¶

A VideoEncoder does not take frames one at a time. You hand it a Flow<Frame> and it drains the whole flow:

suspend fun encode(sink: MediaSink, frames: Flow<Frame>) {
    val encoder = sink.addVideoEncoder(spec)
    encoder.drive(
        input = frames,
        onProgress = { count -> println("encoded $count frames") },
        progressEveryNFrames = 30,           // default
    )
}

drive pushes each frame into the encoder, pulls every packet that comes back out, hands those packets to the muxer, and flushes the encoder when the flow completes. It returns when the input flow is exhausted.

The encode core is EAGAIN-correct: it respects the codec's "I need more input before I can give you output" signal instead of busy-looping or dropping frames. You never see EAGAIN; the loop handles it.

Timestamps are handled for you¶

You do not compute output timestamps. The encoder takes each incoming frame's pts (in the frame's own time-base) and rescales it onto the codec time-base. Frames that arrive with no pts at all fall back to a frame counter. Either way the output is monotonic and zero-based: the first written frame lands at pts 0 and timestamps only ever increase.

The codec time-base of a video encoder has one tick per frame at frameRate, and one tick holds one frame. drive throws FFmpegException for a frame whose pts lands on or before the tick of the frame before it, because moving that frame to a later tick would slow the video down. Feed frames at the encoder's rate. When the source is faster or slower, drop or repeat frames first. Transcoder does this for you. See the transcoding guide for how trim offsets are rebased to zero on top of this.

Adding an audio encoder¶

addAudioEncoder(AudioEncoderSpec) mirrors the video path:

import io.github.yuroyami.kiteffmpeg.AudioEncoderSpec

val audio = sink.addAudioEncoder(
    AudioEncoderSpec(
        codec = CodecId.Aac,
        sampleRate = 48_000,     // default 44_100
        channels = 2,            // default 2
        bitrateBps = 192_000,    // default 128_000
        options = mapOf("profile" to "aac_low"),
    )
)
audio.drive(audioFrames)         // audioFrames: Flow<Frame>

AudioEncoderSpec fields:

Field	Type	Default	Notes
`codec`	`CodecId`	required	The format: `Aac`, `Opus`, `Flac`, `Mp3`, `Vorbis`, `PcmS16`, and so on. Which encoder writes it is `encoder`, below.
`sampleRate`	`Int`	`44_100`	Output sample rate in Hz.
`channels`	`Int`	`2`	Channel count.
`sampleFormat`	`SampleFormat`	`None`	`None` lets the encoder pick its preferred format (`fltp` for AAC).
`bitrateBps`	`Long`	`128_000`	Target bitrate.
`options`	`Map<String,String>`	empty	Encoder options set with `av_opt_set`. A key that repeats a field, such as `b` or `ar`, is refused.
`channelLayoutMask`	`Long?`	`null`	The exact layout as an FFmpeg channel mask, when the count is ambiguous. `null` takes FFmpeg's default for `channels`: six channels are 5.1 with back surrounds.
`encoder`	`EncoderId?`	`null`	The encoder that writes `codec`, such as `EncoderId.Aac`. `null` takes the one FFmpeg picks for the format.

Six channels are 5.1 with back surrounds or 5.1 with side surrounds, and the count alone cannot say which. 0x60FL is 5.1 with side surrounds, the layout AC-3 decodes to. A mask whose channel count differs from channels is refused.

After the encoder opens, the AudioEncoder handle exposes the values that were actually negotiated:

val audio = sink.addAudioEncoder(AudioEncoderSpec(codec = CodecId.Aac))
println(audio.sampleFormat)   // resolved from None, e.g. FltP for aac
println(audio.sampleRate)     // 44100
println(audio.channels)       // 2
println(audio.channelLayoutMask) // 3, the stereo mask
println(audio.frameSize)      // 1024 for aac, 0 for codecs taking any chunk size

AAC's 1024-sample framing is handled for you¶

AAC will not accept arbitrary chunk sizes. It wants exactly 1024 samples per frame, every frame, and the encoder rejects anything else. KiteFFmpeg handles this so you do not have to count samples by hand.

The mechanism is frameSize: an opened AudioEncoder reports the samples-per-frame the codec demands (1024 for AAC, 0 for codecs that take any chunk size). Route your audio through an audio filter graph and pin its output to that size:

import io.github.yuroyami.kiteffmpeg.FilterGraph
import io.github.yuroyami.kiteffmpeg.MediaSource

val source = MediaSource.open("input.mp4")
val audioStream = source.primaryAudio ?: error("no audio track")
val inAudio = audioStream.audio!!            // AudioStreamInfo: the stream's audio detail block

val audio = sink.addAudioEncoder(AudioEncoderSpec(codec = CodecId.Aac))

val graph = FilterGraph.buildAudio(
    description = "anull",
    sampleRate = inAudio.sampleRate,
    sampleFormat = inAudio.sampleFormat,
    channels = inAudio.channels,
    timeBase = audioStream.timeBase,
    outputSampleRate = audio.sampleRate,
    outputSampleFormat = audio.sampleFormat,
    outputChannels = audio.channels,
)
graph.setOutputFrameSize(audio.frameSize)   // 1024: the graph re-chunks to exact AAC frames

audio.drive(graph.process(source.decodedFrames(audioStream)))

The graph re-chunks the audio stream into exact 1024-sample frames before they reach the encoder. Transcoder wires this up automatically, reading frameSize from the encoder and calling setOutputFrameSize for you. When you drive MediaSink by hand, this one call is what satisfies AAC's framing requirement.

Why a filter graph for plain copy of samples?

Even when you are not changing the audio, the filter graph is doing real work: resampling to the encoder's negotiated format and re-chunking to the codec's frame size. "anull" is the no-op filter description; the resampling and chunking happen in the buffer sink regardless.

Stream copy: `addCopyStream`¶

When a stream should pass through untouched, do not decode and re-encode it. addCopyStream declares a verbatim copy of one input stream into the output. This is FFmpeg's -c copy: no decode, no encode, only timestamp rescaling into the output's time-base.

import io.github.yuroyami.kiteffmpeg.KiteFFmpegLowLevelApi
import io.github.yuroyami.kiteffmpeg.MediaSink
import io.github.yuroyami.kiteffmpeg.MediaSource

@OptIn(KiteFFmpegLowLevelApi::class)
fun copyAudio(input: String, output: String) {
    MediaSource.open(input).use { source ->
        val audio = source.primaryAudio ?: error("no audio stream")
        MediaSink.open(output).use { sink ->
            val copy = sink.addCopyStream(source, audio)       // declares the output stream
            source.openPacketReader(listOf(audio)).use { reader ->
                while (true) {
                    val packet = reader.read() ?: break
                    packet.use { copy.write(it) }               // writes it, bit-exact
                }
            }
        }
    }
}

addCopyStream returns a CopyStream, which only declares the output stream and where its packets come from. Nothing writes those packets until you do: read them with a PacketReader and write each one, as above. Driving an encoder on the same sink does not write them. The output starts at zero where its media starts to show, so the sink keeps the first packets until every copied audio and video stream has shown where it starts, at most a second of the input, and then writes them in the order they came. Transcoder with audioCopy = true and Remuxer run this loop for you. To re-encode the video and copy the audio from one file, use Transcoder: it reads the file once and interleaves both.

Bitstream filters are automatic

A format pair that needs a bitstream filter, such as H.264 in MP4 going to MPEG-TS, needs nothing from you: libavformat inserts the filter when the copied packets are written.

For a whole-file lossless container rewrite (every stream copied, no encoders at all), use Remuxer.remux instead. It runs in seconds.

Hardware encode¶

A hardware encoder is selected with the spec's encoder, an EncoderId, the same way a software encoder is. It produces the same kind of VideoEncoder and drives the same way; only the encoder id changes.

macOS (VideoToolbox)Android (MediaCodec)

val spec = VideoEncoderSpec(
    codec = CodecId.H264,
    encoder = EncoderId.H264VideoToolbox, // or HevcVideoToolbox
    width = 1920, height = 1080,
    frameRate = Rational(30, 1),
    bitrateBps = 8_000_000,
)
val encoder = sink.addVideoEncoder(spec)

Verified end-to-end on macOS arm64. h264_videotoolbox encodes on the Apple media engine instead of the CPU.

val spec = VideoEncoderSpec(
    codec = CodecId.H264,
    encoder = EncoderId.H264MediaCodec, // or HevcMediaCodec
    width = 1920, height = 1080,
    frameRate = Rational(30, 1),
    bitrateBps = 8_000_000,
)
val encoder = sink.addVideoEncoder(spec)

h264_mediacodec targets the device's hardware encoder. See Platform support for the current Android status.

`allow_sw` for VMs and headless machines¶

VideoToolbox refuses to run when there is no hardware encode block available, which is common on CI runners and virtual machines. The allow_sw option lets it fall back to a software path instead of failing to open:

VideoEncoderSpec(
    codec = CodecId.H264,
    encoder = EncoderId.H264VideoToolbox,
    width = 1280, height = 720,
    frameRate = Rational(30, 1),
    options = mapOf("allow_sw" to "1"),
)

Check before you commit¶

Hardware encoder availability is a runtime property of the machine and the FFmpeg build. Ask the build which encoders it has for a format with FFmpeg.encodersFor before you choose. The first one is the encoder FFmpeg picks when the spec names none:

import io.github.yuroyami.kiteffmpeg.CodecId
import io.github.yuroyami.kiteffmpeg.EncoderId
import io.github.yuroyami.kiteffmpeg.FFmpeg

val available = FFmpeg.encodersFor(CodecId.H264)
val encoder = EncoderId.H264VideoToolbox.takeIf { it in available } ?: available.firstOrNull()

One encode core for hardware and software

The same EAGAIN-correct encode loop drives software and hardware encoders. There is no separate code path and no separate API: pick the format and the encoder, build the spec, call drive. Everything downstream is identical.

Both streams in one sink¶

A real output usually carries video and audio together. Add both encoders up front, then drive them from their respective flows:

MediaSink.open("output.mp4").use { sink ->
    val video = sink.addVideoEncoder(
        VideoEncoderSpec(
            codec = CodecId.Mpeg4,
            width = 1280, height = 720,
            frameRate = Rational(30, 1),
        )
    )
    val audio = sink.addAudioEncoder(
        AudioEncoderSpec(codec = CodecId.Aac)
    )

    video.drive(videoFrames)
    audio.drive(audioFrames)
}   // close() flushes both encoders, then writes the trailer

The muxer interleaves packets from both encoders into the container. close() waits until both encoders have flushed before writing the trailer, so the file is complete and seekable.

Do not drive one sink's encoders from concurrent coroutines

All encoders attached to a MediaSink share the underlying muxer, and libav contexts are not thread-safe. Drive them from a single coroutine, one after the other, as shown above. When both streams come from the same input file, you can instead use decodeStreams (one demux pass, frames already interleaved) and route each frame to the right encoder as it arrives. That is what Transcoder does internally. See Concurrency for the full threading rules.

For the common case of demux one file, re-encode, and mux, the Transcoder already orchestrates this interleaving for you, including the AAC frame-size wiring and the trim rebasing. Use raw MediaSink when your frames come from a source a single input file cannot describe.

Writing into your own bytes¶

MediaSink.open(sink, format, options) writes the output into a MediaByteSink instead of a path: an in-memory buffer, encrypted storage, a virtual file system or an upload stream. It is the output twin of MediaByteSource. The format is required, because there is no path to infer it from.

class MemorySink : MediaByteSink {
    val bytes = ByteArrayOutputStream()   // any growable buffer
    override val seekable = false
    override fun write(bytes: ByteArray, offset: Int, length: Int) = this.bytes.write(bytes, offset, length)
    override fun seek(position: Long) = error("not seekable")
    override fun close() = Unit
}

val sink = MediaSink.open(MemorySink(), "mp4", mapOf("movflags" to "frag_keyframe+empty_moov"))

The rules:

Every call arrives on the thread that writes the output, one at a time. A write that blocks holds the muxer, so a slow consumer bounds the buffering to the 64 KiB the muxer holds.
write takes all the bytes or throws. The exception becomes the cause of the FFmpegException that fails the write, and the sink cannot write again after it.
A container that has to seek back, such as MP4 without fragments, refuses a sink whose seekable is false with FFmpegError.InvalidArgument when the header is written. Pass "movflags" to "frag_keyframe+empty_moov" to write fragmented MP4, or make the sink seekable.
The MediaSink owns the sink: after the last byte it calls flush() once, then close() once.

Container metadata¶

Tag the output before the first frame:

sink.setMetadata(
    mapOf(
        "title" to "Holiday clip",
        "artist" to "KiteFFmpeg",
        "comment" to "Encoded with VideoToolbox",
    )
)

These are written into the container header, so the call has to come before any frame is written, alongside the encoder declarations.

Closing is not optional¶

close() does three things: it flushes every encoder (draining the packets still buffered inside the codec), it writes the container trailer (the index that makes the file seekable), and it frees the native handles. A file that is never closed is truncated and usually unplayable.

MediaSink is AutoCloseable, so the idiomatic form is Kotlin's use:

MediaSink.open("output.mp4").use { sink ->
    // add encoders, drive frames
}   // close() runs here, even on exception

The encoders (VideoEncoder, AudioEncoder) are also AutoCloseable, but drive flushes them when its input flow completes, so you rarely close them by hand. Closing the sink is the call that matters.

Errors¶

Every libav failure surfaces as an FFmpegException wrapping an FFmpegError. FFmpegError is a sealed hierarchy of semantic categories (FileNotFound, EncoderNotFound, MuxerNotFound, InvalidData, and more) mapped from the raw AVERROR_* codes. Unmapped codes arrive as FFmpegError.AvError. A library-side invariant violation arrives as FFmpegError.Internal. Every subclass exposes the numeric code:

import io.github.yuroyami.kiteffmpeg.FFmpegException

try {
    MediaSink.open("output.mp4").use { sink ->
        sink.addVideoEncoder(spec).drive(frames)
    }
} catch (e: FFmpegException) {
    println("encode failed: ${e.error} (code ${e.code})")
}

A common one to expect: opening a CodecId whose encoder is not present in the linked FFmpeg build. Probe with FFmpeg.hasEncoder(...) first to fail early with a clear message rather than at addVideoEncoder.