You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: doc/concept/layer/moq-lite.md
+26-8Lines changed: 26 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,10 +4,29 @@ description: A fraction of the calories with none of the fat.
4
4
---
5
5
6
6
# moq-lite
7
-
A subset of the [MoqTransport](/concept/standard/moq-transport) specification.
8
-
The useless/optional cruft has been removed so more time can be spent on the core functionality.
7
+
[moq-lite](https://www.ietf.org/archive/id/draft-lcurley-moq-lite-02.html) is a subset of the [MoqTransport](/concept/standard/moq-transport) specification.
8
+
The goal is to keep the core transport layer simple and focused on practical use-cases.
9
9
10
-
See the draft: [draft-lcurley-moq-lite](https://www.ietf.org/archive/id/draft-lcurley-moq-lite-02.html).
10
+
There's too much fringe functionality in the MoqTransport draft that's not practical to implement.
11
+
Most of it is specific to Cisco's implementation and bizarre requirements anyway.
12
+
13
+
## Compatibility
14
+
Keep in mind that moq-lite is forward compatible with the IETF draft.
15
+
For every moq-lite API, there's a corresponding moq-transport API.
16
+
So fortunately, it doesn't matter if I get hit by a bus and moq-lite ceases to exist.
17
+
18
+
The moq.dev libraries negotiate the `moq-lite` or `moq-transport` version as part of the QUIC/WebTransport handshake (via ALPN).
19
+
When `moq-transport` wire format is negotiated, we implement a compatibility layer that enforces the moq-lite API.
20
+
For example, if there's a gap in a group (valid in moq-transport), we drop the tail of the group instead of erroring.
Hopefully you had this square on your buzzword bingo card.
8
+
9
+
WebRTC is a great protocol for conferencing, but it's not designed for AI.
10
+
But I haven't personally worked in this space either so take my suggestions with a grain of salt.
11
+
12
+
## Latency
13
+
Inference is still quite slow and expensive, even for the big players.
14
+
If you're going to spend >300ms and literal dollars on expensive inference, you want at least *some* reliability guarantees.
15
+
16
+
Unfortunately, WebRTC will never try to retransmit audio packets.
17
+
A single lost packet will cause noticeable audio distortion.
18
+
And if you have the audacity to generate audio/video separately, WebRTC won't synchronize them for you.
19
+
Frames are rendered on receipt, so unless you introduce a delay, audio will be out of sync with video.
20
+
21
+
One of the core tenets of MoQ is adjustable latency.
22
+
The viewer (and thus your application) controls how long it's willing to wait for content before it gets skipped/desynced.
23
+
The latency budget of the network protocol can match the latency budget of the application.
24
+
25
+
## On-Demand
26
+
MoQ is pull-based, so nothing is transmitted over the network until there's at least one subscriber.
27
+
You can further extend this by not generating/encoding content either.
28
+
29
+
Both of these were mentioned briefly on the [contribution](/concept/use-case/contribution) page if you want to read more.
30
+
31
+
### Inference
32
+
If you want to save compute resources, you can defer inference until it's actually needed.
33
+
34
+
For example, let's say you're publishing a `captions` track populated by Whisper or something.
35
+
If nobody has enabled captions, then nobody will subscribe to the `captions` track.
36
+
You can stop generating the track (or use a smaller model) until it's actually requested.
37
+
38
+
### Simulcast
39
+
If you want to save bandwidth, you can publish media in a format expected by the AI model.
40
+
41
+
For example, let's say you're doing object detection on a bunch of security cameras.
42
+
The model inputs video at 360p and 10fps or something like that, so that's what you publish.
43
+
But if a human (those still exist) wants to audit the full video, you can separately serve the full resolution video.
44
+
Since this is on-demand, you will only encode/transmit the 1080p video when it's actually needed.
45
+
46
+
## Browser Control
47
+
One of the perks of using WebSockets/MoQ instead of WebRTC is that you get full control over the media pipeline.
48
+
49
+
[WebCodecs](https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API) is used to encode/decode media within the browser.
50
+
- For video, you use [VideoFrame](https://developer.mozilla.org/en-US/docs/Web/API/VideoFrame) which directly maps to a texture on the GPU. You can use WebGPU to perform inference, encoding, rendering, etc without ever touching the CPU.
51
+
- For audio, you get [AudioData](https://developer.mozilla.org/en-US/docs/Web/API/AudioData) which is (usually) just a float32 array. You control exactly how these are processed, captured, emitted, etc.
52
+
53
+
It's more work to do this instead of using a `<video>` element of course, but it opens the door to more possibilities.
54
+
Run additional inference in the browser, render your media to textures on a model, etc.
55
+
56
+
And note that all of this is possible with WebRTC and [insertable streams](https://developer.mozilla.org/en-US/docs/Web/API/Insertable_Streams_for_MediaStreamTrack_API).
57
+
However, you're really not gaining much by using WebRTC only for networking... just use MoQ instead.
58
+
59
+
## Non-Media
60
+
MoQ is not just for media.
61
+
62
+
Send your prompts over the same WebTransport connection as the media.
63
+
Or send non-media stuff like vertex data for 3D models, separate from the texture data.
64
+
It's a versatile protocol with a wide range of use-cases.
65
+
66
+
## Simplicity
67
+
You're working with AI, so you're probably building something new.
68
+
69
+
If you don't want to deal with SDP, or connections that take 10 RTTs, or unsupported media encodings, or STUN/TURN servers, then give MoQ a try.
70
+
It's a lot closer to WebSockets than WebRTC, but with the ability to skip and scale.
0 commit comments