feat(realtime): lean livekit_join with passthrough; ungate publish from initial-state ack#19
feat(realtime): lean livekit_join with passthrough; ungate publish from initial-state ack#19VerioN1 wants to merge 1 commit into
Conversation
…om initial-state ack Reverse the bundled-join approach (#17): the client now sends a lean `livekit_join` carrying only `{ type, passthrough }`, then sends the `set_image`/`prompt` initial-state frame as a separate message right after. Room connect, local-track publish, and the connected/session-started signal proceed off `livekit_room_info` alone — the (often multi-MB) image upload and its ack are taken off the critical path to first frame. The initial-state ack is now observed out-of-band and used only to raise an error event if the server rejects the reference. Its timeout is armed after room_info (never during a queue wait), and acks arriving before room_info are buffered and drained by the out-of-band watcher. - SignalingMessages: livekit_join carries `passthrough` instead of nested initial_state. - SignalingChannel.sendLiveKitJoin: lean join + separate frame; no ack gating. - RealtimeSessionManager: derive passthrough, watch ack out-of-band, drop remote-stream gating. - InitialStateGate: reduced to attempt-staleness tracking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit fe89221. Configure here.
| roomInfoDeferred = null | ||
| throw IllegalStateException("WebSocket is not open") | ||
| } | ||
| initialState.toInitialStateMessage()?.let { send(it) } |
There was a problem hiding this comment.
Initial state frame send ignored
High Severity
After a successful lean livekit_join, the separate initial-state message is sent via send without checking its return value. If that send fails while passthrough is false, the server expects a frame that never arrived, yet the client still waits for livekit_room_info and may complete connect.
Reviewed by Cursor Bugbot for commit fe89221. Configure here.
| config.onError(e, "initial-state") | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Orphan ack watcher spurious errors
Medium Severity
watchInitialStateAck is launched on the manager scope and only suppresses errors when disposed is true. tearDownTransports closes signaling on stale attempts and connect retries without cancelling that job or checking attempt staleness, so a superseded or retried connect can emit a misleading initial-state error on an active session.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit fe89221. Configure here.
1 participant
What
Implements the realtime connect spec: a lean signaling handshake that takes the initial-state image upload off the critical path to first frame. This reverses the bundled-join approach from #17.
Connect sequence (new)
livekit_join:{ type, passthrough }— no nested initial state.set_image/prompt), without awaiting its ack.livekit_room_info, connect the LiveKit room, publish local tracks, and mark connected / emit session-started — all offroom_infoalone.passthrough(required boolean on the join)false— caller set a realimage/prompt; exactly one real initial-state frame follows.true— no real reference (no frame, or a null-bootstrapset_imagewithimage_data: null, prompt: nullfor the local-stream-only case).falseiff the caller supplied a non-null image or prompt;trueotherwise.Ack timing
room_infoare buffered and drained by the out-of-band watcher.room_info, so a longqueue_positionwait can't trip it.Why
Previously the client waited for the
set_image/promptack before publishing the local track and marking the session connected. The multi-MB image upload + its ack sat on the path to first frame. Now they don't.Changes
SignalingMessages:livekit_joincarriespassthroughinstead of nestedinitial_state.SignalingChannel.sendLiveKitJoin: lean join + separate frame; no ack gating;awaitInitialStateAckalways settles the buffering flag.RealtimeSessionManager: derivespassthrough, watches the ack out-of-band (error-only), drops remote-stream gating.InitialStateGate: reduced to attempt-staleness tracking (no longer awaits the ack).Notes
ConnectOptions/RealtimeConfigurationstable per repo convention). The spec's optional "explicit passthrough override from session config" is not wired to a public knob — no caller needs it and the derived value is correct for every case; easy to add later if a use case appears.Test
./gradlew :sdk:testDebugUnitTest— green../gradlew :sdk:assembleRelease :sample:assembleDebug— green.SignalingChannelTest(lean join + separate frame + buffered-then-drained ack),SignalingMessagesTest(passthrough serialization),InitialStateGateTest(staleness-only API).🤖 Generated with Claude Code
Note
Medium Risk
Changes the realtime connect handshake and ordering (protocol shape plus when publish/session start vs ack), which can affect time-to-first-frame and whether remote frames appear before initial state is acknowledged; public API is unchanged but server/client timing assumptions must align.
Overview
Moves realtime connect onto a lean signaling handshake:
livekit_joinis now{ type, passthrough }(no nestedinitial_state), with any real initialset_image/promptsent as a separate frame right after join.passthroughis derived when the caller did not supply a non-null image or prompt.LiveKit room connect, local publish, connected/session-started, and remote stream delivery no longer wait on the initial-state ack—only
livekit_room_infogates that path. The ack is watched out-of-band (buffered if it arrives early; timeout starts afterroom_info); failures surface viaonErrorwithout blocking connect.InitialStateGateis reduced to stale-attempt detection (isCurrent); remote-stream holdback and parallel ack+media connect are removed.Reviewed by Cursor Bugbot for commit fe89221. Bugbot is set up for automated code reviews on this repo. Configure here.