[BUG] CPU admission can exceed max_cpu_utilization in edge cases

[HenriqueOtsuka]

Describe the bug

I'm running LiveKit Egress through OpenVidu V3 and observed a case where the egress node exceeded the configured CPU admission limit and started killing egress sessions due to sustained high CPU.

The egress config has:

cpu_cost:
  max_cpu_utilization: 0.8

However, during a burst of room-composite egress starts on a single egress node, the service accepted new requests and shortly after reported CPU usage around 95–99% with requests: 8.

The service then started logging:

high cpu usage
killing egress due to sustained high cpu

This affected egresses that were already active on the same node.

Based on reviewing pkg/stats/monitor.go, I believe this is not necessarily a concurrent double-accept race, since AcceptRequest is serialized with m.mu. Instead, the issue appears to be caused by CPU admission/accounting edge cases:

1. pendingCPU is reset from a time.AfterFunc callback without holding m.mu, while other goroutines read it under m.mu.
2. When m.requests.Load() == 0, the first request appears to be checked against total CPU, not total * MaxCpuUtilization.
3. cpuHoldDuration = 15s may release the CPU reservation before the real CPU cost of Chrome/GStreamer/RTMP encoding is visible in the process stats. During warmup, lastCPU can still be too low, allowing additional starts that later push the node above max_cpu_utilization.

In the observed incident, two new room-composite egresses were accepted almost at the same time. They became active within a few seconds, and shortly after the node reported sustained high CPU and started killing egresses.

Egress Version

Running through OpenVidu V3.

OpenVidu component observed in logs:

@openvidu-meet/backend@3.6.0

The exact LiveKit Egress image/version still needs to be confirmed from the deployment.

Related log path references show LiveKit Egress code paths such as:

github.com/livekit/egress/pkg/...

and gst-go dependency:

github.com/livekit/gst-go@v0.0.0-20250701011214-e7f61abd14cb

Egress Request

Two room-composite stream egresses were started almost at the same time on the same egress node.

Request 1

{
  "RoomComposite": {
    "room_name": "room-1",
    "layout": "PRESENTERS_ONLY",
    "custom_base_url": "https://<redacted-custom-layout-url>",
    "Output": {
      "Stream": {
        "protocol": 1,
        "urls": ["rtmp://<redacted>/live/<redacted>"]
      }
    },
    "Options": {
      "Preset": 2
    }
  }
}

Request 2

{
  "RoomComposite": {
    "room_name": "room-2",
    "layout": "PRESENTERS_ONLY",
    "custom_base_url": "https://<redacted-custom-layout-url>",
    "Output": {
      "Stream": {
        "protocol": 1,
        "urls": ["rtmp://<redacted>/live/<redacted>"]
      }
    },
    "Options": {
      "Preset": 2
    }
  }
}

Additional context

This was a single-node egress instance used by OpenVidu V3.

At the time of the incident, the node already had multiple active egress requests. The two new egresses were accepted at 2026-04-29T01:04:33Z and became active at 2026-04-29T01:04:36Z.

Immediately after that, the egress service reported:

cpu ~= 0.95–0.99
requests: 8

Then the egress service started killing egresses due to sustained high CPU.

Important timing:

2026-04-29T01:04:33Z - two new egress requests accepted
2026-04-29T01:04:36Z - both became EGRESS_ACTIVE
2026-04-29T01:04:38Z - high CPU warning started, cpu ~0.963, requests: 8
2026-04-29T01:04:47Z - killing egress due to sustained high CPU
2026-04-29T01:05:04Z - killing egress due to sustained high CPU
2026-04-29T01:05:16Z - killing egress due to sustained high CPU

Later, affected egresses eventually ended with Closed Remotely, but that was a delayed final state. The operational issue started around 2026-04-29T01:04:38Z, when CPU admission/capacity was exceeded.

The relevant event on the egress node was sustained high CPU followed by LiveKit Egress killing sessions.

Logs

Relevant sanitized logs:

2026-04-29T01:04:33.140Z INFO egress server/server_rpc.go:64 request received
{"nodeID":"NE_Ld6efiWfXu4w","egressID":"EG_2X8PoxUbrAE2"}

2026-04-29T01:04:33.142Z INFO egress server/server_rpc.go:74 request validated
{"nodeID":"NE_Ld6efiWfXu4w","egressID":"EG_2X8PoxUbrAE2","requestType":"room_composite","sourceType":"EGRESS_SOURCE_TYPE_WEB","outputType":"stream","room":"room-1"}

2026-04-29T01:04:33.458Z INFO egress server/server_rpc.go:64 request received
{"nodeID":"NE_Ld6efiWfXu4w","egressID":"EG_NXLkCKcy6YWg"}

2026-04-29T01:04:33.458Z INFO egress server/server_rpc.go:74 request validated
{"nodeID":"NE_Ld6efiWfXu4w","egressID":"EG_NXLkCKcy6YWg","requestType":"room_composite","sourceType":"EGRESS_SOURCE_TYPE_WEB","outputType":"stream","room":"room-2"}

2026-04-29T01:04:36.029Z INFO egress pipeline/watch.go:259 pipeline playing
{"nodeID":"NE_Ld6efiWfXu4w","handlerID":"EGH_NkbCG4pnGkWS","egressID":"EG_2X8PoxUbrAE2","timeToPlaying":"1.009169877s"}

2026-04-29T01:04:36.031Z INFO egress info/io.go:273 egress_active
{"nodeID":"NE_Ld6efiWfXu4w","egressID":"EG_2X8PoxUbrAE2","requestType":"room_composite","outputType":"stream"}

2026-04-29T01:04:36.218Z INFO egress pipeline/watch.go:259 pipeline playing
{"nodeID":"NE_Ld6efiWfXu4w","handlerID":"EGH_5SHEzXBcsn6u","egressID":"EG_NXLkCKcy6YWg","timeToPlaying":"1.01701603s"}

2026-04-29T01:04:36.222Z INFO egress info/io.go:273 egress_active
{"nodeID":"NE_Ld6efiWfXu4w","egressID":"EG_NXLkCKcy6YWg","requestType":"room_composite","outputType":"stream"}

2026-04-29T01:04:38.145Z WARN egress stats/monitor.go:629 high cpu usage
{"nodeID":"NE_Ld6efiWfXu4w","cpu":0.9633333333333333,"requests":8}

2026-04-29T01:04:39.132Z WARN egress stats/monitor.go:629 high cpu usage
{"nodeID":"NE_Ld6efiWfXu4w","cpu":0.960669456080048,"requests":8}

2026-04-29T01:04:40.143Z WARN egress stats/monitor.go:629 high cpu usage
{"nodeID":"NE_Ld6efiWfXu4w","cpu":0.9591666666666663,"requests":8}

2026-04-29T01:04:41.135Z WARN egress stats/monitor.go:629 high cpu usage
{"nodeID":"NE_Ld6efiWfXu4w","cpu":0.9684123025558992,"requests":8}

2026-04-29T01:04:42.155Z WARN egress stats/monitor.go:629 high cpu usage
{"nodeID":"NE_Ld6efiWfXu4w","cpu":0.9691923397255453,"requests":8}

2026-04-29T01:04:47.153Z WARN egress stats/monitor.go:629 high cpu usage
{"nodeID":"NE_Ld6efiWfXu4w","cpu":0.9766860949296086,"requests":8}

2026-04-29T01:04:47.158Z WARN egress stats/monitor.go:641 killing egress due to sustained high cpu
{"nodeID":"NE_Ld6efiWfXu4w","cpu":0.9766860949296086}

2026-04-29T01:05:04.162Z WARN egress stats/monitor.go:629 high cpu usage
{"nodeID":"NE_Ld6efiWfXu4w","cpu":0.9502487562145029,"requests":8}

2026-04-29T01:05:04.163Z WARN egress stats/monitor.go:641 killing egress due to sustained high cpu
{"nodeID":"NE_Ld6efiWfXu4w","cpu":0.9502487562145029}

2026-04-29T01:05:16.139Z WARN egress stats/monitor.go:629 high cpu usage
{"nodeID":"NE_Ld6efiWfXu4w","cpu":0.9592346089835386,"requests":8}

2026-04-29T01:05:16.139Z WARN egress stats/monitor.go:641 killing egress due to sustained high cpu
{"nodeID":"NE_Ld6efiWfXu4w","cpu":0.9592346089835386}

Possible cause / analysis

After reviewing pkg/stats/monitor.go, I suspect the admission logic can underestimate real CPU during egress warmup.

The two new requests were accepted before their real CPU cost was visible in process stats. A few seconds later, when Chrome/GStreamer/RTMP encoding started, CPU jumped above the configured max_cpu_utilization.

Potential contributing factors:

1. pendingCPU is reset by the cpuHoldDuration timer, but if real CPU samples are still low or missing during warmup, the monitor may admit more work than the node can actually handle.
2. The first request path appears to use full CPU capacity instead of total * MaxCpuUtilization.
3. pendingCPU reset appears to happen from a timer goroutine without holding m.mu, while other paths read it under lock.

I can provide a PR for items (1) and (2), but would like maintainer guidance on how to handle (3), especially whether pendingCPU should remain as a floor until real CPU samples are available.

[biglittlebigben]

1. is indeed a data race, but it seems very unlikely to hit it in practice.
2. Is on purpose to avoid refusing to take any request in case the pod is close to under dimensioned
3. Is possible but not something we have ever noticed. It also doesn't quite match your timeline.

Are your cpu costs configured properly?

[HenriqueOtsuka]

Hi Ben,

Thanks for the clarification.

I agree the data race is probably not the practical root cause of this incident. The more relevant part for this case seems to be admission/cost estimation versus real runtime CPU usage during room-composite startup.

About CPU costs: this is the cpu_cost section from our deployed egress.yaml:

cpu_cost:
  max_cpu_utilization: 0.8
  room_composite_cpu_cost: 2.0
  audio_room_composite_cpu_cost: 1.0
  web_cpu_cost: 2.0
  audio_web_cpu_cost: 0.5
  participant_cpu_cost: 1.0
  track_composite_cpu_cost: 1.0
  track_cpu_cost: 0.5

The observed behavior was:

2026-04-29T01:04:33Z - two new room-composite web egress requests accepted
2026-04-29T01:04:36Z - both became EGRESS_ACTIVE
2026-04-29T01:04:38Z - high CPU warning started: cpu=0.9633, requests=8
2026-04-29T01:04:47Z - killing egress due to sustained high cpu
2026-04-29T01:05:04Z - killing egress due to sustained high cpu
2026-04-29T01:05:16Z - killing egress due to sustained high cpu

So the issue I’m trying to understand is not only whether one request had the right static cost, but why the node reached requests: 8 and then exceeded the configured max_cpu_utilization by a wide margin.

In this specific case, the new egresses were:

requestType: room_composite
sourceType: EGRESS_SOURCE_TYPE_WEB
outputType: stream
pipeline: Chrome / GStreamer / RTMP output

The CPU spike appeared a few seconds after the requests were accepted and became active.

Assuming the static costs are underestimating the real CPU cost for this workload, is the expected mitigation to tune the configured room_composite_cpu_cost and/or web_cpu_cost higher so that fewer concurrent egresses are admitted per node?

Also, is there any recommended way to estimate or calibrate those values? For example, should we set the room-composite web stream cost based on peak CPU during Chrome/GStreamer warmup, steady-state CPU, or some conservative upper bound?

[biglittlebigben]

egress exposes a wide array of prometheus metrics which should allow you to optimize your conf for your workload.

[HenriqueOtsuka]

Thanks. We can definitely use the Prometheus metrics to tune the configuration for our workload.

However, I don’t think that fully addresses the behavior I’m trying to report.

The concern is not only “how do I optimize my config”, but that the admission mechanism allowed the node to reach a state where it had:

requests: 8
cpu: ~0.95–0.99
max_cpu_utilization: 0.8

and then the egress service started killing sessions due to sustained high CPU.

So even if the Prometheus metrics help us tune the cost values after the fact, the runtime behavior still means the node can exceed the configured admission cap significantly before it reacts.

In our case, the problematic workload was:

requestType: room_composite
sourceType: EGRESS_SOURCE_TYPE_WEB
outputType: stream
pipeline: Chrome / GStreamer / RTMP output

The timeline was:

2026-04-29T01:04:33Z - two new room-composite web egress requests accepted
2026-04-29T01:04:36Z - both became EGRESS_ACTIVE
2026-04-29T01:04:38Z - high CPU warning started: cpu=0.9633, requests=8
2026-04-29T01:04:47Z - killing egress due to sustained high cpu

That is only a few seconds between admission and overload.

My concern is that for workloads with heavy startup cost, such as Chrome/GStreamer room-composite egress, the configured max_cpu_utilization does not behave as a hard enough admission guard. The node can accept work based on stale/underestimated CPU and only detect the overload after the egress pipelines are already active.

If the intended design is that max_cpu_utilization is only a best-effort guard and users are expected to tune cpu_cost conservatively using Prometheus metrics, then that’s useful to clarify. But I think this could still benefit from either:

1. documentation explaining that max_cpu_utilization is not a hard cap and that cpu_cost must be calibrated per workload using Prometheus metrics;
2. safer default cpu_cost values for room_composite + web egress;
3. exposing guidance for which Prometheus metrics should be used for admission tuning;
4. or improving admission accounting during the Chrome/GStreamer warmup window so pending/requested CPU remains conservative until real CPU samples are available.

I’m not trying to claim the data race is the root cause of this specific production incident. The main issue is that the current admission model can still oversubscribe the node under bursty room-composite web starts, and the consequence is not just degraded performance — the egress service starts killing active sessions.