Skip to content

adding least response time algorithm#5

Open
vishwakarmaritu wants to merge 3 commits intolugnitdgp:mainfrom
vishwakarmaritu:main
Open

adding least response time algorithm#5
vishwakarmaritu wants to merge 3 commits intolugnitdgp:mainfrom
vishwakarmaritu:main

Conversation

@vishwakarmaritu
Copy link
Copy Markdown

@vishwakarmaritu vishwakarmaritu commented Dec 27, 2025

Changes:

  1. A new algorithm leastresponsetime.go is added in internal/routing/ . The backend with the lowest exponential moving average(EMA) latency is selected. Here, EMA is implemented because it prioritizes most recent requests, which is helpful for least response time(LRT) routing.
  2. adaptive.go in internal/routing is updated to initialize LRT. When the average latency exceeds 50ms, we automatically switch to least response time algorithm.
  3. L7 and H4 proxies are updated to add EMA latency tracking.

Testing:

Tested locally with a simulated delay (0ms , 300ms).
20 requests were sent. Initially, the system used the safest method, round robin. After a few requests, when the latency crossed the threshold 50ms, it switched to least response time.

https://drive.google.com/file/d/1baX5YgwHVzNVvcNYT6m682A73IWxr8Nw/view?usp=drive_link

@sbirmecha99
Copy link
Copy Markdown
Member

Since our demo backends and delays are hardcoded, using averageLatencyMs as the trigger causes LRT to be enabled permanently (the average will always stay high with fixed delays). Once LRT is active, the fastest backend keeps getting selected and the others stop being sampled, so their latency values never update.
Some small improvements could be-

  1. dont auto-switch to LRT based only on averageLatencyMs
    With fixed demo delays, this condition is always true and makes LRT permanently active. Either relax this condition or make LRT an explicit choice (flag/config) for the demo
  2. Ensure all backends continue to be sampled
    Even in LRT mode, allow occasional requests to non-fastest backends so their EMA latency stays updated and does not become stale.
    This will help keep EMA values accurate and routing stable

@vishwakarmaritu
Copy link
Copy Markdown
Author

Changes:

  1. Made LRT an explicit choice for the demo
  2. To prevent staleness, even in LRT mode 5% of requests are sent to random backends

Testing: https://drive.google.com/file/d/1o1WTaTp8sIDWNasmv74Sduzkvq2yu9mt/view?usp=sharing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants