Concurrency documentation

[jverkoey]

What are the ramifications of increasing concurrency? Does it result in the model being loaded multiple times into memory, or is the model only loaded once and somehow the processing/generation tasks are interleaved?

[cubist38]

Hey @jverkoey , I believe you’re referring to a batching approach. I'm planning to implement this in the future.

[jverkoey]

Likely yes, but if that's not implemented, what does setting concurrency to >1 do currently? Documentation as to the effect of this would be handy!

[cubist38]

Hi @jverkoey , after PR #194 the server now runs the inference worker on a single dedicated background thread.
This prevents blocking I/O and keeps things responsive, so endpoints like health checks and /v1/models remain available even while processing heavy requests.

[jverkoey]

Oooh amazing! I was just running into that with the new session viewer tool I'm adding to mlx-openai-server :)

[cubist38]

@jverkoey Have you tried Qwen/Qwen3-Coder-Next? I’m quite impressed with its performance so far.

[jverkoey]

Running it locally it seemed to struggle with maintaining trains of thought and tool calling after even a couple turns, but I was using LMStudio so maybe LMStudio was causing it to have issues.

[cubist38]

@jverkoey I tried mlx-openai-server and tested it with Qwen-Code — the results are really impressive.
I recorded a quick demo here: https://www.youtube.com/watch?v=X5Hsd3QR_E8

mlx-openai-server launch \
 --model-path mlx-community/Qwen3-Coder-Next-4bit \
 --model-type lm \
 --tool-call-parser qwen3_coder \
 --message-converter qwen3_coder

Just make sure you’re running the latest version of mlx-openai-server.

[electroheadfx]

@jverkoey I tried mlx-openai-server and tested it with Qwen-Code — the results are really impressive. I recorded a quick demo here: https://www.youtube.com/watch?v=X5Hsd3QR_E8

mlx-openai-server launch
--model-path mlx-community/Qwen3-Coder-Next-4bit
--model-type lm
--tool-call-parser qwen3_coder
--message-converter qwen3_coder
Just make sure you’re running the latest version of mlx-openai-server.

its amazing, I see some 60 to 70 tokens / secs ? is it right ?

[jverkoey]

Is it possible for the same server instance to serve multiple models?

[cubist38]

Is it possible for the same server instance to serve multiple models?

@jverkoey Yes — it’s something I’m planning to implement, though I’m not entirely sure about the timeline yet. I’m hoping I’ll have enough time to work on it during the Lunar New Year holidays 🤞

[electroheadfx]

it would like amazing because opencode support that

[cubist38]

Hey @jverkoey @electroheadfx , I’ve just opened PR #197 to add multi-model support. When you have a moment, could you please help test it and let me know how it works on your side? I’d really appreciate your feedback 🙏

[electroheadfx]

Hey @jverkoey @electroheadfx , I’ve just opened PR #197 to add multi-model support. When you have a moment, could you please help test it and let me know how it works on your side? I’d really appreciate your feedback 🙏

Just tested and sent report to #135 (comment)
work nice, I can't wait to have the release soon !