feat(chat-completions): server-side fan-out for n>1 choices by lvhan028 · Pull Request #4841 · InternLM/lmdeploy

lvhan028 · 2026-08-09T15:09:09Z

Motivation

The OpenAI-compatible Chat Completions API exposes the n parameter for generating multiple choices, but LMDeploy previously supported only n=1. Users therefore had to send multiple client-side requests and manually
combine their responses.

This PR adds native n > 1 support while reusing the existing single-choice endpoint, preserving its preprocessing, generation, streaming, and output-parsing behavior. The existing n=1 path remains unchanged.

Modification

Add server-side fan-out for n > 1, with one independent single-choice generation per requested choice.
Assign stable choice indices and offset an explicitly supplied seed for independent, reproducible choices.
Combine non-streaming results into one OpenAI-compatible response.
Concurrently drain streaming results and batch ready deltas into shared SSE chunks containing multiple choices when possible.
Aggregate usage by counting shared prompt and cached tokens once while summing completion tokens across choices.
Add ManagedStreamingResponse to reliably release result generators and sessions during normal completion, setup failures, cancellation, and client disconnection.
Restrict n to 1–128 and reject incompatible combinations with explicit session_id, cache migration options, and DistServe.
Keep fan-out and streaming lifecycle logic outside serving.py to avoid complicating the existing endpoint.
Add regression coverage for collation, streaming, usage accounting, validation, seed handling, sibling cancellation, setup failures, and resource cleanup.

Copilot

Pull request overview

This PR extends the OpenAI-compatible /v1/chat/completions endpoint to support n > 1 by performing server-side fan-out (N independent engine.generate() calls aggregated into one response), and also includes the chat-completions “package migration” refactor (protocol model relocation + top-level re-exports for backward compatibility).

Changes:

Add server-side fan-out for n > 1 in chat completions, including concurrent collection and streaming interleaving with aggregated usage.
Introduce request validation specific to chat completions, including a cap for n and non-negative seed validation.
Add unit/integration tests covering n > 1 fan-out behavior plus migration/package invariants.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`lmdeploy/serve/openai/endpoints/chat_completions/serving.py`	New packaged handler implementation, including fan-out collect + stream logic and session cleanup behavior.
`lmdeploy/serve/openai/endpoints/chat_completions/validation.py`	New endpoint-specific request validation (fan-out `n` cap, seed validation, etc.).
`lmdeploy/serve/openai/endpoints/chat_completions/protocol.py`	New home for chat-specific Pydantic models (moved out of top-level protocol).
`lmdeploy/serve/openai/endpoints/chat_completions/logprobs.py`	New helper module for building chat logprobs structures.
`lmdeploy/serve/openai/endpoints/chat_completions/logits_processors.py`	New helper module for logit-bias processor construction.
`lmdeploy/serve/openai/endpoints/chat_completions/__init__.py`	Lazily exposes `register` to avoid circular imports during protocol re-export.
`lmdeploy/serve/openai/protocol.py`	Removes inlined chat models and re-exports them from the new chat_completions protocol module.
`lmdeploy/serve/openai/endpoints/chat_completions.py`	Deletes the old flat chat_completions module in favor of the package layout.
`lmdeploy/serve/openai/endpoints/__init__.py`	Lazily exposes `create_openai_router` to avoid circular imports with protocol re-exports.
`tests/test_lmdeploy/serve/openai/chat_completions/test_n_completions.py`	Adds fan-out aggregation + end-to-end handler tests for `n > 1` (streaming and non-streaming).
`tests/test_lmdeploy/serve/openai/chat_completions/conftest.py`	Shared fake engine/session/context and endpoint fixture for chat handler tests.
`tests/test_chat_completions_package_migration.py`	Migration equivalence tests ensuring package structure + re-export invariants.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    # check sampling settings
+    if request.n <= 0:
+        return f'The n {request.n!r} must be a positive int.'
+    # n > 1 is implemented as server-side fan-out (N independent engine
+    # generate() calls). Cap it to prevent unbounded resource use.
+    if request.n > _MAX_FANOUT_N:
+        return (f'The n {request.n!r} exceeds the maximum supported '
+                f'choices ({_MAX_FANOUT_N}).')


+        - **n** (int): How many chat completion choices to generate for each input
+          message. **Only support one here**.
+        - **stream**: whether to stream the results or not. Default to false.


+    the fan-out is therefore engine-agnostic and works for both pytorch and
+    turbomind. If any generator raises, the whole request fails (OpenAI-style:
+    a single n>1 request is all-or-nothing). ``completion_tokens`` is the sum
+    across choices; ``prompt_tokens`` is counted once (taken from
+    ``prompt_tokens`` if provided, else from the first choice's
+    ``input_token_len`` since all choices share the same prompt).


Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

lmdeploy/serve/openai/chat_completions/fanout.py:255

Minor typo in the inline comment: iterm[2] should be item[2].

                # item[1]: index, iterm[2]: usage payload
                usages[item[1]] = item[2]

lmdeploy/serve/openai/chat_completions/validation.py:15

The PR description mentions adding “non-negative seed validation”, but check_request() currently does not validate request.seed at all (and the added tests exercise negative seeds being accepted and wrapped). Either update the PR description to match the implemented behavior, or add explicit seed validation/normalization in the request contract so single-choice and fan-out behavior are clearly defined.

def check_request(request: ChatCompletionRequest,
                  server_context,
                  json_request: dict | None = None) -> str:
    engine_config = server_context.engine_config

Fans a single n>1 request into N independent engine.generate() calls with distinct random_seeds, collating into N choices. Works for both pytorch and turbomind (engine-agnostic handler-layer approach). n==1 keeps the original single-generator fast path. Co-Authored-By: Claude <noreply@anthropic.com>

…ancellation Fix round 1 (code-review findings): - Non-streaming fan-out now wraps _fanout_nonstream in try/finally calling cleanup_result_generators so N fan-out sessions are removed on every exit path (success, parse-error, disconnect, generator-error). Previously leaked N sessions per non-streaming n>1 request. - Fan-out sub-sessions are auto-generated (create_session(None)) instead of reusing request.session_id N times, which collided in SessionManager.map_user_session_id on the 2nd call for explicit session_ids. - _fanout_generate_collect now runs _consume as explicit Tasks and cancels pending siblings on first exception (asyncio.gather does not cancel siblings by default), then awaits cancellations so engine generators close. - _consume wraps the generator in aclosing() for prompt closure on cancel. - Non-streaming fan-out now propagates with_cache cache_block_ids / remote_token_ids response fields (mirrors n==1 path). Tests: added explicit-session-id, sibling-cancellation, multi-chunk stream, and session-cleanup assertions. All 12 n_completions tests pass; 81 serve tests green. Co-Authored-By: Claude <noreply@anthropic.com>

Copilot AI lite review requested due to automatic review settings August 9, 2026 15:09

Copilot started reviewing on behalf of lvhan028 August 9, 2026 15:09 View session

Copilot AI reviewed Aug 9, 2026

View reviewed changes

lvhan028 force-pushed the feat/chat-n-completions branch 7 times, most recently from b50b631 to 3d7e19f Compare August 10, 2026 13:33

lvhan028 added the improvement label Aug 11, 2026

lvhan028 requested review from lzhangzz and a lite review from Copilot August 11, 2026 14:46

Copilot started reviewing on behalf of lvhan028 August 11, 2026 14:50 View session

Copilot AI reviewed Aug 11, 2026

View reviewed changes

Comment thread lmdeploy/serve/openai/chat_completions/fanout.py

lvhan028 force-pushed the feat/chat-n-completions branch from 4bd2cd0 to ebe717a Compare August 12, 2026 16:53

lvhan028 mentioned this pull request Aug 13, 2026

[Feature] Support n parameter in /v1/chat/completions and /v1/completions #4419

Closed

lvhan028 and others added 7 commits August 14, 2026 11:20

refactor(chat): isolate multi-choice fan-out lifecycle

4db830d

perf(chat): batch ready fan-out stream chunks

07c3b0c

docs(chat): document fan-out helpers

8110e02

fix ut

e5ade8b

fix(chat): aggregate reasoning tokens for n choices

f525848

lvhan028 force-pushed the feat/chat-n-completions branch from a92102d to f525848 Compare August 14, 2026 12:21

Navigation Menu

Uh oh!

feat(chat-completions): server-side fan-out for n>1 choices - #4841

feat(chat-completions): server-side fan-out for n>1 choices#4841
lvhan028 wants to merge 7 commits into
InternLM:mainfrom
lvhan028:feat/chat-n-completions

lvhan028 commented Aug 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

lvhan028 commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lvhan028 commented Aug 9, 2026 •

edited

Loading