Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract generation_manager from tokenizer_manager #3115

Open
wants to merge 55 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
a2f2f48
empty file
fzyzcjy Jan 25, 2025
815dbc3
empty class
fzyzcjy Jan 25, 2025
3c0e52f
mv MetricManager
fzyzcjy Jan 25, 2025
65b3a37
fix
fzyzcjy Jan 25, 2025
6ce5236
mv _ReqState
fzyzcjy Jan 25, 2025
7ca0a47
mv GenerationConverter.init
fzyzcjy Jan 25, 2025
b88e450
mv tokenize_request
fzyzcjy Jan 25, 2025
3b8ed7b
simp branch
fzyzcjy Jan 25, 2025
2f47f92
tokenize_requests
fzyzcjy Jan 25, 2025
e21a05e
mv postprocess_response
fzyzcjy Jan 25, 2025
ab5d79a
simp code
fzyzcjy Jan 25, 2025
053c8f4
extract _compute_meta_info
fzyzcjy Jan 25, 2025
02c451c
mv convert_logprob_style etc
fzyzcjy Jan 25, 2025
ccd5e8a
make private
fzyzcjy Jan 25, 2025
ecf5e21
mv GenerationManager.init
fzyzcjy Jan 25, 2025
818f8cd
mv GenerationManager body
fzyzcjy Jan 25, 2025
022eb4f
fix import
fzyzcjy Jan 25, 2025
dc53f8f
mv modelconfig
fzyzcjy Jan 25, 2025
c4f1668
call generation_converter
fzyzcjy Jan 25, 2025
1670ce1
fix metrics
fzyzcjy Jan 25, 2025
905d247
fix err
fzyzcjy Jan 25, 2025
41bee7d
handle tokenizer_manager.generate_request
fzyzcjy Jan 25, 2025
2b3ca96
handle abort_request
fzyzcjy Jan 25, 2025
e293f1f
add field
fzyzcjy Jan 25, 2025
2424cf2
rm empty func
fzyzcjy Jan 25, 2025
422ea33
extract _RequestDumper
fzyzcjy Jan 25, 2025
3e6e363
call setup
fzyzcjy Jan 25, 2025
56dcbd1
call handle_batch_output
fzyzcjy Jan 25, 2025
0c08f30
more tokenizer_manager call generation_manager
fzyzcjy Jan 25, 2025
deec6af
use property
fzyzcjy Jan 25, 2025
43dd4e2
call request_dumper
fzyzcjy Jan 25, 2025
2d09b58
call on_request
fzyzcjy Jan 25, 2025
5701e20
fix minor field names
fzyzcjy Jan 25, 2025
cff89f0
fix more field names
fzyzcjy Jan 25, 2025
ba0f1b1
more
fzyzcjy Jan 25, 2025
4b03255
extract _RequestLogger
fzyzcjy Jan 25, 2025
75dc737
extract logger body
fzyzcjy Jan 25, 2025
4100d60
fix err
fzyzcjy Jan 25, 2025
ba4ad8e
fix field
fzyzcjy Jan 25, 2025
9080d45
fmt
fzyzcjy Jan 25, 2025
b1932a6
handle max_req_input_len
fzyzcjy Jan 25, 2025
559ecba
fmt
fzyzcjy Jan 25, 2025
8a10a42
Merge branch 'main' into feat/generation_manager
fzyzcjy Jan 25, 2025
45937e6
Merge branch 'main' into feat/generation_manager
fzyzcjy Jan 26, 2025
cfd3852
Merge branch 'main' into feat/generation_manager
fzyzcjy Jan 26, 2025
4543136
bump ci
fzyzcjy Jan 26, 2025
5e16f96
Merge remote-tracking branch 'origin/feat/generation_manager' into fe…
fzyzcjy Jan 26, 2025
b761936
Merge branch 'main' into feat/generation_manager
fzyzcjy Jan 26, 2025
aeed015
Revert "bump ci"
fzyzcjy Jan 26, 2025
c42431b
Merge remote-tracking branch 'origin/feat/generation_manager' into fe…
fzyzcjy Jan 26, 2025
03b5799
Merge branch 'main' into feat/generation_manager
fzyzcjy Jan 26, 2025
2588e23
Merge branch 'main' into feat/generation_manager
fzyzcjy Jan 26, 2025
7daa570
Merge branch 'main' into feat/generation_manager
fzyzcjy Jan 27, 2025
bbd7908
bump ci
fzyzcjy Jan 26, 2025
e669e45
Revert "bump ci"
fzyzcjy Jan 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix more field names
  • Loading branch information
fzyzcjy committed Jan 25, 2025
commit cff89f0ae956cd64c441d2ba50c25a1de0a661e2
4 changes: 2 additions & 2 deletions python/sglang/srt/managers/generation_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ async def generate(
):
created_time = time.time()

if isinstance(obj, EmbeddingReqInput) and self.is_generation:
if isinstance(obj, EmbeddingReqInput) and self.model_config.is_generation:
raise ValueError(
"This model does not appear to be an embedding model by default. "
"Please add `--is-embedding` when launching the server or try another model."
Expand Down Expand Up @@ -323,7 +323,7 @@ async def tokenize_request(
)
input_ids = self.tokenizer.encode(input_text)

if self.is_generation:
if self.model_config.is_generation:
# TODO: also support getting embeddings for multimodal models
image_inputs: Dict = await self.image_processor.process_images_async(
obj.image_data, input_text or input_ids, obj, self.max_req_input_len
Expand Down