Skip to content

From NVIDIA Megatron-LM for visibility #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
RaymondLi0 wants to merge 6,282 commits into
base: multi-query-attention
Choose a base branch
from
Open

From NVIDIA Megatron-LM for visibility #18

RaymondLi0 wants to merge 6,282 commits into from

Conversation

@RaymondLi0
Copy link
Collaborator

@RaymondLi0 RaymondLi0 commented Jan 24, 2023

No description provided.

@RaymondLi0 RaymondLi0 changed the base branch from multi-query-attention to before-merge June 20, 2023 20:12
@RaymondLi0 RaymondLi0 changed the base branch from before-merge to multi-query-attention June 20, 2023 20:12
yuzhongw-nvidia and others added 28 commits January 16, 2026 16:58
@yuzhongw-nvidia @jstjohn @deepakn94
[main] feat(moe): Support apply wd to qk layernorm for Qwen3-Next (4/…
622a06a 
…4) (#2753)

Signed-off-by: John St. John <[email protected]>
Co-authored-by: John St. John <[email protected]>
Co-authored-by: Deepak Narayanan <[email protected]>
@asolergi-nv @ko3n1g
Set token_dtype_code init value in GPTDatasetConfig to fix CI (#2912
b305422 
)

Co-authored-by: oliver könig <[email protected]>
@hexinw-nvidia @rhewett-nv @chtruong814
Re-submit "Various fixes to in-job restarter and better time accounti…
c40b6ea 
…ng of startup operations" (#2954)

Co-authored-by: Russell Hewett <[email protected]>
Co-authored-by: Charlie Truong <[email protected]>
@Phlip79
Use slack-sdk in a different manner (#2950)
100e11e
@lmcafee-nvidia
Inference | Move assert active_request_count > 0. (#2958)
b705148
@parthmannan @xiaoyao0115 @kunlunl
Hybrid Context Parallel Feature (#2282)
98d8c56 
Signed-off-by: tailaim <[email protected]>
Signed-off-by: Parth Mannan <[email protected]>
Co-authored-by: Mcore Bot <[email protected]>
Co-authored-by: tailaim <[email protected]>
Co-authored-by: kunlunl <[email protected]>
Co-authored-by: Kunlun Li <[email protected]>
@yuzhongw-nvidia
[main] ci(moe): Add --apply-wd-to-qk-layernorm flag to the gdn test…
829e461 
… case (#2995)
@ko3n1g
ci: Disable step time on `gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_n… (
a191791 
#2991)

Signed-off-by: oliver könig <[email protected]>
@ko3n1g
ci: Fix workflows on main (#2990)
b49d810 
Signed-off-by: oliver könig <[email protected]>
@shjwudp @cspades
Make Megatron-FSDP torch.compile compatible (#2425)
35129e7 
Co-authored-by: Cory Ye <[email protected]>
@cspades
[Megatron-FSDP] Test FP8 activations + parameter sharding with Megatr…
517dfd4 
…on-FSDP fully-shard. Update README. (#2894)

Signed-off-by: Cory Ye <[email protected]>
@ko3n1g
chore: Escape special chars (#3014)
2f3fa0f 
Signed-off-by: oliver könig <[email protected]>
@deepakn94
Improve memory logging (#2839)
82ea022 
Signed-off-by: Deepak Narayanan <[email protected]>
@github-actions
chore: rotate oncall schedule
bcdd405
@santhnm2
Add a wrapper function for FA3 _flash_attn_forward call (#2933)
a3615d7 
Signed-off-by: Keshav Santhanam <[email protected]>
@ko3n1g
ci(hotfix): Fix unit tests on main
a1ba844 
Signed-off-by: oliver könig <[email protected]>
@ko3n1g
ci(hotfix): Tests on main
e4b18f7 
Signed-off-by: oliver könig <[email protected]>
@ko3n1g
chore: Set umask 0002 (#3027)
ba876ef 
Signed-off-by: oliver könig <[email protected]>
@mathemakitten
Make attn mask inversion in-place instead of allocating it again (#3019)
28c7221
@cspades
[Megatron-FSDP] Fix incorrect gradient scaling target. (#3023)
ba456fd 
Signed-off-by: Cory Ye <[email protected]>
@santhnm2
Ensure that last prefill chunk is handled correctly by Mamba models (#…
096dbeb 
…2897)

Signed-off-by: Keshav Santhanam <[email protected]>
@nschank
Replaces ModuleSpec with Protocols for some of the inputs to SelfAtte…
90e685b 
…ntion/CrossAttention (#2761)
@Phlip79
Update oncall schedule (#3017)
dd72aee
@jiemingz @buptzyb
Various CUDA graph improvements on capture time, replay time, memory …
8baf014 
…footprint (#2572)

Signed-off-by: Jimmy Zhang <[email protected]>
Signed-off-by: Jieming Zhang <[email protected]>
Co-authored-by: Robin Zhang <[email protected]>
@github-actions
chore: rotate oncall schedule
bbbedbb
@jon-barker
Add script for batch running CI tests across distinct nodes (#3047)
8c768ea
@ko3n1g
Revert "Various CUDA graph improvements on capture time, replay time,…
c8caeb2 
… memory footprint (#2572)"

This reverts commit 8baf014.
@wdykas @jon-barker
Refit EP support (#2972)
db7a8a8 
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: William Dykas <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Jon Barker <[email protected]>
asolergi-nv and others added 30 commits February 10, 2026 21:39
@asolergi-nv
Fix SFT Pipeline when TP>1 (#3268)
4bce841
@AAnoosheh
Fixes for KD mode (#3342)
f5238ba 
Signed-off-by: Asha Anoosheh <[email protected]>
@github-actions
chore: rotate oncall schedule
c1169ea
@ko3n1g
chore: Update codeowners file (#3365)
4f025a1 
Signed-off-by: oliver könig <[email protected]>
@sidsingh-nvidia @ko3n1g
Siddharth/fix inference functional tests (#3357)
66ec17e 
Co-authored-by: oliver könig <[email protected]>
@janEbert @ko3n1g
Switch oncall (#3360)
6a9da99 
Co-authored-by: oliver könig <[email protected]>
@AAnoosheh
Add missing RMSNorm to llama train script (#3314)
b6e883b 
Signed-off-by: Asha Anoosheh <[email protected]>
@tdene
Fix inference for MTP models (#3297)
cd14090
@yobibyte
Add a logprobs test with real gpt model. (#2870)
6f5de16
@tdene
Add simple GRPO functional test (#3323)
b5d50cb
@ko3n1g
ci: Concurrency control for merge-queue (#3353)
1c245c7 
Signed-off-by: oliver könig <[email protected]>
@chtruong814
ci: Update golden value download script to work with Github (#3335)
d9f075c 
Signed-off-by: Charlie Truong <[email protected]>
@megnvidia
Removing etc from main index page, shifted name of discussions (#3271)
d0b768f 
Signed-off-by: meg miranda <[email protected]>
@thecaptain789 @Phlip79
fix: correct typos 'seperated' and 'recieved' (#3305)
7d1acf6 
Co-authored-by: thecaptain789 <[email protected]>
Co-authored-by: Philip Petrakian <[email protected]>
@shengf-nv
Improved PyTorch profiler and added PyTorch execution trace (#3273)
2807a4e
@ko3n1g
build: Bump TE on 2.12 (#3371)
1959739 
Signed-off-by: oliver könig <[email protected]>
@ko3n1g
ci(hotfix): job conditions (#3376)
f06e669 
Signed-off-by: oliver könig <[email protected]>
@github-actions
Update copy-pr-bot.yaml [skip ci]
6467cbc
@sidsingh-nvidia @ISEEKYAN
Record moe routing decisions during inference. (#3034)
faced51 
Co-authored-by: litianjian <[email protected]>
Co-authored-by: Yan Bai <[email protected]>
@Wohox @dimapihtar
[Main] Fix EP Overlap Bugs for Full-Iter CG (#3164)
dedb6dd 
Co-authored-by: Dmytro Pykhtar <[email protected]>
@maanug-nv
Avoid direct pickle import (#3375)
11a4659 
Signed-off-by: Maanu Grover <[email protected]>
@Phlip79
Delete old pretrain_* files (#3359)
fe9279e
@xuwchen
Add Qwen3-VL support with Megatron-FSDP (#2841)
7df15bd
@santhnm2
Refactor Mamba chunked prefill (#3265)
c65fb25 
Signed-off-by: Keshav Santhanam <[email protected]>
@jstjohn @ko3n1g
Improved parallel logging of learning rate (#3319)
47938af 
Co-authored-by: oliver könig <[email protected]>
@lmcafee-nvidia @claude
Add enhanced event tracking with TTFT measurement and compact seriali…
a51c1c8 
…zation. (#3253)

Co-authored-by: Claude Opus 4.5 <[email protected]>
@santhnm2
Add assertion that max_requests is divisible by tp_size (#3304)
dbc444f 
Signed-off-by: Keshav Santhanam <[email protected]>
@ArEsKay3 @ISEEKYAN
@sidsingh-nvidia @ko3n1g @tdene
Move to using the Inference OpenAI API server (#3107)
1123fc0 
Co-authored-by: litianjian <[email protected]>
Co-authored-by: Yan Bai <[email protected]>
Co-authored-by: Siddharth Singh <[email protected]>
Co-authored-by: Robert Kirby <[email protected]>
Co-authored-by: Siddharth Singh <[email protected]>
Co-authored-by: oliver könig <[email protected]>
Co-authored-by: Teodor-Dumitru Ene <[email protected]>
Co-authored-by: Teodor-Dumitru Ene <[email protected]>
@ko3n1g
Revert "Move to using the Inference OpenAI API server (#3107)"
4184bfa 
This reverts commit 1123fc0.
@Victarry @cursoragent @Phlip79
Update moe github test cases. (#3077)
9119fae 
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Philip Petrakian <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.