Initial public release

Author: viclzhu

.crux_dry_run_build

@@ -0,0 +1 @@
+AUTOBUILD

.gitignore

@@ -0,0 +1,12 @@
+build
+dist
+*.egg-info
+__pycache__
+cupti_module.*.so
+.pytest_cache
+.coverage*
+/.hatch
+requirements.txt
+*.iml
+/nemo_experiments/*
+/outputs

.pre-commit-config.yaml

@@ -0,0 +1,21 @@
+default_language_version:
+  python: python3
+
+repos:
+
+  - repo: https://github.com/PyCQA/isort
+    rev: 5.13.2
+    hooks:
+      - id: isort
+        exclude: docs/
+
+  - repo: https://github.com/psf/black-pre-commit-mirror
+    rev: 24.10.0
+    hooks:
+      - id: black
+        language_version: python3.10
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.6.9
+    hooks:
+      - id: ruff

CONTRIBUTING.md

@@ -0,0 +1,124 @@
+
+## Nvidia Resiliency Extension (NVRx) OSS Contribution Rules
+
+#### Issue Tracking
+
+* All enhancement, bugfix, or change requests must begin with the creation of a [NVRx Issue Request](TBD).
+  * The issue request must be reviewed by NVRx engineers and approved prior to code review.
+
+
+#### Coding Guidelines
+
+- All source code contributions must follow the existing conventions in the relevant file, submodule, module, and project when you add new code or when you extend/fix existing functionality.
+
+- Avoid introducing unnecessary complexity into existing code so that maintainability and readability are preserved.
+
+- Try to keep pull requests (PRs) as concise as possible:
+  - Avoid committing commented-out code.
+  - Wherever possible, each PR should address a single concern. If there are several otherwise-unrelated things that should be fixed to reach a desired endpoint, our recommendation is to open several PRs and indicate the dependencies in the description. The more complex the changes are in a single PR, the more time it will take to review those changes.
+
+- To ensure code consistency and maintainability across the project, please format and lint your code using the following tools before committing any changes:
+  - We use black to automatically format Python code. It enforces a consistent style by reformatting code according to a set of rules.
+  - To format your code, run:
+```
+black .
+```
+  - isort is used to sort and format import statements automatically. Ensure that your imports are ordered correctly by running:
+```
+isort .
+```
+  - ruff is a fast Python linter that helps catch common issues. Please run ruff to check for and fix linting problems:
+```
+ruff check .
+```
+
+- Write commit titles using imperative mood and [these rules](https://chris.beams.io/posts/git-commit/), and reference the Issue number corresponding to the PR. Following is the recommended format for commit texts:
+```
+#<Issue Number> - <Commit Title>
+
+<Commit Body>
+```
+
+- Ensure that the build log is clean, meaning no warnings or errors should be present.
+
+- Ensure that all unit tests pass prior to submitting your code.
+
+- All OSS components must contain accompanying documentation (READMEs) describing the functionality, dependencies, and known issues.
+
+  - See `README.md` for existing samples and plugins for reference.
+
+- All OSS components must have an accompanying test.
+
+  - If introducing a new component, such as a plugin, provide a test sample to verify the functionality.
+
+- Make sure that you can contribute your work to open source (no license and/or patent conflict is introduced by your code). You will need to [`sign`](#signing-your-work) your commit.
+
+- Thanks in advance for your patience as we review your contributions; we do appreciate them!
+
+
+#### Pull Requests
+Developer workflow for code contributions is as follows:
+
+1. Developers must first [fork](https://help.github.com/en/articles/fork-a-repo) the [upstream](TBD) NVRx OSS repository.
+
+2. Git clone the forked repository and push changes to the personal fork.
+
+  ```bash
+git clone https://github.com/YOUR_USERNAME/YOUR_FORK.git NVRx 
+# Checkout the targeted branch and commit changes
+# Push the commits to a branch on the fork (remote).
+git push -u origin <local-branch>:<remote-branch>
+  ```
+
+3. Once the code changes are staged on the fork and ready for review, a [Pull Request](https://help.github.com/en/articles/about-pull-requests) (PR) can be [requested](https://help.github.com/en/articles/creating-a-pull-request) to merge the changes from a branch of the fork into a selected branch of upstream.
+  * Exercise caution when selecting the source and target branches for the PR.
+    Note that versioned releases of NVRx OSS are posted to `release/` branches of the upstream repo.
+  * Creation of a PR creation kicks off the code review process.
+  * Atleast one NVRx engineer will be assigned for the review.
+  * While under review, mark your PRs as work-in-progress by prefixing the PR title with [WIP].
+
+4. Since there is no CI/CD process in place yet, the PR will be accepted and the corresponding issue closed only after adequate testing has been completed, manually, by the developer and/or NVRx engineer reviewing the code.
+
+
+#### Signing Your Work
+
+* We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
+
+  * Any contribution which contains commits that are not Signed-Off will not be accepted.
+
+* To sign off on a commit you simply use the `--signoff` (or `-s`) option when committing your changes:
+  ```bash
+  $ git commit -s -m "Add cool feature."
+  ```
+  This will append the following to your commit message:
+  ```
+  Signed-off-by: Your Name <[email protected]>
+  ```
+
+* Full text of the DCO:
+
+  ```
+    Developer Certificate of Origin
+    Version 1.1
+    
+    Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
+    1 Letterman Drive
+    Suite D4700
+    San Francisco, CA, 94129
+    
+    Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
+  ```
+
+  ```
+    Developer's Certificate of Origin 1.1
+    
+    By making a contribution to this project, I certify that:
+    
+    (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or
+    
+    (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or
+    
+    (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.
+    
+    (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved.
+  ```

DEVELOPING.md

@@ -0,0 +1,7 @@
+# Developing HyperPodEnginesFaultResiliencyNCCL
+ 
+Put your notes on developing the package here.
+ 
+### Using Peru:
+ 
+* https://builderhub.corp.amazon.com/docs/brazil/user-guide/getting-started-peru.html

LICENSE.txt

@@ -0,0 +1,14 @@
+# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

Makefile

@@ -0,0 +1,31 @@
+.PHONY: all run local build test install
+
+script = ${PWD}/examples/inprocess/sagemaker/hp_basic_example.py
+pp_script = ${PWD}/examples/inprocess/sagemaker/hp_pp_example.py
+sqsh ?= ${PWD}/hyperpod-checkpointless-training+latest.sqsh
+
+all: run
+
+run:
+	${PWD}/examples/inprocess/sagemaker/hprun.sh \
+		-i $(sqsh) \
+		$(script) \
+		--device cuda \
+		--log-interval 5 \
+		--path ${PWD} \
+		--fault-prob 0.001 \
+		--size 256 \
+		--layers 16 \
+		--total-iterations 10000
+
+local:
+	hprun --nproc_per_node=8 $(script) --device cuda --log-interval 5 --total-iterations 1000
+
+build:
+	${PWD}/examples/inprocess/sagemaker/build.sh -n hyperpod-checkpointless-training -f ${PWD}/examples/inprocess/sagemaker/Dockerfile
+
+test:
+	HPWRAPPER_LOG_LEVEL=debug pytest tests/inprocess/sagemaker/ -s
+
+install:
+	pip install --force-reinstall --no-deps .

README.md

@@ -0,0 +1,54 @@
+# NVIDIA Resiliency Extension
+
+The NVIDIA Resiliency Extension (NVRx) integrates multiple resiliency-focused solutions for PyTorch-based workloads.
+
+## Core Components and Capabilities
+
+- **Fault Tolerance**
+  - Detection of hung ranks.  
+  - Restarting training in-job, without the need to reallocate SLURM nodes.
+
+- **In-Process Restarting**
+  - Detecting failures and enabling quick recovery.
+
+- **Async Checkpointing**
+  - Providing an efficient framework for asynchronous checkpointing.
+
+- **Local Checkpointing**
+  - Providing an efficient framework for local checkpointing.
+
+- **Straggler Detection**
+  - Monitoring GPU and CPU performance of ranks.  
+  - Identifying slower ranks that may impede overall training efficiency.
+
+- **PyTorch Lightning Callbacks**
+  - Facilitating seamless NVRx integration with PyTorch Lightning.
+
+## Installation
+
+### From sources
+- `git clone https://github.com/NVIDIA/nvidia-resiliency-ext`
+- `cd nvidia-resiliency-ext`
+- `pip install .`
+
+
+### From PyPI wheel
+- `pip install nvidia-resiliency-ext`
+
+### Platform Support
+
+| Category            | Supported Versions / Requirements            |
+|---------------------|----------------------------------------------|
+| Architecture         | x86_64                                      |
+| Operating System     | Ubuntu 22.04                                |
+| Python Version       | >= 3.10, < 3.13                             |
+| PyTorch Version      | 2.3+                                        |
+| CUDA & CUDA Toolkit  | 12.5+                                       |
+| NVML Driver          | 550 or later                                |
+| NCCL Version         | 2.21.5+                                     |
+
+**Note**: The package is designed to support Python >= 3.10, CUDA >= 11.8, PyTorch >= 2.0 and Ubuntu 20.04, but the recommended and tested environment for production is Python >= 3.10, < 3.13, CUDA 12.5+, and Ubuntu 22.04.
+
+## Usage
+
+For detailed documentation and usage information about each component, please refer to the https://nvidia.github.io/nvidia-resiliency-ext/.

SECURITY.md

@@ -0,0 +1,24 @@
+# Security
+
+NVIDIA is dedicated to the security and trust of our software products and services, including all source code repositories managed through our organization.
+
+If you need to report a security issue, please use the appropriate contact points outlined below. **Please do not report security vulnerabilities through GitHub.**
+
+## Reporting Potential Security Vulnerability in an NVIDIA Product
+
+To report a potential security vulnerability in any NVIDIA product:
+- Web: [Security Vulnerability Submission Form](https://www.nvidia.com/object/submit-security-vulnerability.html)
+- E-Mail: [email protected]
+    - We encourage you to use the following PGP key for secure email communication: [NVIDIA public PGP Key for communication](https://www.nvidia.com/en-us/security/pgp-key)
+    - Please include the following information:
+        - Product/Driver name and version/branch that contains the vulnerability
+     - Type of vulnerability (code execution, denial of service, buffer overflow, etc.)
+        - Instructions to reproduce the vulnerability
+        - Proof-of-concept or exploit code
+        - Potential impact of the vulnerability, including how an attacker could exploit the vulnerability
+
+While NVIDIA currently does not have a bug bounty program, we do offer acknowledgement when an externally reported security issue is addressed under our coordinated vulnerability disclosure policy. Please visit our [Product Security Incident Response Team (PSIRT)](https://www.nvidia.com/en-us/security/psirt-policies/) policies page for more information.
+
+## NVIDIA Product Security
+
+For all security-related concerns, please visit NVIDIA's Product Security portal at https://www.nvidia.com/en-us/security

brazil.ion

@@ -0,0 +1,42 @@
+'[email protected]'
+
+common::{
+  name: "FaradayFaultController",
+  major_version: "1.0",
+
+  dependencies: {
+    default_closure: run,
+
+    closures: {
+      run: public::{
+        include: [self],
+      },
+    },
+
+  build_after: [
+    ]
+  },
+  build: {
+    command: "peru-hatch",
+
+    env: {
+      PATH: [
+        (farm "PeruHatch" "bin"),
+        (env PATH),
+      ],
+    },
+
+    sequences: [
+      {
+        name: 'hyperpod_engines_faraday_sequence',
+        strategy: commit,
+      },
+    ],
+
+    outputs: {
+      public_dir: "./build",
+      private_dir: "./private",
+      cleaned: [],
+    }
+  },
+}