Skip to content

Commit 8c35a7c

Browse files
author
user.email
committed
Initial release
0 parents  commit 8c35a7c

127 files changed

Lines changed: 22326 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/lint.yml

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
name: Lint
5+
6+
on:
7+
push:
8+
branches: [main]
9+
pull_request:
10+
branches: [main]
11+
12+
jobs:
13+
lint:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v4
17+
- uses: actions/setup-python@v5
18+
with:
19+
python-version: "3.10"
20+
- name: Install linters
21+
run: pip install ruff==0.9.7 black==24.10.0
22+
- name: Run ruff
23+
run: ruff check .
24+
- name: Run black
25+
run: black --check .

.gitignore

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
__pycache__/
2+
*.pyc
3+
*.pyo
4+
*.egg-info/
5+
.torch/
6+
inputs
7+
outputs/
8+
wandb/
9+
.ruff_cache/
10+
*.lock
11+
*.ckpt
12+
*.zip
13+
*.pt
14+
*.pth
15+
.venv/
16+
scripts/train_slurm.py
17+
scripts/slurm_init.local.sh
18+
examples/

.gitmodules

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[submodule "third_party/sam-3d-body"]
2+
path = third_party/sam-3d-body
3+
url = https://github.com/facebookresearch/sam-3d-body.git
4+
[submodule "third_party/soma"]
5+
path = third_party/soma
6+
url = https://github.com/NVlabs/SOMA-X.git

.pre-commit-config.yaml

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
repos:
2+
- repo: https://github.com/psf/black
3+
rev: 24.10.0
4+
hooks:
5+
- id: black
6+
language_version: python3.10
7+
args: [--line-length, "100"]
8+
exclude: ^third_party/
9+
- repo: https://github.com/astral-sh/ruff-pre-commit
10+
rev: v0.9.7
11+
hooks:
12+
- id: ruff
13+
args: [--fix]
14+
exclude: ^third_party/
15+
- repo: https://github.com/pre-commit/pre-commit-hooks
16+
rev: v5.0.0
17+
hooks:
18+
- id: trailing-whitespace
19+
exclude: ^third_party/
20+
- id: end-of-file-fixer
21+
exclude: ^third_party/
22+
- id: check-yaml
23+
exclude: ^third_party/
24+
- id: check-added-large-files
25+
args: [--maxkb=500]

ATTRIBUTIONS.md

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
This repository uses the following third-party components. Their respective licenses are reproduced below.
2+
3+
---
4+
5+
## OpenAI — guided-diffusion
6+
7+
https://github.com/openai/guided-diffusion
8+
9+
The following files are derived from OpenAI's guided-diffusion codebase (a PyTorch port of Ho et al., "Denoising Diffusion Probabilistic Models"):
10+
11+
- `gem/diffusion_utils/gaussian_diffusion.py`
12+
- `gem/diffusion_utils/losses.py`
13+
- `gem/diffusion_utils/nn.py`
14+
- `gem/diffusion_utils/respace.py`
15+
16+
**License:** MIT License
17+
18+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
19+
20+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
21+
22+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
23+
24+
---
25+
26+
## Meta Platforms / Facebook Research — PyTorch3D / ACTOR
27+
28+
https://github.com/facebookresearch/pytorch3d
29+
https://github.com/Mathux/ACTOR
30+
31+
The following file is derived from Facebook's PyTorch3D library (via the ACTOR project):
32+
33+
- `gem/utils/rotation_conversions.py`
34+
35+
Copyright (c) Facebook, Inc. and its affiliates. All rights reserved.
36+
37+
**License:** BSD 3-Clause License
38+
39+
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
40+
41+
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
42+
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
43+
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
44+
45+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
46+
47+
---
48+
49+
## SAM 3D Body
50+
51+
https://github.com/facebookresearch/sam-3d-body
52+
53+
SAM License
54+
Last Updated: November 19, 2025
55+
56+
"Agreement" means the terms and conditions for use, reproduction, distribution and modification of the SAM Materials set forth herein.
57+
58+
"SAM Materials" means, collectively, Documentation and the models, software and algorithms, including machine-learning model code, trained model weights, inference-enabling code, training-enabling code, fine-tuning enabling code, and other elements of the foregoing distributed by Meta and made available under this Agreement.
59+
60+
"Documentation" means the specifications, manuals and documentation accompanying
61+
SAM Materials distributed by Meta.
62+
63+
"Licensee" or "you" means you, or your employer or any other person or entity (if you are entering into this Agreement on such person or entity's behalf), of the age required under applicable laws, rules or regulations to provide legal consent and that has legal authority to bind your employer or such other person or entity if you are entering in this Agreement on their behalf.
64+
65+
"Meta" or "we" means Meta Platforms Ireland Limited (if you are located in or, if you are an entity, your principal place of business is in the EEA or Switzerland) or Meta Platforms, Inc. (if you are located outside of the EEA or Switzerland).
66+
67+
"Sanctions" means any economic or trade sanctions or restrictions administered or enforced by the United States (including the Office of Foreign Assets Control of the U.S. Department of the Treasury ("OFAC"), the U.S. Department of State and the U.S. Department of Commerce), the United Nations, the European Union, or the United Kingdom.
68+
69+
"Trade Controls" means any of the following: Sanctions and applicable export and import controls.
70+
71+
By using or distributing any portion or element of the SAM Materials, you agree to be bound by this Agreement.
72+
73+
1. License Rights and Redistribution.
74+
75+
a. Grant of Rights. You are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under Meta's intellectual property or other rights owned by Meta embodied in the SAM Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the SAM Materials.
76+
77+
i. Grant of Patent License. Subject to the terms and conditions of this License, you are granted a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by Meta that are necessarily infringed alone or by combination of their contribution(s) with the SAM 3 Materials. If you institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the SAM 3 Materials incorporated within the work constitutes direct or contributory patent infringement, then any patent licenses granted to you under this License for that work shall terminate as of the date such litigation is filed.
78+
79+
b. Redistribution and Use.
80+
81+
i. Distribution of SAM Materials, and any derivative works thereof, are subject to the terms of this Agreement. If you distribute or make the SAM Materials, or any derivative works thereof, available to a third party, you may only do so under the terms of this Agreement and you shall provide a copy of this Agreement with any such SAM Materials.
82+
83+
ii. If you submit for publication the results of research you perform on, using, or otherwise in connection with SAM Materials, you must acknowledge the use of SAM Materials in your publication.
84+
85+
iii. Your use of the SAM Materials must comply with applicable laws and regulations, including Trade Control Laws and applicable privacy and data protection laws.
86+
87+
iv. Your use of the SAM Materials will not involve or encourage others to reverse engineer, decompile or discover the underlying components of the SAM Materials.
88+
89+
v. You are not the target of Trade Controls and your use of SAM Materials must comply with Trade Controls. You agree not to use, or permit others to use, SAM Materials for any activities subject to the International Traffic in Arms Regulations (ITAR) or end uses prohibited by Trade Controls, including those related to military or warfare purposes, nuclear industries or applications, espionage, or the development or use of guns or illegal weapons.
90+
91+
2. User Support. Your use of the SAM Materials is done at your own discretion; Meta does not process any information nor provide any service in relation to such use. Meta is under no obligation to provide any support services for the SAM Materials. Any support provided is "as is", "with all faults", and without warranty of any kind.
92+
93+
3. Disclaimer of Warranty. UNLESS REQUIRED BY APPLICABLE LAW, THE SAM MATERIALS AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, AND META DISCLAIMS ALL WARRANTIES OF ANY KIND, BOTH EXPRESS AND IMPLIED, INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE SAM MATERIALS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE SAM MATERIALS AND ANY OUTPUT AND RESULTS.
94+
95+
4. Limitation of Liability. IN NO EVENT WILL META OR ITS AFFILIATES BE LIABLE UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, FOR ANY LOST PROFITS OR ANY DIRECT OR INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES, EVEN IF META OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF ANY OF THE FOREGOING.
96+
97+
5. Intellectual Property.
98+
99+
a. Subject to Meta's ownership of SAM Materials and derivatives made by or for Meta, with respect to any derivative works and modifications of the SAM Materials that are made by you, as between you and Meta, you are and will be the owner of such derivative works and modifications.
100+
101+
b. If you institute litigation or other proceedings against Meta or any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the SAM Materials, outputs or results, or any portion of any of the foregoing, constitutes infringement of intellectual property or other rights owned or licensable by you, then any licenses granted to you under this Agreement shall terminate as of the date such litigation or claim is filed or instituted. You will indemnify and hold harmless Meta from and against any claim by any third party arising out of or related to your use or distribution of the SAM Materials.
102+
103+
6. Term and Termination. The term of this Agreement will commence upon your acceptance of this Agreement or access to the SAM Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein. Meta may terminate this Agreement if you are in breach of any term or condition of this Agreement. Upon termination of this Agreement, you shall delete and cease use of the SAM Materials. Sections 3, 4 and 7 shall survive the termination of this Agreement.
104+
105+
7. Governing Law and Jurisdiction. This Agreement will be governed and construed under the laws of the State of California without regard to choice of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement. The courts of California shall have exclusive jurisdiction of any dispute arising out of this Agreement.
106+
107+
8. Modifications and Amendments. Meta may modify this Agreement from time to time; provided that they are similar in spirit to the current version of the Agreement, but may differ in detail to address new problems or concerns. All such changes will be effective immediately. Your continued use of the SAM Materials after any modification to this Agreement constitutes your agreement to such modification. Except as provided in this Agreement, no modification or addition to any provision of this Agreement will be binding unless it is in writing and signed by an authorized representative of both you and Meta.

CHANGELOG.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Changelog
2+
3+
## v1.0.0 — Initial Release
4+
5+
- Monocular video 3D human pose estimation using SOMA body model (77 joints)
6+
- Bundled 2D pose estimation model (77 SOMA keypoints)
7+
- Demo scripts for full 3D pipeline (`demo_soma.py`) and 2D keypoint visualization (`demo_2d_keypoints.py`)
8+
- Apache 2.0 licensed for commercial use

CONTRIBUTING.md

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# How to Contribute
2+
3+
## Contribution License
4+
5+
Contributions will only be accepted under the Apache-2.0 license. By contributing to this project, you agree that your contributions will be licensed under the [Apache 2.0 License](LICENSE).
6+
7+
## Code Reviews
8+
9+
All submissions require review. We use GitHub pull requests for this purpose. Consult
10+
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more information on using pull requests.
11+
12+
## Signing Your Work
13+
14+
* We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
15+
16+
* Any contribution which contains commits that are not Signed-Off will not be accepted.
17+
18+
* To sign off on a commit you simply use the `--signoff` (or `-s`) option when committing your changes:
19+
```bash
20+
$ git commit -s -m "Add cool feature."
21+
```
22+
This will append the following to your commit message:
23+
```
24+
Signed-off-by: Your Name <your@email.com>
25+
```
26+
27+
* Full text of the DCO:
28+
29+
```
30+
Developer Certificate of Origin
31+
Version 1.1
32+
33+
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
34+
1 Letterman Drive
35+
Suite D4700
36+
San Francisco, CA, 94129
37+
38+
Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
39+
```
40+
41+
```
42+
Developer's Certificate of Origin 1.1
43+
44+
By making a contribution to this project, I certify that:
45+
46+
(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or
47+
48+
(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or
49+
50+
(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.
51+
52+
(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved.
53+
```

Dockerfile

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
FROM nvidia/cuda:12.6.0-devel-ubuntu22.04
5+
6+
ENV DEBIAN_FRONTEND=noninteractive
7+
8+
# System dependencies
9+
RUN apt-get update && apt-get install -y --no-install-recommends \
10+
python3.10 \
11+
python3.10-venv \
12+
python3.10-dev \
13+
python3-pip \
14+
git \
15+
git-lfs \
16+
wget \
17+
curl \
18+
gosu \
19+
xvfb \
20+
libegl1-mesa-dev \
21+
libgl1-mesa-glx \
22+
libglib2.0-0 \
23+
&& rm -rf /var/lib/apt/lists/*
24+
25+
RUN git lfs install
26+
27+
# Install uv
28+
RUN pip install uv
29+
30+
# Set up workspace
31+
WORKDIR /workspace/gem
32+
COPY . /workspace/gem
33+
34+
# Create virtual environment
35+
RUN uv venv .venv --python 3.10
36+
37+
# Install PyTorch
38+
RUN . .venv/bin/activate && \
39+
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
40+
41+
# Install SOMA body model
42+
RUN . .venv/bin/activate && \
43+
uv pip install -e third_party/soma
44+
45+
# Pull SOMA LFS assets
46+
RUN cd third_party/soma && git lfs pull && cd ../..
47+
48+
# Install GEM and dependencies
49+
RUN . .venv/bin/activate && \
50+
bash scripts/install_env.sh
51+
52+
# Headless rendering environment
53+
ENV PYOPENGL_PLATFORM=egl
54+
ENV EGL_PLATFORM=surfaceless
55+
56+
# Activate venv by default
57+
ENV PATH="/workspace/gem/.venv/bin:${PATH}"
58+
ENV VIRTUAL_ENV="/workspace/gem/.venv"
59+
60+
ENTRYPOINT ["tools/docker-entrypoint.sh"]
61+
CMD ["bash"]

0 commit comments

Comments
 (0)