Skip to content

Fix libnvdla_compiler.so COPY path for Jetson 6.x runtime - #2306

Closed
alexnorell wants to merge 3 commits into
mainfrom
fix/libnvdla-compiler-copy-path
Closed

Fix libnvdla_compiler.so COPY path for Jetson 6.x runtime#2306
alexnorell wants to merge 3 commits into
mainfrom
fix/libnvdla-compiler-copy-path

Conversation

@alexnorell

Copy link
Copy Markdown
Contributor

Summary

  • Corrects COPY --from=builder source path from /usr/lib/aarch64-linux-gnu/tegra/libnvdla_compiler.so* to /usr/lib/aarch64-linux-gnu/nvidia/libnvdla_compiler.so* in both Dockerfile.onnx.jetson.6.0.0 and Dockerfile.onnx.jetson.6.2.0.
  • The previous path didn't exist in nvcr.io/nvidia/l4t-jetpack:r36.x, so the glob silently matched zero files and produced an empty layer. That layer cached forever, so the fix in PR Fix missing libnvdla_compiler.so in Jetson 6.x TRT runtime #2201 never actually shipped a working libnvdla_compiler.so into the runtime image.

Why this matters

On Jetson 6.x devices, ONNX Runtime's TensorRT execution provider needs libnvdla_compiler.so. Without it:
```
[E:onnxruntime:Default] Failed to load library libonnxruntime_providers_tensorrt.so:
libnvdla_compiler.so: cannot open shared object file: No such file or directory
EP Error ... Falling back to ['CPUExecutionProvider'] and retrying.
```
The model then runs entirely on CPU, ~3x slower (we observed 21 FPS camera vs. 1.3 FPS inference on an AI1/Orin NX).

Verification

Inspected nvcr.io/nvidia/l4t-jetpack:r36.4.0 directly:
```
$ docker run --rm --platform linux/arm64 nvcr.io/nvidia/l4t-jetpack:r36.4.0
sh -c 'find / -name libnvdla*'
/usr/lib/aarch64-linux-gnu/nvidia/libnvdla_compiler.so ← actual location
```

Test plan

  • Trigger Jetson 6.2.0 container build via workflow_dispatch with a custom tag and verify /usr/local/cuda/lib64/libnvdla_compiler.so* is present in the resulting image.
  • Pull on a JP6.2 device, run inference container, confirm ORT loads TensorrtExecutionProvider instead of falling back to CPU.
  • Compare inference FPS before/after on a representative workflow (expecting ~3x improvement).

The COPY in PR #2201 used /usr/lib/aarch64-linux-gnu/tegra/, but in
nvcr.io/nvidia/l4t-jetpack:r36.x the actual path is
/usr/lib/aarch64-linux-gnu/nvidia/. The glob silently matched zero
files and produced an empty layer (cached forever), so libnvdla_compiler.so
never made it into the runtime image. ONNX Runtime's TRT execution
provider then failed to load and silently fell back to CPU.

Verified on the AI1 device: ONNXRuntime logs:
  Failed to load library libonnxruntime_providers_tensorrt.so:
  libnvdla_compiler.so: cannot open shared object file
  Falling back to ['CPUExecutionProvider'] and retrying.

Inspecting nvcr.io/nvidia/l4t-jetpack:r36.4.0 confirms
/usr/lib/aarch64-linux-gnu/nvidia/libnvdla_compiler.so is the correct path.
Jetson 6.0 is no longer maintained. Drops the dedicated build/test
workflows, Dockerfile, container_adapter mappings, and docs referencing
roboflow-inference-server-jetson-6.0.0.

- Delete .github/workflows/docker.jetson.6.0.0.yml
- Delete .github/workflows/test.jetson_6.0.0.yml
- Delete docker/dockerfiles/Dockerfile.onnx.jetson.6.0.0
- Drop L4T 36.0 / 36.3 → 6.0.0 entries from container_adapter
- Drop matching test parametrize cases
- Update install guide and example READMEs to point at 6.2.0
@alexnorell
alexnorell marked this pull request as draft May 2, 2026 15:08
alexnorell added a commit that referenced this pull request May 11, 2026
The actual location in nvcr.io/nvidia/l4t-jetpack:r36.4.0 is
/usr/lib/aarch64-linux-gnu/nvidia/libnvdla_compiler.so, not tegra/.
The original COPY in PR #2201 used the wrong path; the glob silently
matched zero files, so the libnvdla fix shipped an empty layer and
ONNX Runtime kept falling back to CPU on Jetson 6.x.

Same one-line correction as PR #2306. Bundling it here so this PR's
build can also actually ship libnvdla_compiler.so.
@alexnorell

Copy link
Copy Markdown
Contributor Author

Superseded by #2321

@alexnorell alexnorell closed this May 12, 2026
alexnorell added a commit that referenced this pull request May 12, 2026
The actual location in nvcr.io/nvidia/l4t-jetpack:r36.4.0 is
/usr/lib/aarch64-linux-gnu/nvidia/libnvdla_compiler.so, not tegra/.
The original COPY in PR #2201 used the wrong path; the glob silently
matched zero files, so the libnvdla fix shipped an empty layer and
ONNX Runtime kept falling back to CPU on Jetson 6.x.

Same one-line correction as PR #2306. Bundling it here so this PR's
build can also actually ship libnvdla_compiler.so.
PawelPeczek-Roboflow added a commit that referenced this pull request May 14, 2026
… mounts on JP62/JP71, bump JP51 OpenCV (#2321)

* Build OpenCV from source with GStreamer + bake NVIDIA gst plugins into JP62

The pip opencv-python wheel ships without GStreamer, and the l4t-cuda runtime
base used for the JP62 image has no multimedia stack, so cv2.VideoCapture on
Jetson silently falls through to the plain v4l2 ioctl path -- pulling raw 4K
YUYV, doing CPU YUV->BGR, and never reaching NVIDIA's hardware engines.

Three pieces, all needed together:

1. Compile OpenCV from source in the builder stage with WITH_GSTREAMER=ON,
   WITH_FFMPEG=ON, WITH_LIBV4L=ON, WITH_CUDA=ON. Replaces the pip-installed
   opencv-python/opencv-contrib-python with a wheel built from python_loader.
   Adds a build-time cv2.getBuildInformation() check so a regression in the
   detected flags fails the build instead of silently shipping CPU-only cv2.

2. Add the GStreamer runtime + PyGObject stack to the runtime stage
   (libgstreamer1.0-0, plugins-base/good, python3-gi, gir typelibs) so that
   downstream Python code using "from gi.repository import Gst" to construct
   hardware capture/encode pipelines can import and run inside the container.

3. Copy NVIDIA's GStreamer plugins (libgstnv*.so) plus the tegra runtime libs
   from the JetPack builder stage into the runtime image, instead of relying
   on nvidia-container-passthrough host mounts. Self-contained -- works
   without any host-side changes deployed.

Branched off v1.2.7 to keep the diff minimal.

* Pass CMAKE_POLICY_VERSION_MINIMUM=3.5 to OpenCV cmake

CMake 4.x removed compatibility with cmake_minimum_required(VERSION <3.5).
OpenCV 4.10.0's OpenCVGenPkgconfig.cmake still uses the old declaration,
so the configure step aborts with:

  CMake Error at .../OpenCVGenPkgconfig.cmake:113 (cmake_minimum_required):
    Compatibility with CMake < 3.5 has been removed from CMake.

Same workaround the existing onnxruntime build step uses.

* Bump OpenCV to 4.13.0 and reorder JP62 build

OpenCV 4.13.0 (released 2025-12-31) is the current latest stable; it builds
cleanly under CMake 4.x without the cmake_minimum_required workaround the
4.10.0 source needed. Aligns both Jetson images.

JP62-specific:
- Move the from-source OpenCV step to AFTER the inference packages install
  so the cv2 we install (built with GStreamer + FFmpeg) is the last thing
  to touch /usr/local/lib/python3.10/dist-packages/cv2/. Previously the
  inference_core/cli/sdk pip install would refetch opencv-python (as a
  transitive dep) and clobber our from-source cv2.
- Drop CMAKE_POLICY_VERSION_MINIMUM=3.5 -- not needed on 4.13.0.

* Drop the bake-in attempt for NVIDIA gst plugins

The l4t-jetpack:r36.4.0 builder image does not ship the tegra runtime
libs as a regular populated /usr/lib/aarch64-linux-gnu/tegra directory.
Those libs are tied to the host JetPack BSP and only appear inside the
container at run time via nvidia-container-runtime's CSV-driven host
mounts. So `COPY --from=builder /usr/lib/aarch64-linux-gnu/tegra ...`
errors during the build with "not found".

This image now scopes to enabling the cv2 side of the GStreamer stack
(WITH_GSTREAMER=ON OpenCV, python-gi, base gstreamer runtime + GIR
typelibs). The host system has to provide the NVIDIA plugins for any
nv* pipeline element to resolve at run time.

* mkdir tegra/ at end of builder so libnvdla glob COPY can lstat parent

BuildKit needs to lstat the source parent dir before evaluating a glob
COPY pattern, even when "no match" is allowed. The libnvdla fix in
PR #2201 copies tegra/libnvdla_compiler.so*, but l4t-jetpack:r36.4.0
doesn't always ship /usr/lib/aarch64-linux-gnu/tegra/ as a real
directory -- that path is normally populated by nvidia-container-runtime
at container start. Recent builds intermittently fail at this step
depending on layer cache state.

mkdir -p the path so BuildKit can process the COPY. The glob will copy
the real file if it's there, or silently no-op if not. Doesn't paper
over the underlying flakiness of the libnvdla source path, but lets
this PR's build progress.

* Correct libnvdla_compiler.so source path from tegra/ to nvidia/

The actual location in nvcr.io/nvidia/l4t-jetpack:r36.4.0 is
/usr/lib/aarch64-linux-gnu/nvidia/libnvdla_compiler.so, not tegra/.
The original COPY in PR #2201 used the wrong path; the glob silently
matched zero files, so the libnvdla fix shipped an empty layer and
ONNX Runtime kept falling back to CPU on Jetson 6.x.

Same one-line correction as PR #2306. Bundling it here so this PR's
build can also actually ship libnvdla_compiler.so.

* Set OpenCV python install path so the loader stub finds cv2 at runtime

Without OPENCV_PYTHON3_INSTALL_PATH / PYTHON3_INCLUDE_DIR / PYTHON_VERSION
the python_loader config.py bakes the BUILDER path
(/build/opencv/<ver>/release/lib/python3) into sys.path. That path only
exists in the builder stage, so in the runtime image the loader stub
fails to find the native cv2 .so, falls back to importlib.import_module("cv2")
which re-enters the same stub package, and Python raises:

  ImportError: ERROR: recursion is detected during loading of "cv2"
               binary extensions. Check OpenCV installation.

JP51's PR #2100 passes these flags explicitly; replicating the same set
here so the loader references /usr/local/lib/python3.10/dist-packages.

* Rescue install-tree cv2 config files; parameterize for porting

Two related changes to make the from-source OpenCV build correct and
reusable across jp5.1 / jp6.2 / jp7.1:

1. Rescue the install-tree config files. After ninja install, the files
   at ${INSTALL_PATH}/cv2/config*.py have install-tree paths (correct).
   The pip install of the wheel built from /build/.../python_loader then
   overwrites them with BUILD-TREE config files (paths like
   /build/opencv/opencv-X.Y.Z/release/lib/python3) which only exist in
   the builder stage. In a multi-stage runtime image those paths are
   dead, so the loader's bootstrap can't find the native cv2 .so,
   re-imports the stub package, and Python raises:
       ImportError: recursion is detected during loading of "cv2"
   Fix by saving config*.py between ninja install and pip wheel/install,
   restoring after.

2. Pull per-platform variables out of the cmake invocation into ARGs at
   the top of the builder (OPENCV_CUDA_ARCH, OPENCV_PYTHON_INSTALL_PATH,
   OPENCV_PYTHON_INCLUDE_DIR, OPENCV_PYTHON_VERSION). Replicating this
   OpenCV block in jp5.1.1 / jp7.1.0 dockerfiles is now a copy-paste
   plus four ARG value changes; no surgery inside the cmake invocation.

Adds a build-time sanity check that simulates the runtime stage by
stripping /build/ from sys.path before importing cv2. If the install-
tree rescue ever regresses, the docker build fails here -- not 90
minutes later on the device.

* Add BuildKit cache mounts to Jetson builds and split Depot projects

Pins syntax to dockerfile:1.7 on both jp62 and jp71 so ARG interpolation
in cache-mount ids works. Adds apt cache mounts (sharing=locked, id scoped
per JetPack + builder/runtime + arch) to every apt RUN so .debs and lists
persist across builds. Disables docker-clean and turns on
Keep-Downloaded-Packages so the apt cache is actually used. Drops the
trailing rm of /var/lib/apt/lists/* since that path is now the cache mount.

Adds per-build-tree cache mounts on the expensive C++ compiles:
PyTorch, torchvision, onnxruntime (build/cuda12 on jp62, build/Linux on
jp71), OpenCV (release/ on jp62), with the TRT .deb download also cached
on jp62. Each id is scoped by version + arch + SM so a version bump
starts clean instead of reusing a stale build tree.

The OpenCV step on jp62 stays positioned last in the builder because the
inference wheel installs upstream are not --no-deps and an earlier
opencv-python pull would clobber the from-source build. The build-dir
cache mount makes the forced rebuild after COPY-invalidation a fast
ninja no-op rather than a 30-60 min recompile.

Points jp62 and jp71 workflows at the new dedicated Depot projects
(2rp7mfjw7q for jp62, v1xzfwkc4b for jp71) so they no longer share an
NVMe cache pool and evict each other.

* Split git-clone out of cache-mounted build RUNs

The cache mount on build/ auto-creates its parent directory before the
RUN body executes, so a combined 'git clone ... && cd ... && make' fails
with 'destination path already exists and is not an empty directory'.

First jp62 attempt with these cache mounts failed at the PyTorch step:
  fatal: destination path 'pytorch' already exists and is not an empty
  directory.

Split each of PyTorch / torchvision / onnxruntime on jp62 and jp71 into
two RUNs: a clone RUN with no mount, then a build RUN whose cache mount
overlays build/ on the cloned source layer.

* Bump all GitHub Actions to latest major versions

Bumps:
- actions/cache v3 -> v5
- actions/checkout v2/v3/v4 -> v6
- actions/create-github-app-token v1 -> v3
- actions/setup-node v3 -> v6
- actions/setup-python v2/v5 -> v6
- actions/upload-artifact v4 -> v7
- aws-actions/amazon-ecr-login v1 -> v2
- aws-actions/configure-aws-credentials v2 -> v6
- dcarbone/install-jq-action v2.1.0 -> v3.2.0
- digicert/ssm-code-signing v1.0.0 -> v1.2.1
- docker/login-action v2/v3 -> v4
- docker/setup-buildx-action v2 -> v4
- docker/setup-qemu-action v2 -> v4
- google-github-actions/auth v2 -> v3
- google-github-actions/setup-gcloud v2 -> v3

depot/* already pinned to floating v1 (latest major).
actions/upload-release-asset and pypa/gh-action-pypi-publish kept
on their existing pins (v1 already latest; pypi-publish uses the
release/v1 stable-channel ref recommended by upstream).

* Bump jp71 flash-attn MAX_JOBS from 1 to 2

PR #2068 reduced this to 1 because MAX_JOBS=4 was starving the Depot
BuildKit gRPC connection during flash-attn CUDA kernel compilation.
2 is a middle ground: flash-attn's nvcc instances peak ~3-4 GB each
against PyTorch 2.10 headers, so 2 concurrent fits comfortably on a
32 GB arm builder while halving the wall time vs serial.

With the cache mounts from c5701f2 in place, the cost of a regression
here is ~30-60 min of flash-attn rebuild, not the full 4hr build.
jp62 stays at MAX_JOBS=4 because it's on PyTorch 2.6 (lighter headers)
and that build is already known-good.

---------

Co-authored-by: Paweł Pęczek <146137186+PawelPeczek-Roboflow@users.noreply.github.com>
grzegorz-roboflow pushed a commit that referenced this pull request Sep 4, 2026
…egra/ (#2924)

"Build and Push Jetson 6.X Container" has failed on every push to main
since 2026-06-19 with:

    COPY --from=builder /usr/lib/aarch64-linux-gnu/tegra/libnvdla_compiler.so* /usr/local/cuda/lib64/
    ERROR: lstat .../usr/lib/aarch64-linux-gnu/tegra: no such file or directory

In l4t-jetpack r36.x the library lives at
/usr/lib/aarch64-linux-gnu/nvidia/libnvdla_compiler.so. The tegra/ dir is
populated by nvidia-container-runtime host mounts at run time only, so it is
absent in the builder stage and BuildKit cannot lstat the glob's parent.

#2321 made the same one-line correction for Dockerfile.onnx.jetson.6.2.0 but
left the 6.0.0 Dockerfile on the old path (#2306 covered both and was
superseded). This also means the runtime image now actually ships
libnvdla_compiler.so, which the ORT TensorRT EP needs to load (#2201).


Claude-Session: https://claude.ai/code/session_01QxRB1NT6sXZpDiMcTDBg9t

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant