Skip to content
Ghaf Framework

Cross Compilation

Ghaf is targeted at a range of devices and form factors that support different instruction set architectures (ISA). Many small form-factor edge devices are not powerful enough to compile the needed applications or OSs that run on them. As the most common ISA used in desktops and servers is x86_64, this requires cross-compilation for target ISAs such as AArch64.

NixOS and Nixpkgs have excellent support for cross-compilation, and Ghaf leverages this extensively for ARM64 targets.

Cross-Compilation Targets

Section titled “Cross-Compilation Targets”

All Ghaf targets with the -from-x86_64 suffix are cross-compiled from x86_64 host systems to their target architecture. These are the recommended targets for building AArch64 systems from x86_64 development machines.

NVIDIA Jetson Platform (Primary AArch64 Targets)

Section titled “NVIDIA Jetson Platform (Primary AArch64 Targets)”

The NVIDIA Jetson family represents Ghaf’s primary AArch64 platform with full cross-compilation support:

Jetson AGX Orin (32GB)

Section titled “Jetson AGX Orin (32GB)”
Terminal window
# Debug variants
nix build .#nvidia-jetson-orin-agx-debug-from-x86_64
nix build .#nvidia-jetson-orin-agx-debug-nodemoapps-from-x86_64


# Release variants
nix build .#nvidia-jetson-orin-agx-release-from-x86_64
nix build .#nvidia-jetson-orin-agx-release-nodemoapps-from-x86_64


# Flash script generation
nix build .#nvidia-jetson-orin-agx-debug-from-x86_64-flash-script

Jetson AGX Orin (64GB)

Section titled “Jetson AGX Orin (64GB)”
Terminal window
# Debug variants
nix build .#nvidia-jetson-orin-agx64-debug-from-x86_64
nix build .#nvidia-jetson-orin-agx64-debug-nodemoapps-from-x86_64


# Release variants
nix build .#nvidia-jetson-orin-agx64-release-from-x86_64
nix build .#nvidia-jetson-orin-agx64-release-nodemoapps-from-x86_64

Jetson Orin NX

Section titled “Jetson Orin NX”
Terminal window
# Debug variants
nix build .#nvidia-jetson-orin-nx-debug-from-x86_64
nix build .#nvidia-jetson-orin-nx-debug-nodemoapps-from-x86_64


# Release variants
nix build .#nvidia-jetson-orin-nx-release-from-x86_64
nix build .#nvidia-jetson-orin-nx-release-nodemoapps-from-x86_64

NXP i.MX Platform (Secondary AArch64 Target)

Section titled “NXP i.MX Platform (Secondary AArch64 Target)”
Terminal window
# Debug variant
nix build .#nxp-imx8mp-evk-debug-from-x86_64


# Release variant
nix build .#nxp-imx8mp-evk-release-from-x86_64

Cross-Compilation Benefits

Section titled “Cross-Compilation Benefits”

Performance

Section titled “Performance”
  • Faster builds: Leverage powerful x86_64 development machines
  • Consistent environment: Use familiar development tools and IDEs
  • CI/CD integration: Build ARM64 targets in standard x86_64 CI infrastructure

Development Workflow

Section titled “Development Workflow”
  • No hardware dependency: Build ARM64 targets without ARM64 hardware
  • Remote building: Use remote builders for consistent cross-compilation
  • Reproducible builds: Identical output regardless of build host

Setting Up Cross-Compilation

Section titled “Setting Up Cross-Compilation”

Prerequisites

Section titled “Prerequisites”

For cross-compilation to work properly, ensure your development environment has:

  1. Sufficient resources: Cross-compilation can be resource-intensive

    • Minimum 16GB RAM recommended
    • Fast NVMe storage preferred
    • Multiple CPU cores for parallel builds

  2. Network access: Download dependencies and use binary caches

    Terminal window
    # Configure Ghaf binary cache for faster builds
    nix build --option substituters "https://ghaf-dev.cachix.org https://cache.nixos.org"

Remote Builder Setup (Optional)

Section titled “Remote Builder Setup (Optional)”

For consistent cross-compilation across teams, consider setting up remote builders:

# In /etc/nixos/configuration.nix or ~/.config/nix/nix.conf
builders = @/etc/nix/machines
builders-use-substitutes = true

See Remote Build Setup for detailed configuration.

Native vs Cross-Compiled Builds

Section titled “Native vs Cross-Compiled Builds”

Target Naming Convention

Section titled “Target Naming Convention”
  • Native builds: nvidia-jetson-orin-agx-debug (built on/for same architecture)
  • Cross-compiled builds: nvidia-jetson-orin-agx-debug-from-x86_64 (built on x86_64 for AArch64)

When to Use Each

Section titled “When to Use Each”

Use cross-compilation (-from-x86_64) when:

  • Building on x86_64 development machines (most common)
  • Setting up CI/CD pipelines
  • Need faster build times
  • Don’t have native AArch64 hardware for building

Use native builds when:

  • Building directly on target hardware
  • Debugging hardware-specific issues
  • Final validation on actual hardware

Troubleshooting Cross-Compilation

Section titled “Troubleshooting Cross-Compilation”

Common Issues

Section titled “Common Issues”

Build failures with missing dependencies:

Terminal window
# Clear Nix store and retry
nix store gc
nix build .#target --rebuild

Slow builds:

Terminal window
# Use binary cache and parallel builds
nix build .#target \
  --option substituters "https://ghaf-dev.cachix.org https://cache.nixos.org" \
  --option max-jobs 8

Memory issues:

Terminal window
# Limit parallel jobs to reduce memory usage
nix build .#target --option max-jobs 2

Build Time Expectations

Section titled “Build Time Expectations”

Cross-compilation build times for AArch64 targets:

  • NVIDIA Jetson targets: 90-180 minutes (first build)
  • NXP i.MX targets: 60-120 minutes (first build)
  • Subsequent builds: 5-30 minutes (with binary cache)

Next Steps

Section titled “Next Steps”

binfmt Emulated Build (Alternative Method)

Section titled “binfmt Emulated Build (Alternative Method)”

For targets that do not have full cross-compilation support or when debugging architecture-specific issues, binfmt provides an alternative approach by allowing execution of different ISA binaries on your development machine through emulation.

How binfmt Works

Section titled “How binfmt Works”

binfmt enables running target architecture binaries in an emulator such as QEMU. While not true cross-compilation, this allows building for different architectures when cross-compilation is not available or when you need to debug architecture-specific behavior.

Setting Up binfmt

Section titled “Setting Up binfmt”

To enable binfmt, add the following to your host system’s configuration.nix:

boot.binfmt.emulatedSystems = [
  "aarch64-linux"
];

For optimal performance, also enable KVM support:

For AMD processors:

boot.kernelModules = [ "kvm-amd" ];

For Intel processors:

boot.kernelModules = [ "kvm-intel" ];

When to Use binfmt

Section titled “When to Use binfmt”

Use binfmt emulation when:

  • Cross-compilation targets are not available (-from-x86_64 variants don’t exist)
  • Debugging architecture-specific issues
  • Testing compatibility with target architecture
  • Working with packages that don’t support cross-compilation

Performance considerations:

  • Slower than cross-compilation: Emulation overhead can be significant
  • Higher resource usage: QEMU emulation requires more CPU and memory
  • Build times: Expect 2-4x longer build times compared to native or cross-compilation

Example Usage

Section titled “Example Usage”

With binfmt configured, you can build native AArch64 targets on x86_64:

Terminal window
# Build native AArch64 targets (emulated on x86_64)
nix build .#nvidia-jetson-orin-agx-debug
nix build .#nxp-imx8mp-evk-debug


# This works through binfmt emulation when built on x86_64

binfmt vs Cross-Compilation

Section titled “binfmt vs Cross-Compilation”
MethodPerformanceCompatibilityUse Case
Cross-compilationFastHighPrimary method for supported targets
binfmt emulationSlowerUniversalFallback for unsupported targets

Recommendation: Always prefer cross-compilation (-from-x86_64 targets) when available, and use binfmt emulation only for targets without cross-compilation support.