EnclaveOS: Defense in depth for trusted execution environments

Table of contents

• Introduction
• The problem with single-layer attestation
• EnclaveOS architecture
• How EnclaveOS mitigates real attacks
• Platform support
• Attestation deep dive
• Full-source bootstrapping and reproducible builds: The missing link
• Threat model and limitations
• Conclusion
• Get started with EnclaveOS
• Further reading
• References

Introduction

Trusted Execution Environments (TEEs), more informally called “Secure Enclaves”, promise something powerful: compute with hardware-enforced isolation and integrity. The idea is simple - use silicon to create guarantees that software alone cannot provide. Technologies like Intel SGX, Intel TDX, AMD SEV, TPMs, and the AWS Nitro System all aim to deliver confidentiality and integrity for code and data, even when the underlying infrastructure is untrusted.

Imagine being able to prove a VPN or Tor node was not logging your data, that a service provider is actually following their stated privacy policy, that no single system administrator can access your funds at a major custodian, or that a computer that is no longer running faithfully generated artifacts from expected source code.

Remote attestation, properly implemented, can make this and a lot more possible.

A remote attestation document is a hardware signed bundle of cryptographic hashes called PCRs that certify what operating system, software, and configuration is running on a remote computer. We often call this a “bootproof”.

However, if the key used to sign that attestation leaks, or it is possible to execute malicious code inside the enclave, or there is a failure in a particular hardware implementation, all bets are off. Unfortunately secure enclave deployments we have seen or audited have no mitigation for these design flaws.

EnclaveOS is our practical and low cost answer to these challenges. It’s a minimal Linux operating system that targets multiple remote attestation and isolation technologies with a QubesOS inspired, nested VM architecture. An outer offline “host” OS manages two virtual machines: an air-gapped Enclave VM for sensitive computation and secondly a Gateway VM with direct network hardware access. Both are running in parallel linked only by a linux vsock, enforced by IOMMU hardware isolation.

This post explains why this matters, how it works, and how you can leverage this technology in your systems.

The problem with single-layer attestation

Genkin-style side-channel attacks: Primer

A significant body of research often called “Genkin-style attacks” after cryptographer Daniel Genkin has demonstrated that these hardware guarantees can be undermined through physical and microarchitectural “side channel” attacks. Beyond these attacks, the supply chain for TEE technologies is riddled with opaque firmware, proprietary binaries, and tamper-resistant hardware that resists independent verification.

When one uses an enclave, HSM, TEE or similar technology, their goals typically include high integrity and high privacy.

Even for applications where your own application has no secrets, you still need a private key for attestation at a minimum.

If that key leaks to an attacker, they can forge an attestation to effectively “loop the security camera feed”, as is cliche in action movies, and hide malicious code.

Microarchitectural side-channel attacks follow a remarkably consistent pattern¹:

1. Trigger speculative or transient execution that accesses secret data.
2. Encode the secret into microarchitectural state, typically the CPU cache.
3. Measure timing differences to infer the secret.

The canonical examples are Meltdown and Spectre, disclosed in 2018². Meltdown exploited out-of-order execution to read kernel memory from userspace. Spectre exploited branch prediction to leak data across security boundaries. Since then, researchers have discovered dozens of variants: Foreshadow (L1 Terminal Fault), Microarchitectural Data Sampling attacks (RIDL, Fallout, ZombieLoad), Load Value Injection, and many others³.

What makes these attacks particularly devastating for TEEs is that they violate the fundamental security model. Intel SGX, for example, promises that enclave memory is protected even from a compromised operating system. However, Foreshadow demonstrated that a malicious OS could extract SGX enclave secrets through L1 cache side channels⁴. The hardware boundary that was supposed to be impenetrable turned out otherwise.

The supply chain problem

Even if there was a perfect side-channel resistance, there’s another issue: how does one trust the TEE implementation itself?

Modern enclave stacks involve:

• Hardware that uses tamper-proofing designed to destroy components if physically inspected.
• Firmware that is proprietary, signed by the vendor, and often delivered as opaque binary blobs.
• Microcode that can be updated by the vendor, changing CPU behavior in ways that are difficult to audit.
• SDKs and runtime libraries that mediate between code and the enclave hardware.

Each of these is a potential point of compromise. Intel’s SGX system, for example, relies on a chain of trust rooted in keys burned into the CPU during manufacturing. One has to trust that Intel’s key management is secure, that their firmware has no vulnerabilities, and that no one in the supply chain has tampered with the hardware⁵.

This is not theoretical paranoia. The ÆPIC Leak vulnerability (2022) demonstrated that Intel SGX could leak enclave data through the APIC MMIO interface, a bug that existed in silicon for years⁶.

EnclaveOS architecture

EnclaveOS takes a common sense defense in depth approach using the best available software defenses coupled to make use of multiple of the mostly widely available hardware isolation and attestation technologies. Rather than running sensitive workloads alongside network-connected code and hoping hardware boundaries hold, we enforce physical separation through a dual-VM architecture with IOMMU isolation.

Design principles

1. Air-gapped by default: The enclave has no network stack.
2. Hardware enforced isolation: CPU IOMMU features isolates the memory of each VM
3. Minimal attack surface: Only essential kernel features enabled, targeting KSPP compliance.
4. Immutable infrastructure: Root filesystem lives entirely in RAM, extracted from a CPIO archive at boot.
5. Bootstrapping and reproducibility: The entire stack is full source bootstrapped and reproducible.
6. Multi-platform attestation: Support for multiple attestation backends prevents single points of failure and vendor lock-in.

System Architecture

The following diagram shows the complete EnclaveOS architecture (open full size):

Component breakdown

Let’s walk through each component and its role in the security architecture:

Host layer

IOMMU boundary

The IOMMU (Input/Output Memory Management Unit) creates hardware-enforced isolation between the two guest VMs, with the Host OS linking them together via linux vsock. This is crucial: even if one VM is completely compromised, it cannot access the memory or devices assigned to the other VM without a novel side channel attack in the CPU that can also bypass memory encryption.

Enclave VM (air-gapped)

Gateway VM (network-connected)

Hardware layer

How EnclaveOS mitigates real attacks

Against side-channel attacks

1. Memory Encryption: The enclave runs in a dedicated VM using hardware memory encryption on supported CPUs. Even if a side channel does exist that allows an attacker to read raw memory, they will not be able to decrypt it without access to the CPU registers holding the memory encryption key, thus significantly complicating the attack.

2. Platform Diversity: Usually if CPU security is fundamentally compromised due to a design flaw as with the Genkin style attacks, all bets are off, however by supporting multiple attestation platforms, in many cases you have the option to run enclaves with different CPUs in series or in parallel such as to tolerate one of them being compromised at any time. This only provides mitigation of attacks which impact authenticity, availability, and integrity. In the case where sensitive data that should not be leaked is present on the CPU, confidentiality is violated.

Against remote code execution attacks

1. VM Isolation: The enclave runs in a dedicated VM using hardware memory encryption on supported CPUs. Even if an attacker has code execution in the Gateway VM, they’re in a different virtual machine, hardware isolated with separate page tables and (depending on the TEE backend) separate cache partitions.

2. Network isolation: You can’t exploit a vulnerability you can’t reach. The Enclave VM has no network interface, so remote attackers have no direct path.

3. IOMMU enforcement: Hardware prevents the Gateway VM from accessing memory regions assigned to the Enclave VM, even via DMA attacks.

4. Minimal kernel surface: We compile out unnecessary kernel features following Kernel Self Protection Project recommendations. Fewer features mean fewer potential gadgets for speculative execution attacks.

Against man in the middle attacks

1. Multi-platform attestation: We support multiple TEE backends (Nitro, TPM 2.0, TDX, SEV) on each target. Assume compromise unless they both agree on the enclave PCRs.

2. Hardware E2EE: All hardware backends have some kind of encryption support. The “bootproof” attestations will contain signatures of dedicated enclave held keys you can encrypt to. For high security use cases you can encrypt to both.

Against supply chain attacks

1. Deterministic and bootstrapped builds: Our build system produces byte-for-byte identical artifacts regardless of who builds them or on what machine. You can verify that the binary you’re running corresponds exactly to the published source code.

2. Minimal trusted computing base: Less code means less surface area for supply chain attacks to hide in.

3. FOSS Software: The entire system is freely licensed open source software and auditable at git.distrust.co/public/enclaveos.

Against malicious insiders / compromised infrastructure

Even if an attacker has root access to the Gateway VM:

• They cannot read Enclave VM memory (IOMMU).
• They cannot attach a debugger to the Enclave VM.
• They cannot access the disk (assigned to Enclave VM).
• They cannot reach the Host operating system.
• They can only communicate via the defined vsock API.

The enclave remains reasonably protected unless the attacker has multiple 0-days on the core security primitives of a single platform, though as mentioned earlier, a service capable of distributing trust across multiple machines can take advantage of diversified CPU/attestation technologies.

Platform support

EnclaveOS is designed to be portable across multiple TEE platforms:

The goal is to support multiple platforms so that users can:

• Avoid vendor lock-in.
• Run the same workload across platforms.
• Use multiple attestation sources for defense in depth.

Attestation deep dive

The Bootproof system

Attestation in EnclaveOS is handled by the bootproof system, consisting of two components:

• bootproof-agent: Runs on the unreachable host OS, interfaces with available attestation hardware.
• bootproofd: HTTP API running in the Gateway VM, serves attestation documents to clients.

When a client requests attestation:

1. Client connects to bootproofd via the Gateway VM.
2. bootproofd requests a fresh attestation from bootproof-agent via vsock.
3. bootproof-agent interfaces with the hardware (TPM, Nitro, etc.) to generate a signed attestation document.
4. The attestation is returned to the client, who can verify it against the platform’s root of trust.

AWS Nitro attestation

On AWS Nitro, attestation documents contain Platform Configuration Registers (PCRs)⁷:

The attestation document is a COSE-signed CBOR structure containing these PCRs, along with an enclave-specific public key and a certificate chain rooting to AWS’s Nitro attestation PKI.

TPM 2.0 attestation

For platforms using TPM 2.0 (GCP, Azure, generic), attestation uses the standardized TPM quote mechanism⁸:

OS-Owned PCRs (8–15)

TPM PCRs use an extend-only model: they can only be modified by extending with new measurements, never set directly. This creates a tamper-evident log of the boot process.

Intel TDX attestation

Intel Trust Domain Extensions (TDX) provides VM-level isolation with hardware-rooted attestation⁹. TDX operates at VM granularity, exactly matching our dual-VM architecture.

TDX attestation involves two structures: the TD Report (local) and TD Quote (remote).

TD Report (generated by CPU via TDCALL[TDG.MR.REPORT], 1024 bytes)

The TD Report contains three components:

Key measurement registers in TDINFO_STRUCT:

TD Quote (generated by Quoting Enclave, remotely verifiable)

The TD Report is MAC’d with a hardware key only accessible on the local platform. For remote attestation, the TDQE (TD Quoting Enclave) verifies the MAC via EVERIFYREPORT2, then signs the measurements with its Attestation Key:

TD Report (MAC’d) → TDQE verifies locally → TD Quote (ECDSA signed)

┌─────────────────────────────────────────────────────────────┐
│                         TD Quote                            │
├─────────────────────────────────────────────────────────────┤
│  Header (48 bytes)                                          │
│    - Version (4), TEE Type (0x81 = TDX), QE Vendor ID       │
├─────────────────────────────────────────────────────────────┤
│  TD Quote Body (584 bytes)                                  │
│    - TEE_TCB_SVN, MRSEAM, MRTD, RTMR[0-3], REPORTDATA       │
├─────────────────────────────────────────────────────────────┤
│  Signature (ECDSA-256)                                      │
│    - Signed by QE's Attestation Key (AK)                    │
│    - AK certified by PCK → Intel Root CA                    │
├─────────────────────────────────────────────────────────────┤
│  Certification Data                                         │
│    - PCK Certificate chain to Intel Root CA                 │
│    - Enables offline verification                           │
└─────────────────────────────────────────────────────────────┘

Verification involves checking the signature chain up to Intel’s root certificate, then comparing the MRTD and RTMR values against expected measurements from a reproducible build.

EnclaveOS integration: On TDX-enabled platforms, bootproof-agent requests a TD Quote via the /dev/tdx_guest interface. The RTMR registers are extended during boot to measure the EnclaveOS kernel, init system, and user service, providing a complete chain from CPU microcode to your application.

AMD SEV-SNP Attestation

AMD Secure Encrypted Virtualization with Secure Nested Paging (SEV-SNP) provides memory encryption and integrity protection for VMs, with attestation rooted in the AMD Platform Security Processor (PSP)¹⁰.

Platform measurements:

Guest measurements:

Verification Chain

AMD Root Key (ARK)
       │
       ▼
AMD SEV Key (ASK)
       │
       ▼
Versioned Chip Endorsement Key (VCEK)
       │
       ▼
┌─────────────────────────┐
│   Attestation Report    │
│   - MEASUREMENT         │◄── Compare against reproducible build
│   - REPORT_DATA         │◄── Verify nonce/public key binding
│   - POLICY              │◄── Check security policy flags
│   - COMMITTED_TCB       │◄── Verify firmware is up-to-date (rollback protection)
│   - SIGNATURE           │◄── Verify with VCEK
└─────────────────────────┘

The VCEK is unique per processor and per TCB version. AMD provides a Key Distribution Service (KDS) where you can fetch the VCEK certificate for any chip, enabling offline verification.

Key Security Properties

• Memory encryption: All guest memory is encrypted with a per-VM key; the hypervisor cannot read it.
• Integrity protection: SNP adds reverse map tables (RMP) that prevent the hypervisor from remapping guest pages.
• No migration by default: The POLICY field can prohibit VM migration, ensuring secrets never leave the original CPU.

EnclaveOS integration: On SEV-SNP platforms, bootproof-agent interfaces with the PSP via /dev/sev-guest to generate attestation reports. The MEASUREMENT field captures the hash of the initial VM image-which, thanks to deterministic builds, can be independently reproduced and verified. The REPORT_DATA field is used to bind the attestation to a fresh nonce or the enclave’s public key, preventing replay attacks.

Why Multiple Attestation Backends Matter

Each attestation technology has different trust assumptions:

By supporting multiple backends, EnclaveOS allows you to:

1. Avoid single-vendor trust: Run on AMD hardware to avoid trusting Intel, or vice versa.
2. Layer attestations: On platforms with both TPM and CPU TEE, get independent attestations from both.
3. Migrate without lock-in: Move workloads between clouds without rewriting attestation logic.

The bootproof system abstracts these differences, presenting a unified attestation interface regardless of the underlying hardware.

A note on Intel SGX

You may notice that Intel SGX is not included among EnclaveOS’s supported attestation backends. SGX operates at process-level granularity: it creates isolated enclaves within a running operating system, where the OS itself is untrusted but still present. EnclaveOS, by contrast, is architected around VM-level isolation-the Enclave VM and Gateway VM are entire virtual machines separated by IOMMU hardware boundaries.

Intel TDX is the VM-granularity equivalent of SGX, designed to protect entire virtual machines from the hypervisor rather than processes from the OS. Since EnclaveOS’s security model depends on full VM isolation (no shared kernel, no shared network stack), TDX is the appropriate Intel technology. SGX would require a fundamentally different architecture where sensitive code runs as an enclave within a host OS-precisely the co-residency model we’re trying to avoid. That said TDX does make use of some SGX technologies under the hood for attestation.

Full-source bootstrapping and reproducible builds: The missing link

A Nitro attestation tells you the hash of the EIF file. A TPM quote tells you the hash of the kernel. But a hash is meaningless if you can’t independently verify which source code produced that hash, which is in turn impossible unless it is possible to build the entire system image and everything in it exactly and get the same hash every time.

This is why full source bootstrapped and deterministic builds are a core foundation required for EnclaveOS to exist, not an afterthought.

To this end, the Distrust team designed and built StageX, a Linux distribution and build toolchain focused on full source bootstrapping and reproducibility. Lance Vick, a security engineer and co-founder of Distrust, first published the “packages” repo under the Distrust umbrella which was forked twice and ultimately spun out as StageX for public release as a standalone community project.

It has since been adopted by many security-critical organizations and projects in multiple industry sectors. The Distrust team still acts as the primary maintainers and contributors to StageX, though we have since, and will continue to, welcome many external maintainers to limit bus factor and ensure StageX is here to stay long term, free of control from any single entity.

Threat model and limitations

We believe in being explicit about what EnclaveOS does and does not protect against.

What EnclaveOS defends against

What EnclaveOS does NOT defend against

Note: On physical attacks, we have a project in R&D we feel will dramatically reduce this risk.

Trust assumptions

EnclaveOS assumes:

1. At least one attestation layer is honest: If all enclave security hardware is compromised, we cannot detect it.
2. IOMMU is correctly implemented: We rely on hardware for VM isolation.
3. Cryptographic primitives are secure: We use standard cryptography (ECDSA, SHA-256, etc.).
4. You verify the builds: Attestations are meaningless if you don’t check them against reproducible builds.

Note: For #1 and #2 you can mitigate this in applications that can distribute trust across multiple machines using different supported enclave technologies.

Conclusion

Single-layer hardware attestation was a reasonable starting point for enclave technology, but it’s insufficient for high-security applications. The combination of side-channel attacks and supply chain risks means that trusting a single hardware vendor-or running network-connected code alongside secrets-is accepting significant unnecessary risk.

We’re not claiming EnclaveOS is unbreakable. We’re claiming it’s significantly harder to break than the alternatives, and that when something does go wrong, you have alternative platform choices, and the layered architecture limits blast radius and improves forensics.

Further reading

• Meltdown and Spectre - Comprehensive resource on transient execution attacks.
• TEE.fail - Collection of TEE vulnerabilities and research.
• Kernel Self Protection Project - Kernel hardening recommendations.
• Reproducible Builds Project - Community effort for reproducible software.
• StageX - Reproducible package distribution.
• Intel TDX Documentation - Official Intel TDX specifications and whitepapers.
• AMD SEV-SNP Developer Resources - AMD’s confidential computing documentation.
• Confidential Computing Consortium - Industry consortium for TEE standardization.

Found an error, or want to contribute? Open an issue or PR on the repository.

References