Netflix has built a workload attestation system that lets Apache Spark jobs running on Amazon EMR swap their AWS execution identity for a temporary internal workload identity, eliminating the need to manually inject credentials into managed compute workloads. The approach was detailed by Dhruv Pratap, a Big Data Platform Engineer at Netflix, in a LinkedIn post that described how the streaming company connects AWS Identity and Access Management with its internal Metatron PKI to authenticate workloads to internal services. The design solves a problem unique to environments where cloud providers manage infrastructure but companies need separate internal identity systems for their own services.
Netflix operates two distinct identity systems: AWS supplies managed workloads with an IAM execution role, while internal services rely on Metatron, a private public key infrastructure that issues temporary X.509 certificates for mutual TLS authentication. The company also uses Data Projects as the ownership and identity boundary for data access, meaning a Spark job running on EMR must prove its AWS identity matches the internal identity tied to its Data Project. To handle this, Netflix maps each Data Project to a dedicated IAM role, and because the company expects Data Projects to reach tens of thousands, these roles are deterministically spread across a pool of dedicated AWS accounts rather than concentrated in a single account. When a job is submitted, Netflix's control plane validates the request, resolves the Data Project's IAM role, and creates a signed workload metadata payload that identifies the intended workload—the control plane is the only service authorized to create these claims.
The authentication mechanism works through a multi-step verification process. The Spark driver uses credentials from its execution role to create a pre-signed request to the AWS Security Token Service GetCallerIdentity endpoint, and Netflix's Identity service retrieves the request from AWS STS to determine which IAM role signed it, providing an independently verifiable statement of the workload's AWS identity. The Identity service then compares the AWS response with the signed metadata from the control plane—the AWS response proves possession of the expected IAM role, while the signed metadata identifies the intended workload, and their intersection allows the service to issue a Metatron certificate. According to Pratap, "On infrastructure you build yourself, you can bootstrap your own identity however you like," capturing the challenge the system addresses. The architecture also handles Spark's executor fan-out by having the driver attest once and distribute credentials through authenticated and encrypted Spark RPC, which avoids attestation traffic proportional to executor count while making the driver an additional credential distribution trust boundary.
Credential renewal operates separately from the initial bootstrap process. The driver periodically re-attests before its short-lived certificate expires, while executors don't independently renew credentials—Netflix notes that repeatable attestation is important for long-running workloads. The pattern separates internal workload identity from the managed compute provider while retaining the provider's identity as cryptographic proof, and Netflix says the approach can also apply to other managed environments where workloads receive a cloud identity but need to establish a separate organizational identity. The design effectively turns cloud provider identity into a foundation for internal authentication without requiring manual credential injection at scale. Organizations running large-scale data platforms on managed cloud services face a persistent tension between delegating infrastructure management and maintaining fine-grained internal access controls. Netflix's solution suggests that cryptographic attestation can bridge external and internal identity domains more securely than traditional approaches that rely on embedded secrets or static credentials.