Overview
As organizations adopt Amazon Elastic Kubernetes Service (EKS) for critical workloads, the security of sensitive data within those clusters becomes paramount. Kubernetes Secrets are essential for managing credentials like API keys, database passwords, and OAuth tokens. However, the default configuration for storing these secrets presents a significant security gap. By default, Kubernetes stores secrets in its etcd datastore as Base64-encoded strings, which is an encoding format, not a form of encryption. Anyone with access to the etcd backups or API can easily decode this data.
This vulnerability undermines the security posture of any application running on EKS. The solution is to implement envelope encryption, a defense-in-depth strategy that adds a crucial layer of security. By integrating Amazon EKS with AWS Key Management Service (KMS), you can ensure that all Kubernetes secrets are encrypted at rest. This process uses a customer-managed key from KMS to encrypt the data encryption keys that, in turn, encrypt the secrets, rendering them unreadable even if the underlying storage is compromised.
Adopting this practice is a core tenet of the AWS Shared Responsibility Model. While AWS manages the EKS control plane, you are responsible for securing the data and configurations within your cluster. Proactively enabling envelope encryption is a critical step in fulfilling that responsibility and building a secure, production-ready Kubernetes environment on AWS.
Why It Matters for FinOps
Failing to properly secure EKS secrets has direct and severe financial consequences. From a FinOps perspective, the cost of non-compliance is not just a technical risk but a significant business liability. A data breach resulting from exposed secrets can trigger millions of dollars in regulatory fines under frameworks like GDPR, HIPAA, and PCI-DSS, where encryption of sensitive data at rest is a mandatory control.
Beyond fines, the operational costs of a breach are staggering. A compromise requires immediate and expensive remediation efforts, including rotating every credential across all connected systems, forensic investigations to determine the scope of the breach, and application downtime that directly impacts revenue. This operational fire drill diverts engineering resources from value-generating projects to crisis management, creating a significant drag on productivity and innovation.
Furthermore, a public security failure leads to irreversible reputational damage. For SaaS providers and enterprises, customer trust is a key asset. The discovery that sensitive data was stored improperly can lead to customer churn, stalled enterprise sales cycles, and a long-term loss of market confidence. Investing in proper security measures like envelope encryption is a cost-effective strategy to prevent these catastrophic financial and business outcomes.
What Counts as “Idle” in This Article
In the context of this security practice, we are not looking for “idle” resources but rather “at-risk” configurations. An Amazon EKS cluster is considered at-risk or non-compliant if it is not configured to use AWS KMS for envelope encryption of its Kubernetes secrets. The default state of an EKS cluster, which relies solely on Base64 encoding, is inherently at-risk.
The primary signal for this vulnerability is a configuration audit that checks the cluster’s encryption settings. If the audit reveals that no customer-managed key from AWS KMS is associated with the EKS cluster for encrypting secrets, the cluster is flagged. This absence of a configured encryption provider means that all sensitive credentials stored within the cluster are protected only by reversible encoding, not true cryptographic encryption.
Common Scenarios
Scenario 1
In multi-tenant SaaS platforms running on a single EKS cluster, workloads for different customers must be strictly isolated. If secrets are not encrypted, a vulnerability in one tenant’s application could potentially expose the credentials of all other tenants. Envelope encryption provides a critical layer of cryptographic isolation, ensuring that even with control plane access, an attacker cannot read the secrets of other tenants.
Scenario 2
Organizations with hybrid cloud architectures often store credentials for on-premises systems, such as VPN keys or Active Directory service accounts, as Kubernetes secrets in EKS. If these secrets are compromised from the cloud environment, an attacker could pivot back into the corporate on-premises network, escalating a cloud breach into a full enterprise compromise.
Scenario 3
Modern GitOps and CI/CD workflows automate the deployment of applications and their configurations, including secrets. While tools may manage secrets securely in transit, once they are applied to the EKS cluster and stored in etcd, they become vulnerable if not encrypted at rest. End-to-end security requires that the final state of the secret within the cluster is protected by envelope encryption.
Risks and Trade-offs
The primary risk of not implementing envelope encryption is the exposure of sensitive credentials. The common misconception that Base64 is a form of security leaves a wide-open door for attackers. If an unauthorized actor gains read access to the cluster’s etcd datastore or its backups, they can decode every secret and gain access to databases, third-party APIs, and other critical systems.
This also eliminates the capability for cryptographic erasure. With envelope encryption, disabling the KMS key instantly renders all secrets encrypted by it permanently inaccessible, providing a powerful and immediate way to contain a breach. Without it, simply deleting a secret from the API does not guarantee it cannot be recovered from storage backups.
The trade-off for implementing this control is minimal, involving a small increase in operational overhead to manage KMS keys and IAM policies. This includes the negligible cost of the KMS keys and API calls. However, when weighed against the catastrophic financial and reputational cost of a data breach, the decision is clear. The robust security, compliance adherence, and risk mitigation provided by envelope encryption far outweigh the minor management effort required.
Recommended Guardrails
To ensure consistent security and prevent configuration drift, organizations should establish strong governance and automated guardrails around EKS secret management.
Start by creating a corporate policy that mandates the use of AWS KMS envelope encryption for all production EKS clusters. This policy should specify the use of customer-managed keys (CMKs) rather than default AWS-managed keys to ensure full control over the key lifecycle, including rotation and permissions.
Implement robust tagging and ownership standards for both EKS clusters and KMS keys. This practice is crucial for cost allocation in showback/chargeback models and for accountability during security audits. Use AWS IAM to enforce least-privilege access, ensuring that only the EKS cluster’s service role has the necessary permissions to use its designated KMS key.
Leverage infrastructure-as-code (IaC) tools to define and deploy EKS clusters, baking your encryption policy directly into the templates. Finally, use automated policy-as-code tools or AWS native services like AWS Config to continuously monitor EKS configurations and trigger alerts or automated remediation if a cluster is detected without envelope encryption enabled.
Provider Notes
AWS
In the AWS ecosystem, securing EKS secrets is achieved through the integration of Amazon EKS and AWS Key Management Service (KMS). When you create or update an EKS cluster, you can specify a customer-managed key (CMK) from KMS to be used for envelope encryption. Once enabled, the EKS control plane uses this key to protect the data encryption keys that encrypt and decrypt your Kubernetes secrets. The master key itself never leaves the secure hardware security modules (HSMs) within KMS, providing a high level of assurance. All KMS API calls related to this activity can be logged in AWS CloudTrail, creating a detailed audit trail for compliance and security analysis.
Binadox Operational Playbook
Binadox Insight: Default configurations are rarely secure enough for enterprise workloads. Relying on the default Base64 encoding for EKS secrets is a significant oversight that links poor security posture directly to financial risk. Proactive configuration is essential for protecting sensitive data and avoiding the high cost of a data breach.
Binadox Checklist:
- Audit all existing Amazon EKS clusters to identify which ones lack envelope encryption.
- Provision dedicated, customer-managed AWS KMS keys for your EKS environments.
- Define and apply strict IAM policies to control which principals can use the KMS keys.
- Systematically enable envelope encryption on all non-compliant clusters.
- Update infrastructure-as-code templates to enforce encryption on all future EKS clusters.
- Configure automated monitoring and alerting to detect any clusters that drift from this compliant state.
Binadox KPIs to Track:
- Percentage of production EKS clusters with envelope encryption enabled.
- Mean Time to Remediate (MTTR) for newly discovered non-compliant clusters.
- Number of high-severity audit findings related to Kubernetes secret management.
- Reduction in security incidents tied to credential exposure from EKS.
Binadox Common Pitfalls:
- Confusing Base64 encoding with true cryptographic encryption, creating a false sense of security.
- Using a single, overly permissive KMS key across multiple disparate workloads and environments.
- Forgetting to grant the EKS cluster’s IAM service role the required
kms:Encryptandkms:Decryptpermissions.- Enabling encryption on a cluster but failing to cycle existing secrets to ensure they are re-written and encrypted.
- Neglecting to configure key rotation policies in AWS KMS, falling short of compliance requirements.
Conclusion
Implementing envelope encryption for Amazon EKS secrets is not an optional enhancement; it is a foundational security requirement for any organization running production workloads on Kubernetes in AWS. The default state of storing secrets as Base64-encoded text is insufficient for protecting the sensitive credentials that power your applications.
By leveraging the native integration between Amazon EKS and AWS KMS, you can establish a robust, auditable, and compliant security posture. This protects your organization from the significant financial and reputational damage of a data breach. The next step is to audit your current environment, identify at-risk clusters, and implement the guardrails needed to make encryption a standard, non-negotiable practice for all your containerized workloads.
7-day free trial