Amazon Simple Storage Service (S3) is the primary data storage system for cloud-native platforms, housing assets ranging from user uploads to database backups and financial logs. However, default S3 configurations prioritize flexibility, which can lead to accidental data exposure. Accidental public bucket leaks remain one of the most common security breaches in cloud environments.
Securing cloud storage requires a layered security model. Teams must disable legacy Access Control Lists (ACLs), enforce S3 Block Public Access globally, implement Server-Side Encryption with Customer Managed Keys (SSE-KMS), and configure bucket policies that restrict access to specific Virtual Private Cloud (VPC) Endpoints while denying non-HTTPS connections.
This guide analyzes S3 security architectures, provides a complete production-grade Terraform HCL configuration with S3 bucket policies, compares S3 encryption models, and details four common production failure modes with tested mitigations.
S3 Authorization Layering and Access Policy Evaluation
When a request attempts to read or write an object in an S3 bucket, AWS evaluates multiple authorization layers in sequence. The request is authorized only if it is explicitly allowed by all active policies and has no matching explicit denials:
AWS S3 Access Request Evaluation Path:
[Incoming Read/Write Request]
│
▼
[S3 Block Public Access] ──► Denies public ACLs and public bucket policies
│
▼
[IAM Policy (User/Role)] ──► Checks caller permissions
│
▼
[S3 Bucket Policy] ───────► Checks resource-based constraints (e.g. source IP, SSL)
│
▼
[KMS Key Policy] ─────────► Evaluates decrypt/encrypt permission for Customer Managed Keys
│
▼
[Access Granted / Denied]
1. S3 Block Public Access
This account and bucket-level override acts as a safety barrier. When enabled, it blocks any configurations that attempt to expose the bucket to the public internet, ignoring any public permissions configured in ACLs or bucket policies.
2. IAM Policies
Identity-based policies are attached to the users, groups, or roles making the request. They specify what actions the caller is authorized to execute on S3 resources.
3. S3 Bucket Policies
Resource-based policies are attached directly to the bucket. They are evaluated alongside identity policies. Bucket policies are ideal for enforcing structural constraints, such as blocking requests that do not use SSL transport or restricting access to specific corporate network ranges.
4. KMS Key Policies
If a bucket is encrypted using Server-Side Encryption with a Customer Managed Key (SSE-KMS), the caller must also possess permissions on the KMS key. The KMS key policy must explicitly authorize the caller to execute decrypt and encrypt operations.
KMS Envelope Encryption and S3 Bucket Keys
By default, S3 encrypts objects using AWS-managed keys (SSE-S3). While this meets basic encryption-at-rest requirements, SSE-S3 does not provide audit logs for key access and cannot restrict key operations to specific IAM roles.
To secure sensitive compliance data, teams should use Customer Managed Keys (CMKs) in AWS KMS:
- Envelope Encryption: S3 generates a unique symmetric data key for each object. S3 uses this key to encrypt the raw object data. The data key itself is then encrypted using the root Customer Managed Key (CMK) in KMS and stored alongside the object metadata in S3.
- S3 Bucket Keys: By default, each S3 read/write call triggers a call to KMS to encrypt or decrypt the data key. Under heavy workloads, this can cause applications to exceed KMS request limits, leading to performance bottlenecks and high KMS charges. Enabling S3 Bucket Keys configures S3 to cache a temporary bucket-level key in memory. This reduces KMS API calls by more than 99%, lowering operational costs.
Production Integration Code: Terraform S3 Configuration
Below is a complete, production-grade Terraform HCL configuration. It provisions a KMS Customer Managed Key, creates a secure S3 bucket with ACLs disabled, enforces S3 Block Public Access, enables S3 Bucket Keys, and attaches a bucket policy that denies non-HTTPS requests and restricts read/write operations to a specific VPC Gateway Endpoint.
# main.tf
terraform {
required_version = ">= 1.5.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
provider "aws" {
region = "us-west-2"
}
# 1. Create KMS Customer Managed Key (CMK) for S3 Encryption
resource "aws_kms_key" "s3_encryption_key" {
description = "KMS CMK for production secure S3 bucket storage"
deletion_window_in_days = 30
enable_key_rotation = true # Enforce automated annual rotation
tags = {
Environment = "production"
Tier = "security"
}
}
resource "aws_kms_alias" "s3_key_alias" {
name = "alias/production-secure-s3-key"
target_key_id = aws_kms_key.s3_encryption_key.key_id
}
# 2. Create the Secure S3 Bucket
resource "aws_s3_bucket" "secure_bucket" {
bucket = "dexnox-secure-storage-prod-99"
force_destroy = false
tags = {
Environment = "production"
Compliance = "pci-dss"
}
}
# 3. Disable Legacy S3 Access Control Lists (ACLs) - Enforce Bucket Owner
resource "aws_s3_bucket_ownership_controls" "ownership_controls" {
bucket = aws_s3_bucket.secure_bucket.id
rule {
object_ownership = "BucketOwnerEnforced"
}
}
# 4. Enable Versioning for Rollback Protection
resource "aws_s3_bucket_versioning" "versioning" {
bucket = aws_s3_bucket.secure_bucket.id
versioning_configuration {
status = "Enabled"
}
}
# 5. Enforce KMS Server-Side Encryption with S3 Bucket Keys
resource "aws_s3_bucket_server_side_encryption_configuration" "sse_config" {
bucket = aws_s3_bucket.secure_bucket.id
rule {
bucket_key_enabled = true # Reduces KMS costs by caching bucket keys
apply_server_side_encryption_by_default {
kms_master_key_id = aws_kms_key.s3_encryption_key.arn
sse_algorithm = "aws:kms"
}
}
}
# 6. Apply S3 Block Public Access overrides
resource "aws_s3_bucket_public_access_block" "public_block" {
bucket = aws_s3_bucket.secure_bucket.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
# 7. Apply the S3 Bucket Policy
resource "aws_s3_bucket_policy" "secure_policy" {
bucket = aws_s3_bucket.secure_bucket.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
# A. Enforce HTTPS - Deny all HTTP connections
{
Sid = "EnforceHTTPSOnly"
Effect = "Deny"
Principal = "*"
Action = "s3:*"
Resource = [
aws_s3_bucket.secure_bucket.arn,
"${aws_s3_bucket.secure_bucket.arn}/*"
]
Condition = {
Bool = {
"aws:SecureTransport" = "false"
}
}
},
# B. Scrape Network Boundary - Only permit access from target VPC Endpoint
{
Sid = "LimitToVpcEndpoint"
Effect = "Deny"
Principal = "*"
Action = "s3:*"
Resource = [
aws_s3_bucket.secure_bucket.arn,
"${aws_s3_bucket.secure_bucket.arn}/*"
]
Condition = {
StringNotEquals = {
"aws:sourceVpce" = "vpce-0123456789abcdef0" # Target VPC Endpoint ID
}
}
}
]
})
}
S3 Encryption Models Comparison
The table below compares the different encryption methods available for securing objects:
| Security Metric | SSE-S3 (Default) | SSE-KMS (AWS Managed Key) | SSE-KMS (Customer Managed Key) | SSE-C (Customer Provided Key) |
|---|---|---|---|---|
| Key Owner | AWS | AWS | Customer (KMS CMK) | Customer |
| Key Rotation control | Managed by S3 | Managed by AWS | Configurable (Annual) | Managed by Customer |
| Access Control (Key) | None | Managed by IAM | Key Policy + IAM | Client Managed |
| Audit Logging (CloudTrail) | None | Logged (Accesses) | Logged (Accesses) | None |
| Request Cost | Free | Varies with call volume | Varies with call volume | Free |
| Bucket Key Cache | N/A | Supported | Supported | N/A |
| Network Boundary integration | Supported | Supported | Supported | Complex |
What Breaks in Production: Failure Modes and Mitigations
Deploying secure S3 storage architectures exposes systems to KMS authorization blocks, VPC route configuration errors, key disablement issues, and cross-account file ownership traps. Below are four common production failure modes and their mitigations.
1. Cross-Account S3 Access Denied errors (KMS Policy Mismatch)
An application running in Account B attempts to read an object from an S3 bucket located in Account A. Although both the S3 bucket policy and Account B’s IAM policy explicitly permit access, the application receives AccessDenied errors.
- Root Cause: The S3 bucket in Account A is encrypted using a KMS Customer Managed Key. S3 cannot decrypt the object data key without calling the KMS CMK. While the S3 bucket permits access, the KMS key policy in Account A does not authorize Account B’s IAM identity to run decrypt operations on the key.
- Mitigation: Update the key policy of the KMS Customer Managed Key in Account A. Add a statement that explicitly grants cross-account decrypt permissions to Account B’s IAM role:
{ "Sid": "AllowCrossAccountDecrypt", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::ACCOUNT_B_ID:role/app-execution-role" }, "Action": [ "kms:Decrypt", "kms:DescribeKey" ], "Resource": "*" }
2. VPC Gateway Endpoint Route Table Disconnection (Read Timeouts)
After launching container nodes in private subnets, applications fail to upload or download files from S3, throwing connection timeout errors.
- Root Cause: Private subnets do not have public IP routes. To access S3, subnets must route traffic through an S3 VPC Gateway Endpoint. If the S3 VPC endpoint is created but not bound to the private subnet’s route tables, the subnet route tables send S3 traffic to the default gateway (which fails for private nodes), causing connections to time out.
- Mitigation: Verify route configurations in Terraform. Bind the S3 VPC Gateway Endpoint explicitly to all route table IDs associated with your private subnets:
resource "aws_vpc_endpoint_route_table_association" "s3_association" { route_table_id = aws_route_table.private_subnet_route_table.id vpc_endpoint_id = aws_vpc_endpoint.s3_gateway_endpoint.id }
3. Application Failures from Disabled KMS Keys
Applications suddenly lose access to all files inside an S3 bucket, throwing KMS.DisabledException errors for all operations.
- Root Cause: An administrator or automated process disables the KMS Customer Managed Key used for bucket encryption, or the key is scheduled for deletion. Because S3 cannot call KMS to decrypt object data keys, all read and write operations are blocked.
- Mitigation: Set up automated CloudWatch alerts to track KMS key state changes. Alert security teams immediately if a key is disabled or scheduled for deletion. Implement multi-factor authentication (MFA) delete guards on KMS configuration APIs to prevent accidental key deletions.
4. S3 Object Ownership Conflicts in Cross-Account Uploads
An IAM identity in Account B uploads files to an S3 bucket owned by Account A. Although the upload succeeds, administrators in Account A cannot read, copy, or delete the uploaded files, receiving AccessDenied errors despite being the bucket owners.
- Root Cause: By default, in legacy configurations, the AWS account that writes an object to S3 owns that object and retains full permission controls over it. This allows Account B to write files that the bucket owner (Account A) cannot access.
- Mitigation: Disable legacy S3 ACLs by enforcing the Bucket Owner Enforced ownership setting in your bucket controls (as shown in the
aws_s3_bucket_ownership_controlsresource above). This configures S3 to automatically assign object ownership to the bucket owner for all writes, resolving cross-account access issues.
Frequently Asked Questions
What is S3 Block Public Access and why should it be enabled?
S3 Block Public Access is a bucket and account-level control that overrides existing ACLs and bucket policies, preventing any accidental exposure of S3 objects to the public internet.
Why does cross-account access fail if the bucket is encrypted with a custom KMS key?
If an S3 bucket uses a KMS Customer Managed Key, the client in the external account must have both S3 permissions on the bucket and KMS decrypt permissions on the key policy of the CMK.
How does enforcing HTTPS on S3 buckets work?
You apply a bucket policy with an explicit Deny effect that triggers if the condition aws:SecureTransport evaluates to false, blocking any unencrypted HTTP requests.
What is the difference between SSE-S3 and SSE-KMS?
SSE-S3 uses keys managed directly by S3, providing zero auditing or access controls. SSE-KMS uses AWS KMS keys, enabling detailed access tracking via CloudTrail and custom permissions scoped to specific IAM roles.
Wrapping Up
Securing cloud storage with Amazon S3 requires eliminating default assumptions. By disabling legacy ACLs, applying account-wide Block Public Access rules, enforcing envelope encryption using KMS Customer Managed Keys, enabling S3 Bucket Keys to control API costs, and scoping bucket policies to block non-HTTPS connections and isolate traffic to specific VPC Gateway Endpoints, you can build a secure data storage perimeter.