Storage: S3 and Friends

S3 stores files ("objects") in buckets, with practically unlimited space and very high durability. It is the standard home for user uploads, backups, static sites and data exports.

Basics with the CLI

aws s3 mb s3://pandadev-uploads-eu --region eu-central-1
aws s3 cp report.pdf s3://pandadev-uploads-eu/reports/
aws s3 ls s3://pandadev-uploads-eu/reports/
aws s3 sync ./backups s3://pandadev-backups-eu/db/ --storage-class STANDARD_IA

Bucket names are global across all AWS customers, so include something unique.

Keep buckets private

Leave Block Public Access turned on (the default) for every bucket. Leaked customer files from "public" buckets are among the most common cloud breaches. To share a file, use a presigned URL, which grants temporary access to one object:

import boto3

s3 = boto3.client("s3")
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "pandadev-uploads-eu", "Key": "projects/42/source.zip"},
    ExpiresIn=600,  # 10 minutes
)

This is exactly the pattern for paid downloads: the app checks the user may download, then hands out a link that expires.

Direct uploads from the browser work the same way with generate_presigned_post, so large files never pass through your web server.

Storage classes and lifecycle rules

A lifecycle rule for backups: move to Standard-IA after 30 days, Glacier after 90, delete after 365. Set it once and storage costs stay under control.

Protecting data

• Encryption at rest is on by default (SSE-S3). Use SSE-KMS when you need key-level audit and control.
• Versioning keeps old versions when objects are overwritten or deleted, protecting against mistakes and ransomware.
• Object Lock makes backups immutable for a set period; even an admin cannot delete them.
• Replication copies objects to a bucket in another region for disaster recovery.

S3-compatible alternatives

Cloudflare R2 (no egress fees), Backblaze B2 and others speak the same API, so boto3 and rclone work with them by changing the endpoint URL. Compare egress pricing: serving lots of downloads from S3 can cost more than storing them.

Other AWS storage

EBS is a disk attached to one EC2 instance; take snapshots for backups. EFS is a shared network filesystem many instances can mount.

Check yourself

• How do you give a customer temporary access to a private file?
• What does a lifecycle rule do for backup costs?
• Which feature protects backups from deletion, even by admins?