What object storage is: bucket, object, key, storage classes and presigned URL

← Back to the section

When you need to store user files, videos, backups or logs, the first thought is often "let's set up NFS" or "let's save it to the server's disk". That works up to a point, and then the problems start: the disk fills up, files are not visible to other servers, backing everything up is hard. Object storage solves exactly these problems, and it is built quite differently from a file system.

How object storage differs from a file system

On an ordinary disk, files live in folders, and folders are nested inside each other. There are operations like rename, seek (read a chunk from the middle), and atomic directory renaming.

Object storage is a different model:

• No folders. There is a single flat namespace where every file has a string key.
• No rename and no seek. An object is read and written whole (or by a byte range).
• Access is only through an HTTP API.

That is exactly why object storage scales to trillions of objects and exabytes of data: there is no file-system overhead and no folder locking.

For orientation, here are three types of storage in one table:

Amazon S3 (Simple Storage Service) is the first commercially successful object storage and the de facto standard. The "S3-compatible API" is understood today by MinIO, Cloudflare R2, Yandex Object Storage, Backblaze B2, Google Cloud Storage and dozens of others. For the rest of the article we talk about the S3 model, which applies to all of them.

Bucket, object and key

Three main concepts:

s3://my-bucket/products/2026/05/cover-3.jpg
   │           │                  │
   │           └─ key ────────────┘
   └─ bucket

Bucket is the root container. Its name is globally unique in AWS S3 (other providers scope uniqueness to the account). Every bucket has a single region.

Object is the unit of storage. It consists of content (bytes), system and user metadata, and a version identifier (if versioning is enabled).

Key is just a string, for example products/2026/05/cover-3.jpg. S3 builds no hierarchy inside. Slashes in a key are a convenience for grouping when listing, not a directory structure. The AWS console shows "folders", but that is only a visualization on top of keys.

Practical consequence: there is no mkdir in S3. A "folder" exists exactly as long as there is at least one object with that prefix.

Consistency guarantees

Until December 2020 S3 was eventually consistent: after uploading a new file, an attempt to read it right away could return an old version or a 404 error. This produced hard-to-catch bugs.

Since December 2020 it offers strong read-after-write consistency: read after a write and you always get the current data, guaranteed. This works for listing too: the object list is also consistent.

MinIO and Yandex Object Storage also provide strong consistency. If you use a less common S3-compatible service, it is worth checking its documentation separately.

Storage classes

Not all data is needed equally fast. Last year's logs are read rarely, while active user avatars are read constantly. S3 offers storage classes with a different balance of storage price and read price:

An important point: all classes except Glacier Flexible and Deep Archive return the object instantly. The difference is only in price, not in speed. Moving an object to a colder class is easy; moving it back requires a fresh copy.

The class is set on upload via the x-amz-storage-class header, or set automatically through a lifecycle policy.

Versioning

By default, a new upload of a file with the same key overwrites the previous one. Enable versioning at the bucket level, and every upload creates a new version while old ones are kept:

# First upload of the file
PUT s3://bucket/report.pdf → versionId=v1

# Uploaded a new version
PUT s3://bucket/report.pdf → versionId=v2  # v1 stays available

# Deleted the file
DELETE s3://bucket/report.pdf → creates a "delete marker", v1 and v2 remain

After deletion, GET s3://bucket/report.pdf returns 404 — S3 reads the delete marker. But GET s3://bucket/report.pdf?versionId=v1 still works.

Versioning is a mandatory attribute of a production store with user files. It protects against accidental deletion, against data-encryption attacks (an attacker cannot physically destroy the versions), and against bugs that overwrote content with wrong data.

The downside: old versions accumulate and take up space. This is solved with a lifecycle policy (see below).

Encryption

S3 encrypts data on its side — server-side encryption (SSE). Since January 2023, SSE-S3 is enabled by default for all new buckets — AWS manages the keys automatically, and there is nothing to configure.

When you need more control:

• SSE-KMS — keys in AWS KMS (yours), with an audit log of every access to the key. Used in finance and healthcare, where an audit trail is required at the encryption-key level.
• SSE-C — you pass your own key with every request. Very specific compliance requirements.
• Client-side encryption — you encrypt before sending. For when S3 must never see unencrypted data at all.

Data transfer over the network is always over HTTPS. This is not configurable; it is how it works by default.

Lifecycle policy — automatic data management

Data grows. Nobody reads month-old logs, but they occupy expensive Standard storage. A lifecycle policy is a set of rules by which S3 automatically moves objects to a cheaper class or deletes them.

A typical configuration for a log store:

<LifecycleConfiguration>
  <Rule>
    <ID>logs-retention</ID>
    <Filter><Prefix>logs/</Prefix></Filter>
    <Status>Enabled</Status>
    <Transition>
      <Days>30</Days>
      <StorageClass>STANDARD_IA</StorageClass>
    </Transition>
    <Transition>
      <Days>90</Days>
      <StorageClass>GLACIER</StorageClass>
    </Transition>
    <Expiration><Days>365</Days></Expiration>
  </Rule>
  <Rule>
    <ID>cleanup-old-versions</ID>
    <Status>Enabled</Status>
    <NoncurrentVersionExpiration>
      <NoncurrentDays>30</NoncurrentDays>
    </NoncurrentVersionExpiration>
  </Rule>
  <Rule>
    <ID>abort-incomplete-uploads</ID>
    <Status>Enabled</Status>
    <AbortIncompleteMultipartUpload>
      <DaysAfterInitiation>7</DaysAfterInitiation>
    </AbortIncompleteMultipartUpload>
  </Rule>
</LifecycleConfiguration>

The three rules here solve three different problems:

1. Logs move to a cold class after 30 days, to the archive after 90, and are deleted after a year.
2. Old versions (when versioning is enabled) are deleted 30 days after being replaced.
3. Incomplete multipart uploads (see below) are deleted after 7 days — otherwise they are not visible in a normal listing, but you still pay for them.

A lifecycle policy is a standard part of setting up any production bucket.

Presigned URL — direct upload without going through the server

A typical task: a user uploads an avatar. The obvious solution is for the client to send the file to your backend, and the backend puts it in S3. The problem: the file passes through your server, loading its bandwidth and CPU.

Better — a presigned URL: the backend generates a temporary signed link with a short lifetime (10–15 minutes), and the client uploads the file directly to S3. The server does not participate in the data transfer.

The server side generates the link:

import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest;
import java.time.Duration;

// S3Presigner is injected as a bean
PresignedPutObjectRequest req = s3Presigner.presignPutObject(b -> b
    .signatureDuration(Duration.ofMinutes(10))
    .putObjectRequest(p -> p
        .bucket("avatars")
        .key("users/" + userId + "/avatar.jpg")
        .contentType("image/jpeg")));
return req.url().toString();

// presignClient — *s3.PresignClient, created once at startup
presignReq, err := presignClient.PresignPutObject(context.Background(),
    &s3.PutObjectInput{
        Bucket:      aws.String("avatars"),
        Key:         aws.String(fmt.Sprintf("users/%s/avatar.jpg", userID)),
        ContentType: aws.String("image/jpeg"),
    },
    s3.WithPresignExpires(10*time.Minute),
)
if err != nil {
    return "", fmt.Errorf("presign: %w", err)
}
return presignReq.URL, nil

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

// s3Client — an S3Client instance, created once
const command = new PutObjectCommand({
    Bucket: "avatars",
    Key: `users/${userId}/avatar.jpg`,
    ContentType: "image/jpeg",
});
const url = await getSignedUrl(s3Client, command, { expiresIn: 600 });
return url;

# s3_client — boto3.client("s3"), created once
url = s3_client.generate_presigned_url(
    "put_object",
    Params={
        "Bucket": "avatars",
        "Key": f"users/{user_id}/avatar.jpg",
        "ContentType": "image/jpeg",
    },
    ExpiresIn=600,
)
return url

The client uploads the file directly using the received link:

fetch(presignedUrl, {
    method: 'PUT',
    body: fileBlob,
    headers: { 'Content-Type': 'image/jpeg' }
});

The same works for downloading (presignGetObject): serving private files with a limited access window.

A few things about presigned URL security:

• Set the minimum lifetime: 10 minutes for uploads, 5 minutes for downloads.
• The signature includes the bucket, key, method (PUT/GET) and content-type. The client cannot change any of them — otherwise the signature check fails.
• You can limit the file size through S3 POST policy conditions.

Multipart upload — files larger than 100 MB

A regular PUT accepts a file up to 5 GB. For large files there is multipart upload: the file is split into parts (from 5 MB to 5 GB each), the parts are uploaded in parallel, and at the end S3 assembles them into a single object.

1. InitiateMultipartUpload  → uploadId
2. UploadPart (part 1, uploadId) → ETag-1
3. UploadPart (part 2, uploadId) → ETag-2
   ... in parallel
4. CompleteMultipartUpload (uploadId, [ETag-1, ETag-2, ...]) → object is ready

Why this is useful:

• Parallelism — parts are uploaded simultaneously, limited only by network throughput.
• Resumption — if part 5 of 10 fails to upload, only it is retried.
• Size — with multipart the maximum is 5 TB instead of 5 GB.

A pitfall: if the upload is interrupted between InitiateMultipartUpload and CompleteMultipartUpload, the parts remain in storage. They are not visible in a normal listing, but you still have to pay for them. That is why a lifecycle policy always adds a rule "delete incomplete uploads after 7 days".

In practice you do not need to manage multipart manually — the SDKs do it automatically:

import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.Upload;
import java.nio.file.Paths;

// S3TransferManager is built on top of a ready S3AsyncClient
S3TransferManager tm = S3TransferManager.builder().s3Client(s3AsyncClient).build();
Upload upload = tm.uploadFile(b -> b
    .source(Paths.get("big-video.mp4"))
    .putObjectRequest(p -> p.bucket("videos").key("user-42/video.mp4")));
upload.completionFuture().join();
// the SDK itself chooses multipart or a regular PUT depending on the size

// uploader — *manager.Uploader, created once on top of s3.Client
file, err := os.Open("big-video.mp4")
if err != nil {
    return fmt.Errorf("open file: %w", err)
}
defer file.Close()

_, err = uploader.Upload(context.Background(), &s3.PutObjectInput{
    Bucket: aws.String("videos"),
    Key:    aws.String("user-42/video.mp4"),
    Body:   file,
})
// manager.Uploader itself chooses multipart when size > PartSize (5 MB by default)

import { Upload } from "@aws-sdk/lib-storage";
import { createReadStream } from "fs";

// s3Client — an S3Client instance
const upload = new Upload({
    client: s3Client,
    params: {
        Bucket: "videos",
        Key: "user-42/video.mp4",
        Body: createReadStream("big-video.mp4"),
    },
});
await upload.done();
// Upload from @aws-sdk/lib-storage automatically uses multipart

from boto3.s3.transfer import TransferConfig

# s3_client — boto3.client("s3")
config = TransferConfig(multipart_threshold=100 * 1024 * 1024)  # 100 MB
s3_client.upload_file(
    Filename="big-video.mp4",
    Bucket="videos",
    Key="user-42/video.mp4",
    Config=config,
)
# boto3 automatically switches to multipart when the threshold is exceeded

In short

• Object storage is a flat namespace of objects with HTTP access. No folders, no rename, no seek. Thanks to that it scales to trillions of objects.
• Three concepts: bucket (a globally unique container), object (bytes + metadata), key (the string address of an object).
• Since 2020, S3 provides strong read-after-write consistency — data is available immediately after a write.
• Storage classes differ in price, not speed (except Glacier Flexible/Deep). Hot data — Standard, rarely used — IA or Glacier.
• Versioning protects against accidental deletion and overwriting. Enable it for any production store with user files.
• Lifecycle policy — automatically moves objects to a cold class and deletes them. Essential for cost control and version management.
• Presigned URL lets you upload and download files directly from S3, bypassing the backend. Keep the lifetime as short as needed.
• Multipart upload — for files larger than 100 MB. SDKs manage it automatically.

What to read next

• Spring + AWS SDK v2 for S3 — concrete client code for Java.
• Operations: backups, replication, costs — the operational side of S3.