What object storage is: bucket, object, key, storage classes and presigned URL

← Back to the section

When you need to store user files, videos, backups or logs, the first thought is often "let's set up NFS" or "let's save it to the server's disk". That works up to a point, and then the problems start: the disk fills up, files are not visible to other servers, backing everything up is hard. Object storage solves exactly these problems, and it is built quite differently from a file system.

How object storage differs from a file system

On an ordinary disk, files live in folders, and folders are nested inside each other. There are operations like rename, seek (read a chunk from the middle), and atomic directory renaming.

Object storage is a different model:

• No folders. There is a single flat namespace where every file has a string key.
• No rename and no seek. An object is read and written whole (or by a byte range).
• Access is only through an HTTP API.

That is exactly why object storage scales to trillions of objects and exabytes of data: there is no file-system overhead and no folder locking.

Three types of storage in one table:

Amazon S3 (Simple Storage Service) is the first commercially successful object storage and the de facto standard. The "S3-compatible API" is understood today by MinIO, Cloudflare R2, Yandex Object Storage, Backblaze B2, Google Cloud Storage and dozens of others. For the rest of the article we talk about the S3 model, which applies to all of them.

Bucket, object and key

Three main concepts:

Bucket is the root container. Its name is globally unique in AWS S3 (other providers scope uniqueness to the account). Every bucket has a single region.

Object is the unit of storage. It consists of content (bytes), system and user metadata, and a version identifier (if versioning is enabled).

Key is just a string, for example products/2026/05/cover-3.jpg. S3 builds no hierarchy inside. Slashes in a key are a convenience for grouping when listing, not a directory structure. The AWS console shows "folders", but that is only a visualization on top of keys.

Practical consequence: there is no mkdir in S3. A "folder" exists exactly as long as there is at least one object with that prefix.

The whole "hierarchy" boils down to comparing the start of a string:

live example

import java.util.Map;
import java.util.TreeMap;

public class KeySpaceDemo {
    public static void main(String[] args) {
        TreeMap<String, Integer> bucket = new TreeMap<>();
        bucket.put("avatars/u-7/ava.jpg", 18_400);
        bucket.put("products/2026/05/cover-3.jpg", 240_100);
        bucket.put("products/2026/05/cover-4.jpg", 198_700);
        bucket.put("products/2026/06/cover-9.jpg", 205_300);
        bucket.put("logs/2026-05-14.log", 9_120);

        String prefix = "products/2026/05/";
        System.out.println("listing with prefix=" + prefix);
        for (Map.Entry<String, Integer> entry : bucket.subMap(prefix, prefix + "\uffff").entrySet()) {
            System.out.println("  " + entry.getKey() + "  " + entry.getValue() + " bytes");
        }

        bucket.remove("products/2026/05/cover-3.jpg");
        bucket.remove("products/2026/05/cover-4.jpg");
        System.out.println("prefix products/2026/05/ still exists after both objects are gone: "
                + !bucket.subMap(prefix, prefix + "\uffff").isEmpty());
    }
}

Consistency guarantees

Until December 2020 S3 was eventually consistent: after uploading a new file, an attempt to read it right away could return an old version or a 404 error. This produced hard-to-catch bugs.

Since December 2020 it offers strong read-after-write consistency: read after a write and you always get the current data, guaranteed. This works for listing too: the object list is also consistent.

MinIO and Yandex Object Storage also provide strong consistency. If you use a less common S3-compatible service, it is worth checking its documentation separately.

Storage classes

Not all data is needed equally fast. Last year's logs are read rarely, while active user avatars are read constantly. S3 offers storage classes with a different balance of storage price and read price:

An important point: all classes except Glacier Flexible and Deep Archive return the object instantly. The difference is only in price, not in speed. Moving an object to a colder class is easy; moving it back requires a fresh copy.

The class is set on upload via the x-amz-storage-class header, or set automatically through a lifecycle policy.

Versioning

By default, a new upload of a file with the same key overwrites the previous one. Enable versioning at the bucket level, and every upload creates a new version while old ones are kept:

# First upload of the file
PUT s3://bucket/report.pdf → versionId=v1

# Uploaded a new version
PUT s3://bucket/report.pdf → versionId=v2  # v1 stays available

# Deleted the file
DELETE s3://bucket/report.pdf → creates a "delete marker", v1 and v2 remain

After deletion, GET s3://bucket/report.pdf returns 404 — S3 reads the delete marker. But GET s3://bucket/report.pdf?versionId=v1 still works.

Versioning is a mandatory attribute of a production store with user files. It protects against accidental deletion, against data-encryption attacks (an attacker cannot physically destroy the versions), and against bugs that overwrote content with wrong data.

The downside: old versions accumulate — a lifecycle policy (see below) clears them out.

Encryption

S3 encrypts data on its side — server-side encryption (SSE). Since January 2023, SSE-S3 applies by default to every new object in every bucket, not only in freshly created ones — AWS manages the keys automatically, and there is nothing to configure.

When you need more control:

• SSE-KMS — keys in AWS KMS (yours), with an audit log of every access to the key. Used in finance and healthcare, where an audit trail is required at the encryption-key level.
• SSE-C — you pass your own key with every request. Very specific compliance requirements.
• Client-side encryption — you encrypt before sending. For when S3 must never see unencrypted data at all.

The storage also answers over plain HTTP, so encryption in transit belongs in a bucket policy rather than in defaults: deny every request that did not arrive over a secure connection. One library with an old address is enough to send files in the clear.

Lifecycle policy — automatic data management

Data grows. Nobody reads month-old logs, but they occupy expensive Standard storage. A lifecycle policy is a set of rules by which S3 automatically moves objects to a cheaper class or deletes them.

A typical configuration for a log store:

<LifecycleConfiguration>
  <Rule>
    <ID>logs-retention</ID>
    <Filter><Prefix>logs/</Prefix></Filter>
    <Status>Enabled</Status>
    <Transition>
      <Days>30</Days>
      <StorageClass>STANDARD_IA</StorageClass>
    </Transition>
    <Transition>
      <Days>90</Days>
      <StorageClass>GLACIER</StorageClass>
    </Transition>
    <Expiration><Days>365</Days></Expiration>
  </Rule>
  <Rule>
    <ID>cleanup-old-versions</ID>
    <Status>Enabled</Status>
    <NoncurrentVersionExpiration>
      <NoncurrentDays>30</NoncurrentDays>
    </NoncurrentVersionExpiration>
  </Rule>
  <Rule>
    <ID>abort-incomplete-uploads</ID>
    <Status>Enabled</Status>
    <AbortIncompleteMultipartUpload>
      <DaysAfterInitiation>7</DaysAfterInitiation>
    </AbortIncompleteMultipartUpload>
  </Rule>
</LifecycleConfiguration>

The three rules here solve three different problems:

1. Logs move to a cold class after 30 days, to the archive after 90, and are deleted after a year.
2. Old versions (when versioning is enabled) are deleted 30 days after being replaced.
3. Incomplete multipart uploads (see below) are deleted after 7 days — otherwise they are not visible in a normal listing, but you still pay for them.

Presigned URL — direct upload without going through the server

A typical task: a user uploads an avatar. The obvious solution is for the client to send the file to your backend, and the backend puts it in S3. The problem: the file passes through your server, loading its bandwidth and CPU.

Better — a presigned URL: the backend generates a temporary signed link with a short lifetime (10–15 minutes), and the client uploads the file directly to S3. The server does not participate in the data transfer.

The server side generates the link:

import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest;
import java.time.Duration;

// S3Presigner is injected as a bean
PresignedPutObjectRequest req = s3Presigner.presignPutObject(b -> b
    .signatureDuration(Duration.ofMinutes(10))
    .putObjectRequest(p -> p
        .bucket("avatars")
        .key("users/" + userId + "/avatar.jpg")
        .contentType("image/jpeg")));
return req.url().toString();

// presignClient — *s3.PresignClient, created once at startup
presignReq, err := presignClient.PresignPutObject(context.Background(),
    &s3.PutObjectInput{
        Bucket:      aws.String("avatars"),
        Key:         aws.String(fmt.Sprintf("users/%s/avatar.jpg", userID)),
        ContentType: aws.String("image/jpeg"),
    },
    s3.WithPresignExpires(10*time.Minute),
)
if err != nil {
    return "", fmt.Errorf("presign: %w", err)
}
return presignReq.URL, nil

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

// s3Client — an S3Client instance, created once
const command = new PutObjectCommand({
    Bucket: "avatars",
    Key: `users/${userId}/avatar.jpg`,
    ContentType: "image/jpeg",
});
const url = await getSignedUrl(s3Client, command, { expiresIn: 600 });
return url;

# s3_client — boto3.client("s3"), created once
url = s3_client.generate_presigned_url(
    "put_object",
    Params={
        "Bucket": "avatars",
        "Key": f"users/{user_id}/avatar.jpg",
        "ContentType": "image/jpeg",
    },
    ExpiresIn=600,
)
return url

The client uploads the file directly using the received link:

fetch(presignedUrl, {
    method: 'PUT',
    body: fileBlob,
    headers: { 'Content-Type': 'image/jpeg' }
});

The same works for downloading (presignGetObject): serving private files with a limited access window.

A few things about presigned URL security:

• Set the minimum lifetime: 10 minutes for uploads, 5 minutes for downloads.
• The signature includes the bucket, key, method (PUT/GET) and content-type. The client cannot change any of them — otherwise the signature check fails.
• You can limit the file size through S3 POST policy conditions.

Multipart upload — files larger than 100 MB

A regular PUT accepts a file up to 5 GB. For large files there is multipart upload: the file is split into parts (from 5 MB to 5 GB each), the parts are uploaded in parallel, and at the end S3 assembles them into a single object.

Why this is useful:

• Parallelism — parts are uploaded simultaneously, limited only by network throughput.
• Resumption — if part 5 of 10 fails to upload, only it is retried.
• Size — with multipart the maximum is 5 TB instead of 5 GB.

A pitfall: if the upload is interrupted between InitiateMultipartUpload and CompleteMultipartUpload, the parts remain in storage. They are not visible in a normal listing, but you still have to pay for them. That is why a lifecycle policy always adds a rule "delete incomplete uploads after 7 days".

How the parts become one object is visible in plain Java: every part gets its own checksum, and the ETag of the finished object is the checksum of the concatenated part checksums plus the part count after a dash. That is why a multipart ETag never matches the MD5 of the source file:

live example

import java.security.MessageDigest;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.List;
import java.util.Random;

public class MultipartDemo {
    static final int PART_SIZE = 5 * 1024 * 1024;

    public static void main(String[] args) throws Exception {
        byte[] file = new byte[12 * 1024 * 1024];
        new Random(42).nextBytes(file);

        List<byte[]> partDigests = new ArrayList<>();
        for (int offset = 0; offset < file.length; offset += PART_SIZE) {
            int length = Math.min(PART_SIZE, file.length - offset);
            MessageDigest md5 = MessageDigest.getInstance("MD5");
            md5.update(file, offset, length);
            byte[] digest = md5.digest();
            partDigests.add(digest);
            System.out.printf("part %d: %d bytes, ETag %s%n",
                    partDigests.size(), length, HexFormat.of().formatHex(digest));
        }

        MessageDigest combined = MessageDigest.getInstance("MD5");
        for (byte[] digest : partDigests) {
            combined.update(digest);
        }
        System.out.println("ETag of the assembled object: "
                + HexFormat.of().formatHex(combined.digest()) + "-" + partDigests.size());
    }
}

In practice you do not need to manage multipart manually — the SDKs do it automatically:

import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.Upload;
import java.nio.file.Paths;

// S3TransferManager is built on top of a ready S3AsyncClient
S3TransferManager tm = S3TransferManager.builder().s3Client(s3AsyncClient).build();
Upload upload = tm.uploadFile(b -> b
    .source(Paths.get("big-video.mp4"))
    .putObjectRequest(p -> p.bucket("videos").key("user-42/video.mp4")));
upload.completionFuture().join();
// the SDK itself chooses multipart or a regular PUT depending on the size

// uploader — *manager.Uploader, created once on top of s3.Client
file, err := os.Open("big-video.mp4")
if err != nil {
    return fmt.Errorf("open file: %w", err)
}
defer file.Close()

_, err = uploader.Upload(context.Background(), &s3.PutObjectInput{
    Bucket: aws.String("videos"),
    Key:    aws.String("user-42/video.mp4"),
    Body:   file,
})
// manager.Uploader itself chooses multipart when size > PartSize (5 MB by default)

import { Upload } from "@aws-sdk/lib-storage";
import { createReadStream } from "fs";

// s3Client — an S3Client instance
const upload = new Upload({
    client: s3Client,
    params: {
        Bucket: "videos",
        Key: "user-42/video.mp4",
        Body: createReadStream("big-video.mp4"),
    },
});
await upload.done();
// Upload from @aws-sdk/lib-storage automatically uses multipart

from boto3.s3.transfer import TransferConfig

# s3_client — boto3.client("s3")
config = TransferConfig(multipart_threshold=100 * 1024 * 1024)  # 100 MB
s3_client.upload_file(
    Filename="big-video.mp4",
    Bucket="videos",
    Key="user-42/video.mp4",
    Config=config,
)
# boto3 automatically switches to multipart when the threshold is exceeded

In short

• Object storage is a flat key space reached over HTTP: no folders, no rename, no seek. A "folder" is a shared prefix, and it disappears together with the last object under it.
• Three concepts: bucket (a globally unique container tied to one region), object (bytes plus metadata), key (the string address).
• Since December 2020 S3 gives strong read-after-write consistency: a read right after a write, and the listing too, return current data.
• Storage classes differ in price, not speed — everything except Glacier Flexible and Deep Archive returns instantly, and moving back to a hot class is a fresh copy.
• Versioning saves you from accidental deletes and overwrites but piles up versions, so it travels with a lifecycle policy — which also clears abandoned multipart uploads.
• Presigned URL routes the file past your server; multipart is for files over 100 MB and lifts the ceiling from 5 GB to 5 TB.

What to read next

• Spring + AWS SDK v2 for S3 — concrete client code for Java.
• S3 in production: backups, replication, cost — the operational side of the storage.
• IAM in AWS — who gets access to a bucket and under which rule.
• Cost optimization in AWS — where the storage bill comes from and how to keep it down.