An application changes constantly: you added a field to an order, introduced a new status, renamed an entity. And nearly every such change runs into the same question: what happens to the data and the code that don't yet know about this change?
There's a systematic answer — a pair of compatibility concepts and a short list of rules for changing a data format safely. The rules are the same for the database, for a REST request, and for an event in Kafka, though they hurt differently in each channel.
The same row passes through two versions of the code. Old code doesn't see a field that isn't in its model — and everything depends on how it writes the row back: saving the whole object wipes the unfamiliar field silently, updating one own field keeps it.
Why the old and new versions always live together
It seems the version changes instantly on rollout: it was old, now it's new. Actually, no. A service is updated with a rolling upgrade — nodes are replaced one at a time so there's no downtime. So for several minutes — and with a gradual rollout to a percentage of users, several days — both versions run in production at once. With client apps it's even worse: a user might not update a mobile app for months.
Hence a simple consequence: data that travels between versions must be readable both ways. This has two names:
- Backward compatibility — new code reads data written by old code. Usually easy: whoever writes the new code remembers the old format.
- Forward compatibility — old code reads data written by new code. This is harder: the old code has to calmly swallow what it doesn't understand — and you can't rewrite it, it's already running.
And one more fact that makes the topic serious: data outlives code. Code is replaced whole in minutes, but a five-year-old record still sits in the database in a five-year-old format — nobody is going to rewrite terabytes for one new field. So the database at any moment is a mix of formats written by different code versions.
Three data channels and their traps
The database. A write to the database is like "a letter to yourself in the future," and it needs both compatibilities at once: new code reads old rows (backward), and during the rollout old code reads rows written by new (forward). Here hides the sneakiest trap of the whole topic. Old code reads a record with a new field, changes something in it, and saves it back whole — silently wiping out that very new field it doesn't know about. Data is lost, and there's not a single error in the logs. The defense is simple: update only your own fields (UPDATE … SET status = …, not "read the whole object → write the whole object"), and if you do read-modify-write a document, carefully preserve the unfamiliar fields in it.
The difference shows in a small program: a row here is a bag of fields, version 1 knows only two.
live example
import java.util.LinkedHashMap;
import java.util.Map;
public class SchemaEvolutionDemo {
record OrderV1(String id, String status) {}
public static void main(String[] args) {
Map<String, String> row = new LinkedHashMap<>();
row.put("id", "1001");
row.put("status", "NEW");
row.put("promo_code", "SPRING25");
System.out.println("written by v2: " + row);
System.out.println("whole-object save: " + rewriteWhole(row));
System.out.println("own field only: " + updateStatusOnly(row));
}
static Map<String, String> rewriteWhole(Map<String, String> row) {
OrderV1 order = new OrderV1(row.get("id"), row.get("status"));
Map<String, String> back = new LinkedHashMap<>();
back.put("id", order.id());
back.put("status", "PAID");
return back;
}
static Map<String, String> updateStatusOnly(Map<String, String> row) {
Map<String, String> back = new LinkedHashMap<>(row);
back.put("status", "PAID");
return back;
}
}
Run
Running examples is part of paid access. There the same code runs inside the article: editor, run and check next to the paragraph. Three free days →
Both versions "work correctly" — one of them just drops the field it doesn't know.
Synchronous APIs (REST, gRPC). Life is easier here: usually you update servers first, then clients. So requests need only backward compatibility, and responses only forward. In practice that's two rules: "a new request parameter must be optional" and "the client must ignore a new field in the response, not crash on it." When compatibility can't be kept — you turn on API versioning, and the old version has to be kept alive until the last consumer drops off: you can't force other people's clients to update.
Events (Kafka, RabbitMQ). Here compatibility matters most. The sender and receiver of an event are different services from different teams, their versions inevitably diverge, and an event in a topic outlives more than one rollout of its receivers. So here compatibility is guarded not by discipline but by infrastructure: a Schema Registry checks every new schema for compatibility with the previous ones and simply won't let the sender register a breaking change. A separate pitfall: a receiver that forwards an event into another topic must preserve the fields it doesn't understand — otherwise it becomes the same "old code wiping out the new."
The rules of evolution
Across all schema formats (Protobuf, Avro, JSON Schema) the rules boil down to a short list:
- A new field must be optional or have a default value. Make a new field required, and backward compatibility breaks: new code won't be able to read old records where the field simply isn't there.
- Field identifiers are untouchable. In Protobuf each field has a number (tag), and the encoded data references fields by those numbers. Reuse a freed-up number, and old records turn into garbage. In Protobuf a field's name can be changed (data doesn't reference names), but the number — never. In Avro it's the other way round: fields are matched by name, and a rename without
aliasesbreaks reading old data. - You may only delete optional fields, and their number is then retired for good — it can't be handed to a new field.
- Change a type carefully. A widening (say, int32 → int64) new code survives, but old code will silently truncate a too-long value.
No need to enforce the rules by hand — that's what the schema is for: a registry or a contract check in CI catches a breaking change before the rollout.
JSON or a binary schema format
JSON won as the format for integrations — you read it with your eyes, and it works everywhere. But everything above hurts more with it. The schema is implicit (what old code does with a new field depends on the specific library); numbers larger than 2⁵³ lose precision in the JavaScript world (Twitter, because of this, returned a tweet's id as both a string and a number); binary data has to travel through Base64 with a +33% size markup; and field names repeat in every record.
Binary schema formats — Protobuf (field numbers, code generation) and Avro (a writer schema and a reader schema kept separate) — are several times more compact and, crucially, make compatibility checkable: the schema is documentation that can't go stale, and a contract that CI can verify automatically. The practical frame: outward, to other people's clients — JSON and REST (keep compatibility by discipline and versioning); between your own services and in events — a schema format with a registry.
And a separate warning: a language's built-in serialization (java.io.Serializable and kin) is unfit for anything long-lived — it's tied to the language, doesn't let the format evolve properly, and has known security holes when parsing untrusted data.
Where this applies
Every incident of the form "after the rollout, deserialization errors rained down" or "a field disappeared for some users" is broken compatibility. The check question before a format change is one: "what will the code that hasn't updated yet, and the data already written, do with this?" If both answers are calm ("it'll read it" and "it'll survive"), the change is safe.
In short
- A rollout isn't instant: nodes are replaced one at a time, and for a while both versions of the code run in production at once.
- Backward compatibility is new code reading old data, forward is old code reading new. The database and events need both.
- Data outlives code: the database is always a mix of formats, and nobody rewrites terabytes for one field.
- The quietest data loss comes from old code that saved a record back whole without the unfamiliar field. Update only your own fields.
- A new field must be optional or have a default; a Protobuf field number is never reused, and an Avro field is never renamed without
aliases. - Outward — JSON and versioning; between your own services — a schema and a registry that rejects a breaking change before the rollout.
What to read next
- Schema Registry in Kafka — infrastructural compatibility checking for events.
- gRPC and protobuf — a schema format in practice.
- REST API versioning — what to do when compatibility can't be kept.