An application changes constantly: you added a field to an order, introduced a new status, renamed an entity. And nearly every such change runs into the same question: what happens to the data and the code that don't yet know about this change?
There's a systematic answer — a pair of compatibility concepts and a short list of rules for changing a data format safely. The rules are the same for the database, for a REST request, and for an event in Kafka, though they hurt differently in each channel. Let's build it up from scratch.
Why the old and new versions always live together
It seems the version changes instantly on deploy: it was old, now it's new. Actually, no. A service is updated with a rolling upgrade — nodes are replaced one at a time so there's no downtime. So for several minutes (and with a gradual rollout to a percentage of users, several days) both the old and the new version of the code run in production at the same time. With client apps it's even worse: a user might not update a mobile app for months.
Hence a simple consequence: data that travels between versions must be readable both ways. This has two names:
- Backward compatibility — new code reads data written by old code. Usually easy: whoever writes the new code remembers the old format.
- Forward compatibility — old code reads data written by new code. This is harder: the old code has to calmly swallow what it doesn't understand — and you can't rewrite it, it's already running.
And one more fact that makes the topic serious: data outlives code. Code is replaced whole in minutes, but a five-year-old record still sits in the database in a five-year-old format — nobody is going to rewrite terabytes for one new field. So the database at any moment is a mix of formats written by different versions of the code.
Three data channels and their traps
The database. A write to the database is like "a letter to yourself in the future," and it needs both compatibilities at once: new code reads old rows (backward), and during the rollout old code reads rows written by new (forward). Here hides the sneakiest trap of the whole topic. Old code reads a record with a new field, changes something in it, and saves it back whole — silently wiping out that very new field it doesn't know about. Data is lost, and there's not a single error in the logs. The defense is simple: update only your own fields (UPDATE … SET status = …, not "read the whole object → write the whole object"), and if you do read-modify-write a document, carefully preserve the unfamiliar fields in it.
Synchronous APIs (REST, gRPC). Life is easier here: usually you update servers first, then clients. So requests need only backward compatibility, and responses only forward. In practice that's two rules: "a new request parameter must be optional" and "the client must ignore a new field in the response, not crash on it." When compatibility can't be kept — you turn on API versioning, and the old version has to be kept alive until the last consumer drops off: you can't force other people's clients to update.
Events (Kafka, RabbitMQ). Here compatibility matters most. The sender and receiver of an event are different services from different teams, their versions inevitably diverge, and an event in a topic outlives more than one deploy of its receivers. So here compatibility is guarded not by discipline but by infrastructure: a Schema Registry checks every new schema for compatibility with the previous ones and simply won't let the sender register a breaking change. A separate pitfall: a receiver that forwards an event into another topic must preserve the fields it doesn't understand — otherwise it becomes the same "old code wiping out the new."
The rules of evolution
Across all schema formats (Protobuf, Avro, JSON Schema) the rules boil down to a short list:
- A new field must be optional or have a default value. Make a new field required, and backward compatibility breaks: new code won't be able to read old records where the field simply isn't there.
- Field identifiers are untouchable. In Protobuf each field has a number (tag), and the encoded data references fields by those numbers. Reuse a freed-up number, and old records turn into garbage. A field's name can be changed (data doesn't reference names), but the number — never.
- You may only delete optional fields, and their number is then retired for good — it can't be handed to a new field.
- Change a type carefully. A widening (say, int32 → int64) new code survives, but old code will silently truncate a too-long value.
You don't need to enforce the rules by hand — that's what the schema is for: a schema registry or a contract check in CI catches a breaking change before the rollout.
JSON or a binary schema format
JSON won as the format for integrations — you read it with your eyes, and it works everywhere. But everything above hurts more with it. The schema is implicit (what old code does with a new field depends on the specific library); numbers larger than 2⁵³ lose precision in the JavaScript world (Twitter, because of this, returns a tweet's id as both a string and a number); binary data has to travel through Base64 with a +33% size markup; and field names repeat in every record.
Binary schema formats — Protobuf (field numbers, code generation) and Avro (a writer schema and a reader schema kept separate) — are several times more compact and, crucially, make compatibility checkable: the schema is documentation that can't go stale, and a contract that CI can verify automatically. The practical frame: outward, to other people's clients — JSON and REST (keep compatibility by discipline and versioning); between your own services and in events — a schema format with a registry.
And a separate warning: a language's built-in serialization (java.io.Serializable and kin) is unfit for anything long-lived — it's tied to the language, doesn't let the format evolve properly, and has known security holes when parsing untrusted data.
Where this applies
Every incident of the form "after the rollout, deserialization errors rained down" or "a field disappeared for some users" is a broken compatibility. The check question before any format change is one: "what will the code that hasn't updated yet, and the data already written, do with this?" If both answers are calm ("it'll read it" and "it'll survive"), the change is safe.
Where beginners stumble:
- Adding a required field — and new code crashes on the very first old record where the field is missing. Only optional or with a default.
- Thinking about compatibility only for the API, forgetting the database and events. And it's the old code wiping out new fields on rewrite that breaks data most quietly of all.
- Reusing a freed-up field number in Protobuf — and old records start reading as nonsense.
- Keeping the schema "in their head" in a JSON integration. Compatibility that nobody checks automatically will sooner or later be broken.
What to read next: Schema Registry in Kafka — infrastructural compatibility checking for events; gRPC and protobuf — a schema format in practice; REST API versioning — what to do when compatibility can't be kept.