An application changes constantly: a new field on the order, a new status, a renamed entity. Every such change runs into the same question: what happens to the data and code that do not know about the change yet? That question has a systematic answer — the notions of compatibility and a short set of schema evolution rules. They are the same for a database row, a REST request, and a Kafka event, though each channel hurts in its own way.
Why versions always live in pairs
A service does not update instantaneously. A rolling upgrade replaces nodes one by one: for a few minutes — or a few days, with a gradual rollout to a share of users — production runs the old and the new code versions at the same time. Client applications are worse: a user may not update a mobile app for months.
So any data in motion must be readable in both directions:
- Backward compatibility — newer code reads data written by older code. Usually easy: the author of the new code knows the old format.
- Forward compatibility — older code reads data written by newer code. Harder: the old code must calmly ignore what it does not understand — and you can no longer rewrite it.
And the second fact that makes this serious: data outlives code. Code is replaced wholesale in minutes, while five-year-old records still sit in the database in a five-year-old format — a full rewrite of terabytes for the sake of a new field will never happen. At any moment the database holds a mix of formats written by different code versions.
Three data channels and their traps
The database. A database write is "a message to your future self," and it needs both compatibilities: new code reads old rows (backward), and during a rollout old code reads rows written by new code (forward). Here lives the chapter's sneakiest trap: old code reads a record with a new field, modifies it, and saves it back — silently wiping the field it knows nothing about. Data is lost with not a single error in the logs. The protection is application-level care: update only your own fields (UPDATE … SET status = …, not "read the whole object — write the whole object"), and preserve unknown fields on read-modify-write of documents.
Synchronous APIs (REST, gRPC). Here a simplifying assumption applies: servers are usually updated before clients, so requests need only backward compatibility, and responses need forward compatibility. In practice that means "a new request parameter must be optional" and "clients must ignore new fields in responses." When compatibility cannot be preserved, API versioning kicks in — and the old version has to stay alive until the last consumer leaves: you cannot force other people's clients to update.
Events (Kafka, RabbitMQ). The producer and the consumer are different services owned by different teams, versions inevitably drift, and an event in a topic outlives more than one consumer deploy. That is why compatibility here is guarded by infrastructure — a Schema Registry checks every new schema against the previous ones and simply refuses to register a breaking change. A separate rake: a consumer that republishes an event into another topic must preserve fields it does not understand — otherwise it is that same "old code wiping the new."
The evolution rules
Across schema-based formats (Protobuf, Avro, JSON Schema) the rules boil down to a short list:
- A new field must be optional or carry a default value. A required new field breaks backward compatibility: new code cannot read old records where the field is absent.
- Field identifiers are untouchable. In Protobuf that is the tag number: encoded data references fields by number, and reusing a number turns old records into garbage. A field's name may change (data never references names); the number — never.
- Only optional fields may be removed, and their identifier retires forever.
- Type changes — with care: widening (int32 → int64) survives in new code, while old code silently truncates the long value.
You do not verify these rules by hand — that is what the schema is for: a schema registry or contract checks in CI catch a breaking change before the rollout.
JSON or a schema-based binary format
JSON won as the integration format — human-readable, works everywhere. But everything above hurts more with it: the schema is implicit (what old code does with a new field depends on each library), numbers beyond 2⁵³ lose precision in the JavaScript world (Twitter returns a tweet id both as a number and as a string), binary data travels through Base64 at +33% size, and field names repeat in every record.
Schema-based binary formats — Protobuf (field tags, code generation) and Avro (writer's schema + reader's schema, friendly to dynamically generated schemas) — are several times more compact and, more importantly, make compatibility checkable: the schema is documentation that cannot go stale and a contract CI can verify. The practical frame: externally — JSON/REST (compatibility by discipline and versioning); between your own services and in events — a schema-based format with a registry. And separately: a language's built-in serialization (java.io.Serializable and kin) is not an option for anything long-lived: language lock-in, no evolution story, and well-known security holes when deserializing untrusted data.
Where this applies
Every incident of the "deserialization errors after the rollout" or "some users lost a field" kind is compatibility not held. The check question before any format change: "what will code that has not updated yet do with this, and what happens to data already written?" If the answers are "reads it" and "survives it" — the change is safe.
Where beginners stumble:
- Adding a required field — and the new code crashes on the first old record. Optional / default only.
- Thinking about compatibility only for the API, forgetting the database and events: old code wiping new fields on rewrite corrupts data most quietly of all.
- Reusing a freed Protobuf field number — old records start parsing as nonsense.
- Keeping the schema "in your head" with a JSON integration: compatibility that nobody checks automatically will sooner or later be broken.
What to read next: Schema Registry in Kafka — infrastructure-enforced event compatibility; gRPC and protobuf — a schema-based format in practice; REST API versioning — when compatibility cannot be preserved. The primary source is Martin Kleppmann, "Designing Data-Intensive Applications," chapter 4.