What are the biggest challenges in Event-Driven Architecture (EDA)? Mine is change.

As with everything in life, change is always something to which humans naturally react negatively. However, change is common, so we need to be able to react to it.

Any business needs to evolve, so introducing new event types, modifying existing ones, or scaling your system will happen. But here's the catch: If you don't plan for change (growth), it will break your consumers, causing chaos.

So, how do you design events to evolve gracefully?

I'll explore three best practices here to keep your EDA future-proof. But before that, what's a breaking change?

What is a breaking change?

A breaking change in EDA is a modification to an event schema that prevents existing consumers from correctly processing the data. In event-driven systems, these changes often lead to errors, crashes, or data loss.

Types of Breaking Changes

There are several ways you can unintentionally break an event schema:

1. Removing a Required Field

Breaking Change Example:

Old schema

1{
2"type": "record",
3"name": "UserCreated",
4"fields": [
5{"name": "userId", "type": "string"},
6{"name": "email", "type": "string"}
7]
8}

New schema (breaking change)

1{
2"type": "record",
3"name": "UserCreated",
4"fields": [
5{"name": "email", "type": "string"}
6]
7}

Problem: Older consumers expect userId, but it's missing, leading to deserialization failures.

2. Changing a Field Type

Breaking Change Example:

Old schema

1{"name": "age", "type": "double"}

New schema (breaking change)

1{"name": "age", "type": "string"}

Problem: If a consumer expects age as a double but receives a string, it might crash.

3. Changing a Field's Name

Breaking Change Example:

Old schema

1{"name": "userId", "type": "string"}

New schema (breaking change)

1{"name": "customerId", "type": "string"}

Problem: Older consumers still expect userId, but it's now called customerId, causing failures.

4. Making an Optional Field Required

Breaking Change Example:

Old schema (optional field)

1{"name": "age", "type": ["null", "double"], "default": null }

New schema (breaking change)

1{"name": "age", "type": "double"}

Problem: Older producers never sent age, but now it's required, causing event rejections.

5. Removing an Event Type

Breaking Change Example:

Before:

1{"eventType": "UserCreated"}

After:

// Event type deleted!

Problem: Consumers listening for UserCreated won't receive it anymore, breaking downstream services.

Best Practices

Now that we are both on the same page, let's explore some of the strategies that can be used.

Versioning Your Events – Don’t Break Your Consumers!

Events are contracts between producers and consumers, and breaking a contract can cause failures across your system. It's the same analogy as legal contracts: If you break them, you suffer the legal consequences. If you need to change the legal contract, you must amend it. That's why versioning is essential.

How Event Versioning Works

Suppose you have an event: UserCreated_v1 – Contains { userId, email }.

Your business wants to add a new field to capture user age. How can you do this without breaking the contract?

You could start by creating a new version of the event contract: UserCreated_v2 with { userId, email, age }. And keep v1 active for older consumers.

Or, if the new field is not mandatory, you can keep the same contract version and define a default value so it's a new non-required field, without breaking anything.

With these hints, consumers can gradually upgrade to v2 without breaking the "agreed " contract v1.

As a golden rule, never suddenly remove or change required fields. Always extend, don't replace!

It also works for type field changes. However, in this case, creating a new version is the only option for preventing breakage.

Schema Registries – Your Safety Net for Event Evolution

Consumers should be notified when a contract exists and changes. Even if it doesn't change, there should be a way to show the catalogue of events available to consumers. And here is where the Schema Registry steps in.

A Schema Registry ensures that when events evolve, nothing breaks. Think of it as a safety net that:

• Enforces compatibility rules: Stops producers from publishing breaking changes.
• Tracks event versions: Assigns each schema a version to manage changes over time.
• Protects consumers: Rejects invalid events before they cause failures.

Real-World Example: Confluent Schema Registry + Avro

Let me describe, in short, the role of Avro and Confluent Schema Registry.

• Avro is designed with schema evolution in mind. It’s an event schema intended to allow managing fields with clear rules for what’s considered a compatible change (you can add a field with a default value, but removing a required field would be incompatible).
• Confluent’s Schema Registry checks every new schema, assigns a version number, and checks against previous versions to enforce backward compatibility.

Now, with both solutions in place, when a producer tries to send an event missing a required field, the schema registry rejects it immediately, preventing failures downstream.

Confluent Schema Registry and Avro combine to version every schema, enforce compatibility rules, and let registries handle governance. This governance avoids nasty surprises in production, such as when one team's update crashes another team's service due to a missing field.

Message Adapters – Bridging Old and New Events

Sometimes, you can’t update every consumer instantly. That’s where message adapters come in.

How Adapters Help

Let’s say you must change an event format. Instead of breaking everything, you:

1. Run both versions in parallel for a transition period.
2. Use an adapter service that translates old events to the new format.
3. Gradually deprecate the older version once all consumers have migrated.

Think of it like a translator. Old consumers still receive the UserCreated_v1, the new consumers will start processing the new version, UserCreated_v2, and the adapter converts v1 events into v2 format when needed.

This soft transition ensures zero downtime while teams migrate at their own pace.

In Conclusion

Designing an EDA that evolves smoothly comes down to good engineering practices such as versioning your events (carefully), using schema registries and leveraging message adapters.

These good practices will make your EDA flexible, resilient, and future-proof. And so, your engineering teams can build without fear of breaking things.

We at Qala are building an Event Gateway called Q-Flow—a cutting-edge solution designed to meet the challenges of real-time scalability head-on. If you're interested in learning more, check out Q-Flow here or feel free to sign up for free. Let’s take your system to the next level together.