> Are all downstream consumers blocked until the DLQ is emptied
No other service should know about the state of another services DLQ
> do all consumers know how to deal with late-arriving events
In my approach in architecting event driven systems, this question is meaningless. For me some of the core tenants (for me) of architecting these systems are
1. Message ordering should not matter
2. Message delays should not matter (no temporal coupling)
3. Message replays should have no side effects
This means that Service B should have no knowledge of what happens in Service A. If something in Service A fails and gets sent to Service A's DLQ, it should have no impact on Service B. This often involves rethinking business requirements and processes to account for workflows that get interrupted - I've yet to find an insurmountable issue.
If however, we are talking about a message broker trying to deliver a message from Service A to Service B, failing and forwarding to Service B's DLQ, then there is a very simple solution. Have your message broker deliver messages to queues. Service B then processes and handles messages from the queue rather than having to immediately handle all incoming messages from the broker. This protects against a lot of failure modes. The only thing(s) that arrive at Service B's DLQ are exceptions thrown internally in Service B
> do all consumers know how to deal with late-arriving events In my approach in architecting event driven systems, this question is meaningless. For me some of the core tenants (for me) of architecting these systems are
1. Message ordering should not matter 2. Message delays should not matter (no temporal coupling) 3. Message replays should have no side effects
This means that Service B should have no knowledge of what happens in Service A. If something in Service A fails and gets sent to Service A's DLQ, it should have no impact on Service B. This often involves rethinking business requirements and processes to account for workflows that get interrupted - I've yet to find an insurmountable issue.
If however, we are talking about a message broker trying to deliver a message from Service A to Service B, failing and forwarding to Service B's DLQ, then there is a very simple solution. Have your message broker deliver messages to queues. Service B then processes and handles messages from the queue rather than having to immediately handle all incoming messages from the broker. This protects against a lot of failure modes. The only thing(s) that arrive at Service B's DLQ are exceptions thrown internally in Service B