消息语义

  • At-least once
  • At-most once，
  • exactly-once

异常链路

  • broker异常
  • producer -> broker 链路异常
  • 客户端异常

解决方案

幂等性

Producer通过幂等性在服务端对消息进行去重。和TCP类似，但是TCP的Seq是内存状态，而Kafka的Seq会持久化。 SeqId: ProducerId + PartitionId + Msg SeqId。Producer初始化的时候Broker会分配Pid，并维护Pid当前的SeqId，低于或高于SeqId的会被丢弃。

Transaction

  • 语义是原子写入。即一批消息要么写入成功，对Consumer可见。要么写入失败，对Consumer完全不可见。
  • 可以跨Partition

  • consumer有两个level:

    • read-commited: 只读取commited的消息
    • read-uncommited: 感知所有消息

exactly-once processing

exactly-once processing 语义需要区分确定性的流处理和不可确定的流处理。

  • 流处理应用同时处理多个Kafka topics，发生故障后重启，一致性的应用要求多个流处理的顺序和故障前一致。
  • 写入多个Kafka topic partition时如果发生部分失败，如果没有原子写，也可能导致不一致。
  • 多个流Join，如果某个流失败，需要重建状态。
  • 如果一个流处理应用依赖于外界存储，就是天然的不可确定的。结果取决于外界存储当时的状态。

  • 对于一个确定性的流， exactly-once 的语义是：对一个「读-处理-写」模式的应用，输出永远和流处理器只看到流仅仅一次一样，就像从来没有故障发生过。

    The correct way to think of exactly-once stream processing guarantees for deterministic operations is to ensure that the output of a read-process-write operation would be the same as it would if the stream processor saw each message exactly one time–as it would in a case where no failure occurred.

  • 对于一个非确定性的流，一个读-处理-写 模式的应用，输出永远是合法操作的子集。即与非确定性输出合法值的组合。

    The correct way to think of exactly-once guarantees for non-deterministic operations is to ensure that the output of a read-process-write stream processing operation belongs to the subset of legal outputs that would be produced by the combinations of legal values of the non-deterministic input.

性能：

For 1 KB messages and transactions lasting 100 ms, the producer throughput declines only by 3%, compared to the throughput of a producer configured for at least once, in-order delivery (acks=all, max.in.flight.requests.per.connection=1), and by 20% compared to the throughput of a producer configured for most once delivery with no ordering guarantees (acks=1, max.in.flight.requests.per.connection=5), which is the current default