なぜカフカノードのデータディレクトリサイズが非常に大きいですか

Question

私たちのデベロッパークラスでは1つのカフカブローカーがダウンしています。

私は、ダウンノードのkafkaデータディレクトリが他のノードより非常に大きく、大きくなっていることがわかりました。ノードダウン

• ：

[root@5 ~]# cd /var/local/kafka/ [root@5 kafka]# du --max-depth=1 -ch|sort -hr 100G total 100G ./data 100G [root@5 data]# ll total 24 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-0 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-1 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-10 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-11 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-12 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-13 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-14 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-15 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-16 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-17 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-18 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-19 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-2 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-20 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-21 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-22 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-23 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-24 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-25 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-26 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-27 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-28 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-29 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-3 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-30 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-31 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-32 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-33 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-34 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-35 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-36 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-37 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-38 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-39 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-4 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-40 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-41 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-42 drwxr-xr-x 2 kafka kafka 70 Apr 6 11:25 __consumer_offsets-43 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-44 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-45 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-46 drwxr-xr-x 2 kafka kafka 70 Apr 1 13:26 __consumer_offsets-47 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-48 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-49 drwxr-xr-x 2 kafka kafka 70 Apr 6 11:25 __consumer_offsets-5 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-6 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-7 drwxr-xr-x 2 kafka kafka 70 Apr 6 11:25 __consumer_offsets-8 drwxr-xr-x 2 kafka kafka 70 Apr 6 14:00 __consumer_offsets-9 drwxr-xr-x 2 kafka kafka 8192 Apr 6 14:00 appTimeLog-0 -rw-r--r-- 1 kafka kafka 0 Apr 1 11:32 cleaner-offset-checkpoint -rw-r--r-- 1 kafka kafka 55 Apr 1 11:34 meta.properties -rw-r--r-- 1 kafka kafka 1218 Apr 6 01:34 recovery-point-offset-checkpoint -rw-r--r-- 1 kafka kafka 1224 Apr 6 01:35 replication-offset-checkpoint -rw-r--r-- 1 kafka kafka 0 Apr 6 01:35 replication-offset-checkpoint.tmp

そして、ここでは（ないASC時間順に）いくつかのエラー役割ログです

... 2017-04-01 11:38:06,357 ERROR kafka.server.ReplicaFetcherThread: [ReplicaFetcherThread-0-95], Error for partition [__consumer_offsets,19] to broker 95:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. ... 2017-04-01 11:38:30,324 ERROR state.change.logger: Broker 97 received LeaderAndIsrRequest with correlation id 1 from controller 96 epoch 1 for partition [__consumer_offsets,20] but cannot become follower since the new leader 96 is unavailable. ... 2017-04-01 11:38:36,352 ERROR kafka.server.ReplicaFetcherThread: [ReplicaFetcherThread-0-96], Error for partition [__consumer_offsets,16] to broker 96:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. ... 2017-04-06 11:25:36,079 WARN kafka.log.Log: Found a corrupted index file, /var/local/kafka/data/__consumer_offsets-18/00000000000000000000.index, deleting and rebuilding index... ... 2017-04-01 13:26:41,962 WARN kafka.server.ReplicaFetcherThread: [ReplicaFetcherThread-0-95], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@769e1ff5. Possible cause: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'responses': Error reading array of size 1204063, only 290 bytes available

• その他の通常のブローカーノード：

[root@3 ~]# cd /var/local/kafka/ [root@3 kafka]# du --max-depth=1 -ch|sort -hr 48M total 48M ./data 48M . [root@4 ~]# cd /var/local/kafka/ [root@4 kafka]# du --max-depth=1 -ch|sort -hr 48M total 48M ./data 48M .

私はこれが原因と理由を探していますか？

誰もが考えていますか？

ありがとうございました。

Answer 1

複数の原因による可能性があります。パーティションとプロセスによって異なります。

たとえば、データはキーで区切られています。つまり、同じキーを持つすべてのレコードが同じパーティションで処理されることを意味します。