Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to check corrupted/bad files on kafka machines

how to check corrupted/bad files on kafka machines

we have hadoop cluster ( version - 2.6.0.x ) with 3 physical kafka machines

we want to check or verify the logs under /var/kafka/kafka-logs/ in order to understand which files are corrupted or have missing info ... etc

kafka version - 10.0.2

is it possible to verify the logs by CLI or other approach?

the reason for my answer is because we notice from /var/log/kafka/server.log problem ( indexes corupted files ) with the indexes and timeindexes and we worry if the files are also part of the problem

ls -ltr /var/kafka-kafka-logs

drwxr-xr-x. 2 kafka hadoop 16384 May 15 14:22 kuapi.jurg.pri.processed-1
drwxr-xr-x. 2 kafka hadoop 20480 May 15 14:22 kuapi.jurg.pri.decoded-95
drwxr-xr-x. 2 kafka hadoop 16384 May 15 14:22 kuapi.jurg.pri.decoded-36
drwxr-xr-x. 2 kafka hadoop 16384 May 15 14:22 kuapi.jurg.pri.processed-53
drwxr-xr-x. 2 kafka hadoop 20480 May 15 14:22 kuapi.jurg.pri.decoded-44
drwxr-xr-x. 2 kafka hadoop 20480 May 15 14:22 kuapi.jurg.pri.processed-17
drwxr-xr-x. 2 kafka hadoop 20480 May 15 14:22 kuapi.jurg.pri.decoded-83
drwxr-xr-x. 2 kafka hadoop 20480 May 15 14:22 kuapi.jurg.pri.decoded-15
drwxr-xr-x. 2 kafka hadoop 16384 May 15 14:22 kuapi.jurg.pri.processed-66
drwxr-xr-x. 2 kafka hadoop  4096 May 15 14:22 kuapi.jurg.tc.ppe_retry.gad-0
drwxr-xr-x. 2 kafka hadoop  4096 May 15 14:22 kuapi.jurg.tc.ppe_retry.wce-2
drwxr-xr-x. 2 kafka hadoop  4096 May 15 14:22 kuapi.jurg.tc.ppe_retry.wce-3
drwxr-xr-x. 2 kafka hadoop  4096 May 15 14:22 kuapi.jurg.tc.ppe_mission.imei-4
drwxr-xr-x. 2 kafka hadoop  4096 May 15 14:22 kuapi.jurg.pri.tc.initial.nce-4
drwxr-xr-x. 2 kafka hadoop  4096 May 15 14:22 kuapi.jurg.tc.ppe_update.wce-5
drwxr-xr-x. 2 kafka hadoop  4096 May 15 14:22 __consumer_offsets-14
drwxr-xr-x. 2 kafka hadoop 16384 May 15 14:22 kuapi.jurg.pri.decoded-55
drwxr-xr-x. 2 kafka hadoop 28672 May 15 14:22 kuapi.jurg.mdr.initial-1
drwxr-xr-x. 2 kafka hadoop 16384 May 15 14:22 kuapi.jurg.pri.processed-79
drwxr-xr-x. 2 kafka hadoop 16384 May 15 14:22 kuapi.jurg.pri.decoded-8
Michael-Bronson