Background Customer attached bigger disks to expand the data node storage. If one disk physical fail, can also use this solution. Step by Step Decommission HDFS DataNode component on the host It takes hours to finish, depends on your existing data size DataNode is decommissioned Turn the host in maintenance Stop all components on the host Change linux /etc/fstab mount new disks to existing mount points. If possible use uuid rather than disk device. uuid is much stable especially in cloud environment. Manually create the yarn log and local folders in the mount points. Because we don't reprovision the host, yarn won't create create these dirs in your configure, but try to reuse them. # for disk in /hadoop/disk-sd{d..j}/Hadoop > do > mkdir ${disk}/yarn/log && chown yarn:hadoop ${disk}/yarn/log > mkdir ${disk}/yarn/local && chown yarn:hadoop ${disk}/yarn/local > done After change Linux disk mount configuration, start all components on the host Recommission DateNode Turn off maintenance Check hdfs blocks $ hdfs fsck / | egrep -v '^\.+$' | grep -v eplica FSCK started by hdfs (auth:KERBEROS_SSL) from /192.168.141.39 for path / at Tue Nov 29 10:42:34 UTC 20161.............................................................................................Status: HEALTHY Total size: 769817156313 B (Total open files size: 75189484 B) Total dirs: 4934 Total files: 23693 Total symlinks: 0 (Files currently being written: 30) Total blocks (validated): 27536 (avg. block size 27956753 B) (Total open file blocks (not validated): 24) Corrupt blocks: 0 Number of data-nodes: 7 Number of racks: 1FSCK ended at Tue Nov 29 10:42:34 UTC 2016 in 433 millisecondsThe filesystem under path '/' is HEALTHY ... View more

Background When we use NiFi flow to load Adobe ClickStream tsv file into hive, we found around 3% rows are in wrong format or missed. Source Data Quality $ awk -F "\t" '{print NF}' 01-weblive_20161014-150000.tsv | sort | uniq -c | sort 1 154 1 159 1 162 1 164 1 167 1 198 1 201 1 467 2 446 2 449 2 569 6 13 10 3 13 146 13 185 15 151 16 54 18 433 21 432 22 238 23 102 26 2 34 138 179 1 319412 670 After clean the tsv $ awk -F "\t" 'NF == 670' 01-weblive_20161014-150000.tsv >> cleaned.tsv $ awk -F "\t" '{print NF}' cleaned.tsv | sort | uniq -c | sort 319412 670 Still missed a few percent rows. Root Cause and Solution We are using ConvertCSVToAvro and ConvertAvroToORC. The clickstrem tsv files have " in them and the ConvertCSVtoAvro processor uses " as the value for the "CSV quote Character" processor configuration property by default. As a result many tabbed fields end up in the same record. We can get good output by changing this configuration property to another character that is not used in input files anywhere. We used ¥ So when use CSV related processor, double check the contents don't have the quote character. ... View more

Error Ranger Tagsync shows lots of KafkaException in log file, which causes disk space alert in Ambari. Also it used out all of the client port. /var/log/ranger/tagsync/tagsync.log 10 Nov 2016 11 : 46 : 43 ERROR TagSynchronizer [main] - 262 tag-source:atlas initialization failed with javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, b ut the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user kafka.common.KafkaException: fetching topic metadata for topics [Set(ATLAS_ENTITIES)] from broker [ArrayBuffer(BrokerEndPoint( 1001 ,host.domain , 6667 ))] failed at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala: 73 ) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala: 96 ) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala: 67 ) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala: 63 ) Caused by: java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala: 122 ) at kafka.producer.SyncProducer.liftedTree1$ 1 (SyncProducer.scala: 82 ) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$doSend(SyncProducer.scala: 81 ) at kafka.producer.SyncProducer.send(SyncProducer.scala: 126 ) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala: 59 ) ... 3 more Background The Atlas was installed after the HDP2.5.0 was kerberosed. Ambari2.4.1 doesn't create the kerberos principal for Ranger Tagsync, and distributed to the node. Could find the hint from Tagsync log: /var/log/ranger/tagsync/tagsync.log 10 Nov 2016 11 : 46 : 41 WARN SecureClientLogin [main] - 119 /etc/security/keytabs/rangertagsync.service.keytab doesn't exist. 10 Nov 2016 11 : 46 : 41 WARN SecureClientLogin [main] - 130 Can't find principal : rangertagsync/host.domain @REALM Fix Manually create rangertagsync principal and keytab. kadmin.local: add_principal -randkey rangertagsync/ <code>rangertagsync/host.domain @REALM kadmin.local: xst -k rangertagsync.service.keytab rangertagsync/<code>rangertagsync/host.domain @REALM Deploy keytab to the node $ sudo cp rangertagsync.service.keytab /etc/security/keytabs/ $ sudo chown ranger:hadoop /etc/security/keytabs/rangertagsync.service.keytab $ sudo chmod 440 /etc/security/keytabs/rangertagsync.service.keytab No errors in the Ranger Tagsync log. ... View more