Hello, We have a file which we can't backup by distcp due to a mismatch lenght. After a fsck on this file I can see it is still open, but after speaking with file owner it was not. Searching on application logs show us this message: 2017-10-31 18:28:13.466 INFO [pr-8243-exec-12][o.a.hadoop.hdfs.DFSClient] Unable to close file because dfsclient was unable to contact the HDFS servers. clientRunning false hdfsTimeout -1 2017-10-31 18:28:13.468 ERROR [pr-8243-exec-12][r.c.a.LogExceptionHandler] error while processing request - uri: /xxx/yyy- query string: [zzzzzzzzzzzzz] - exception: java.io.IOException: Unable to close file because dfsclient was unable to contact the HDFS servers. clientRunning false hdfsTimeout -1 com.aaaa.bbb.ccc.ddd.exception.AmethystRuntimeException: java.io.IOException: Unable to close file because dfsclient was unable to contact the HDFS servers. clientRunning false hdfsTimeout -1 We restart HDFS service at this hour, so we think it's the main reason of this problem. Copying the file by a "hdfs dfs -cp" create a new file correctly closed, so we probably can replace the unclosed file. Nevertheless I want to know if there is another simple method to directly close a file still opened from a HDFS point of vue? Thanks for your help ... View more

Hello everyone, From what I understood every files use a minimum of 1 block in HDFS. So how can it be possible to have more files than blocks? Below is an exemple: $ hdfs fsck /app-logs Total size: 14663835874 B Total dirs: 2730 Total files: 8694 Total symlinks: 0 Total blocks (validated): 8690 (avg. block size 1687437 B) Minimally replicated blocks: 8690 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 4 Number of racks: 1 FSCK ended at Mon Sep 04 19:54:45 CEST 2017 in 353 milliseconds The filesystem under path '/app-logs' is HEALTHY I appreciate any help in understanding this ouput. Regards ... View more

Hi, In our clusters we have a strange behavior certainly due to a misconfiguration. In all ranger actions (like downloading policies, asking ranger KMS, get audits , etc....) we have a failed authentication before a success one. Everything works as expected, but do not appear optimal. Below is some examples extracted from access logs in ranger admin and kms 192.168.0.1 - - [19/Jul/2017:10:29:52 +0200] "GET /service/plugins/secure/policies/download/clusterName_kafka?lastKnownVersion=42&pluginId=kafka@host.domain-clusterName_kafka HTTP/1.1" 401 - 192.168.0.1 - - [19/Jul/2017:10:29:52 +0200] "GET /service/plugins/secure/policies/download/clusterName_kafka?lastKnownVersion=42&pluginId=kafka@host.domain-clusterName_kafka HTTP/1.1" 304 - 192.168.0.1 - - [17/Jul/2017:14:11:42 +0000] "GET /service/assets/accessAudit?page=0&pageSize=25&total_pages=66&totalCount=1626&startIndex=0&sortBy=eventTime&startDate=07%2F17%2F2017 HTTP/1.1" 401 1113 192.168.0.1 - - [17/Jul/2017:14:11:43 +0000] "GET /service/assets/accessAudit?page=0&pageSize=25&total_pages=66&totalCount=1626&startIndex=0&sortBy=eventTime&startDate=07%2F17%2F2017 HTTP/1.1" 200 11187 192.168.0.1 - - [17/Jul/2017:14:06:03 +0000] "GET /service/plugins/policy/52/versionList HTTP/1.1" 401 1113 192.168.0.1 - - [17/Jul/2017:14:06:03 +0000] "GET /service/plugins/policy/52/versionList HTTP/1.1" 200 23 192.168.0.1 - - [17/Jul/2017:14:06:03 +0000] "GET /service/plugins/policies/eventTime?eventTime=2017-07-17T14%3A05%3A47Z&policyId=52&_=1500297123319 HTTP/1.1" 401 1113 192.168.0.1 - - [17/Jul/2017:14:06:03 +0000] "GET /service/plugins/policies/eventTime?eventTime=2017-07-17T14%3A05%3A47Z&policyId=52&_=1500297123319 HTTP/1.1" 200 708 192.168.0.1 - - [19/Jul/2017:10:20:19 +0200] "OPTIONS /kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhost.domain%40CIB.NET HTTP/1.1" 401 997 192.168.0.1 - - [19/Jul/2017:10:20:19 +0200] "OPTIONS /kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhost.domain%40CIB.NET HTTP/1.1" 200 3484 192.168.0.1 - - [19/Jul/2017:10:20:19 +0200] "GET /kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhost.domain%40CIB.NET HTTP/1.1" 200 132 Is anyone get some idea of what can be wrong? We have secured clusters and two ranger admin/kms hosts on each. Thanks ... View more

Hi all, After "sucessfully" install kafka on a host with HDP 2.5.3, started the service from ambari, we are not able to produce or consum topics. We can create and describe a topic: -bash-4.2$ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper zoo.host.com:2181 --create --topic newmikl --partition 1 --replication-factor 1 Created topic "newmikl". -bash-4.2$ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper zoo.host.com:2181 --describe --topic newmikl Topic:newmikl PartitionCount:1 ReplicationFactor:1 Configs: Topic: newmikl Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001 Unfortunately in the logs.dir we can't see the tpoic files: -bash-4.2$ ls -l /hadoop/kafka/kafka-logs/ total 4 -rwxrwxrwx 1 kafkauser kafkagroup 0 Mar 17 09:45 cleaner-offset-checkpoint -rwxrwxrwx 1 kafkauser kafkagroup 57 Mar 17 09:45 meta.properties -rwxrwxrwx 1 kafkauser kafkagroup 0 Mar 17 09:45 recovery-point-offset-checkpoint -rwxrwxrwx 1 kafkauser kafkagroup 0 Mar 17 09:45 replication-offset-checkpoint I case we want to produce ad consum messages, we're facing an error: -bash-4.2$ /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list kafka.host.com:6667 --topic newmikl --security-protocol SASL_PLAINTEXT test test1 test2 test3 ^C[2017-03-17 11:14:24,025] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin) -bash-4.2$ /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper zoo.host.com:2181 --topic newmikl --security-protocol SASL_PLAINTEXT --from-beginning {metadata.broker.list=kafka.host.com:6667, request.timeout.ms=30000, client.id=console-consumer-56271, security.protocol=SASL_PLAINTEXT} [2017-03-17 11:14:35,594] WARN Fetching topic metadata with correlation id 0 for topics [Set(newmikl)] from broker [BrokerEndPoint(1001,kafka.host.com,6667)] failed (kafka.client.ClientUtils$) java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:122) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:81) at kafka.producer.SyncProducer.send(SyncProducer.scala:126) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) [2017-03-17 11:14:35,597] WARN [console-consumer-56271_slbifregkhn01.fr.intranet-1489745675236-1a4e34e3-leader-finder-thread], Failed to find leader for Set([newmikl,0]) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread) kafka.common.KafkaException: fetching topic metadata for topics [Set(newmikl)] from broker [ArrayBuffer(BrokerEndPoint(1001,kafka.host.com,6667))] failed at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) Caused by: java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:122) at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:81) at kafka.producer.SyncProducer.send(SyncProducer.scala:126) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59) ... 3 more {metadata.broker.list=kafka.host.com:6667, request.timeout.ms=30000, client.id=console-consumer-56271, security.protocol=SASL_PLAINTEXT} [...] etc... Looking kafka logs, we can see errors since the start of kafka: # tail -f /var/log/kafka/controller.log java.io.IOException: Connection to kafka.host.com:6667 (id: 1003 rack: null) failed at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63) at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59) at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112) at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120) at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59) at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233) at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) [2017-03-18 15:15:29,340] WARN [Controller-1003-to-broker-1003-send-thread], Controller 1003's connection to broker kafka.host.com:6667 (id: 1003 rack: null) was unsuccessful (kafka.controller.RequestSendThread) java.io.IOException: Connection to kafka.host.com:6667 (id: 1003 rack: null) failed at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63) at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59) at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112) at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120) at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59) at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233) at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) Anyone have an idea about what can be wrong in our configuration? Thanks. ... View more