we want to find the **approach / test /cli / API** that gives us the results about HeartBeat Lost between Ambari agent to Ambari server HeartBeat Lost could be as results of poor connection between Ambari agent to Ambari server or because Ambari server was down for along time , etc   Note - from Ambari GUI the machine with **HeartBeat Lost** state usually colored by yellow state         clarification:     the case as described here appears when `ambari-agent status` is in running state as the following ambari-agent status Found ambari-agent PID: 119315 ambari-agent running. Agent PID at: /run/ambari-agent/ambari-agent.pid Agent out at: /var/log/ambari-agent/ambari-agent.out Agent log at: /var/log/ambari-agent/ambari-agent.log ... View more

we have 3 Kafka brokers machines on on RHEL 7.9 Linux version ,   ( when each machine is physical strong `DELL HW` - memory = 512G and 96 CORE CPU )   Kafka cluster is in production mode     Kafka version is 2.7.x , and Kafka disks are in Jbod configuration each Kafka broker has 8 Jbod disks , as we can see from the following ( df -h details ) df -h /dev/sdc 1.7T 929G 748G 56% /kafka/kafka_logs2 /dev/sdd 1.7T 950G 727G 57% /kafka/kafka_logs3 /dev/sde 1.7T 999G 678G 60% /kafka/kafka_logs4 /dev/sdf 1.7T 971G 706G 58% /kafka/kafka_logs5 /dev/sdg 1.7T 1.7T 20K 100% /kafka/kafka-logs6 <----------------- /dev/sdh 1.7T 962G 714G 58% /kafka/kafka_logs7 /dev/sdi 1.7T 1.1T 621G 63% /kafka/kafka_logs8     as we can see from above disk - `/kafka/kafka_logs6` get `100%` used after short investigation we found that Kafka broker isn't tolerant when one disk is failed or disk reached to 100% , as results of this Kafka broker now is down here the Kafka `server.log`     [2022-11-29 15:43:59,723] ERROR Error while writing to checkpoint file /kafka/kafka-logs6 ............. java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:326) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at java.io.BufferedWriter.flush(BufferedWriter.java:254) at kafka.server.checkpoints.CheckpointFile.liftedTree1$1(CheckpointFile.scala:108) at kafka.server.checkpoints.CheckpointFile.write(CheckpointFile.scala:92) at kafka.server.checkpoints.LeaderEpochCheckpointFile.write(LeaderEpochCheckpointFile.scala:70) at kafka.server.epoch.LeaderEpochFileCache.flush(LeaderEpochFileCache.scala:292) at kafka.server.epoch.LeaderEpochFileCache.$anonfun$truncateFromEnd$1(LeaderEpochFileCache.scala:238) at kafka.server.epoch.LeaderEpochFileCache.truncateFromEnd(LeaderEpochFileCache.scala:235) at kafka.log.Log.$anonfun$new$1(Log.scala:305) at kafka.log.Log.<init>(Log.scala:305) at kafka.log.Log$.apply(Log.scala:2549) at kafka.log.LogManager.loadLog(LogManager.scala:273) at kafka.log.LogManager.$anonfun$loadLogs$12(LogManager.scala:352) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) **from my perspective** when we have **8 disks** on each broker and one disks is failed ( like reached 100% ) , then we expect Kafka broker still be alive even one disk is failed so as results of above scenario we searched in Kafka `server.properties` the parameter that can help us to configure the Kafka broker to be tolerant when one disk is failed but we not found it , or maybe we not know what to set in order to define the kafka broker to be tolerant with one disk failure the full parameters are: more server.properties auto.create.topics.enable=false auto.leader.rebalance.enable=true background.threads=10 log.retention.bytes=-1 log.retention.hours=48 delete.topic.enable=true leader.imbalance.check.interval.seconds=300 leader.imbalance.per.broker.percentage=10 log.dir=/kafka/kafka-logs2,/kafka/kafka-logs3 ............... log.flush.interval.messages=9223372036854775807 log.flush.interval.ms=1000 log.flush.offset.checkpoint.interval.ms=60000 log.flush.scheduler.interval.ms=9223372036854775807 log.flush.start.offset.checkpoint.interval.ms=60000 compression.type=producer log.roll.jitter.hours=0 log.segment.bytes=1073741824 log.segment.delete.delay.ms=60000 message.max.bytes=1000012 min.insync.replicas=1 num.io.threads=10 num.network.threads=48 num.recovery.threads.per.data.dir=1 num.replica.fetchers=1 offset.metadata.max.bytes=4096 offsets.commit.required.acks=-1 offsets.commit.timeout.ms=5000 offsets.load.buffer.size=5242880 offsets.retention.check.interval.ms=600000 offsets.retention.minutes=10080 offsets.topic.compression.codec=0 offsets.topic.num.partitions=50 offsets.topic.replication.factor=3 offsets.topic.segment.bytes=104857600 queued.max.requests=1000 quota.consumer.default=9223372036854775807 quota.producer.default=9223372036854775807 replica.fetch.min.bytes=1 replica.fetch.wait.max.ms=500 replica.high.watermark.checkpoint.interval.ms=5000 replica.lag.time.max.ms=10000 replica.socket.receive.buffer.bytes=65536 replica.socket.timeout.ms=30000 request.timeout.ms=30000 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 socket.send.buffer.bytes=102400 transaction.max.timeout.ms=900000 transaction.state.log.load.buffer.size=5242880 transaction.state.log.min.isr=2 transaction.state.log.num.partitions=50 transaction.state.log.replication.factor=3 transaction.state.log.segment.bytes=104857600 transactional.id.expiration.ms=604800000 unclean.leader.election.enable=false zookeeper.connection.timeout.ms=600000 zookeeper.max.in.flight.requests=10 zookeeper.session.timeout.ms=600000 zookeeper.set.acl=false broker.id.generation.enable=true connections.max.idle.ms=600000 connections.max.reauth.ms=0 controlled.shutdown.enable=true controlled.shutdown.max.retries=3 controlled.shutdown.retry.backoff.ms=5000 controller.socket.timeout.ms=30000 default.replication.factor=3 delegation.token.expiry.time.ms=86400000 delegation.token.max.lifetime.ms=604800000 delete.records.purgatory.purge.interval.requests=1 fetch.purgatory.purge.interval.requests=1000 group.initial.rebalance.delay.ms=3000 group.max.session.timeout.ms=1800000 group.max.size=2147483647 group.min.session.timeout.ms=6000 log.cleaner.backoff.ms=15000 log.cleaner.dedupe.buffer.size=134217728 log.cleaner.delete.retention.ms=86400000 log.cleaner.enable=true log.cleaner.io.buffer.load.factor=0.9 log.cleaner.io.buffer.size=524288 log.cleaner.io.max.bytes.per.second=1.7976931348623157e308 log.cleaner.max.compaction.lag.ms=9223372036854775807 log.cleaner.min.cleanable.ratio=0.5 log.cleaner.min.compaction.lag.ms=0 log.cleaner.threads=1 log.cleanup.policy=delete log.index.interval.bytes=4096 log.index.size.max.bytes=10485760 log.message.timestamp.difference.max.ms=9223372036854775807 log.message.timestamp.type=CreateTime log.preallocate=false log.retention.check.interval.ms=300000 max.connections=2147483647 max.connections.per.ip=2147483647 max.incremental.fetch.session.cache.slots=1000 num.partitions=1 producer.purgatory.purge.interval.requests=1000 queued.max.request.bytes=-1 replica.fetch.backoff.ms=1000 replica.fetch.max.bytes=1048576 replica.fetch.response.max.bytes=10485760 reserved.broker.max.id=1500 transaction.abort.timed.out.transaction.cleanup.interval.ms=60000 transaction.remove.expired.transaction.cleanup.interval.ms=3600000 zookeeper.sync.time.ms=2000 broker.rack=/default-rack   I want to add my personal feeling: *just to gives here the absurder of the above scenario lets say we have on each Kafka broker 100 disks ( in Jbod ) is it make sense that Kafka broker will be shutdown just because one disk is failed ?* ... View more

we have in our Hadoop cluster 3 Kafka machines, Kafka machines include the following services on kafka01: confluent kafka service confluent schema registry service zookeeper service on kafka02: confluent kafka service confluent schema registry service zookeeper service on kafka03: confluent kafka service confluent schema registry service zookeeper service on our Kafka cluster , we have different 34 topics one of the topic is topic name - `car.to.go` this topic include 3 replica in Kafka cluster what we found regarding to schema registry service , is that topic - `car.to.go` have different versions on kafka03 from other machines - `kafka01/02` so actually we get that versions are not in sync here is example: [root@kafka01 ~]# curl -X GET http://kafka01:8081/subjects/car.to.go-value/versions [1,2,3,4,5,6,7] [root@kafka01 ~]# curl -X GET http://kafka02:8081/subjects/car.to.go-value/versions [1,2,3,4,5,6,7] [root@kafka01 ~]# curl -X GET http://kafka03:8081/subjects/car.to.go-value/versions [1,2,3,4] from above info what could be the resewn that versions on kafka03 is diff from kafka01/02 regarding to topic `car.to.go` - what is the right way to fix this issue in way that we get on kafka03 the number versions as - `1,2,3,4,5,6,7` notes: all Kafka services and schema registry services and zookeeper services are up and running   links: https://github.com/confluentinc/schema-registry/blob/master/README.md https://docs.confluent.io/platform/current/schema-registry/develop/using.html#check-if-a-schema-is-registered-under-subject-kafka-key https://kafkawize.com/2019/03/17/commands-for-schemaregistry/   ... View more

we have Ambari hadoop cluster based on Ambari platform and installed with HDP version - 2.6.5 , when all machines in the cluster are with RHEL 7.9 version   Ambari cluster of course include the YARN service with two resource manager services   we are facing a problems about ( when master1 and master2 nodes running with the resources manager services )   Connection failed to http://master2.start.com:8088 (timed out)   we tested the below alert with following `wget` approach ,   when alert appears on Ambari then following `wget` test is hang and sometimes its take a time until `wget` finished with results   [root@master2 yarn]# wget http://master2.start.com:8088 --2022-10-28 08:12:49-- http://master2.start.com:8088/ Resolving master2.start.com (master2.start.com)... 172.3.45.68 Connecting to master2.start.com (master2.start.com)|172.3.45.68|:8088... connected. HTTP request sent, awaiting response... 307 TEMPORARY_REDIRECT Location: http://master1.start.com:8088/ [following] --2022-10-28 08:12:50-- http://master1.start.com:8088/ Resolving master1.start.com (master1.start.com)... 172.3.45.61 Connecting to master1.start.com (master01.start.com)|172.3.45.61|:8088... connected. HTTP request sent, awaiting response... 302 Found Location: http://master1.start.com:8088/cluster [following] --2022-10-28 08:12:50-- http://master1.start.com:8088/cluster Reusing existing connection to master1.start.com:8088. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: ‘index.html.35’ [ <=> ] 5,419,141 9.46MB/s in 0.5s 2022-10-28 08:12:52 (9.46 MB/s) - ‘index.html.35’ saved [5419141]     port 8088 are licensing from both nodes   ps -ef | grep `lsof -i :8088 | grep -i listen | awk '{print $2}'` yarn 1977 1 16 Oct27 ? 02:37:32 /usr/jdk64/jdk1.8.0_112/bin/java -Dproc_resourcemanager   also we cheeked with jps   jps -l | grep -i resourcemanager 1977 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager     we also verify the resources manager logs and we see the following 2022-10-27 08:04:30,071 WARN webapp.GenericExceptionHandler (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR java.lang.NullPointerException at org.apache.hadoop.yarn.api.records.ContainerId.toString(ContainerId.java:196) at org.apache.hadoop.yarn.util.ConverterUtils.toString(ConverterUtils.java:165) at org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo.<init>(AppInfo.java:169) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getApps(RMWebServices.java:603) at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) 2022-10-27 08:04:32,056 WARN webapp.GenericExceptionHandler (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR java.lang.NullPointerException at org.apache.hadoop.yarn.api.records.ContainerId.toString(ContainerId.java:196) at org.apache.hadoop.yarn.util.ConverterUtils.toString(ConverterUtils.java:165) at org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo.<init>(AppInfo.java:169) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getApps(RMWebServices.java:603) at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResource 2022-10-27 08:05:43,170 ERROR recovery.RMStateStore (RMStateStore.java:notifyStoreOperationFailedInternal(992)) - State store operation failed org.apache.hadoop.yarn.server.resourcemanager.recovery.StoreFencedException: RMStateStore has been fenced at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1213) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:1001) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doStoreMultiWithRetries(ZKRMStateStore.java:1009) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createWithRetries(ZKRMStateStore.java:1042) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.storeApplicationStateInternal(ZKRMStateStore.java:639) 2022-10-27 08:05:49,584 ERROR delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:run(659)) - ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted 2022-10-28 08:43:31,259 ERROR metrics.SystemMetricsPublisher (SystemMetricsPublisher.java:putEntity(549)) - Error when publishing entity [YARN_APPLICATION,application_1664925617878_1896] com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: Stream closed. at com.sun.jersey.api.client.ClientResponse.bufferEntity(ClientResponse.java:583) at org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPostingObject(TimelineWriter.java:157) at org.apache.hadoop.yarn.client.api.impl.TimelineWriter$1.run(TimelineWriter.java:115) at org.apache.hadoop.yarn.client.api.impl.TimelineWriter$1.run(TimelineWriter.java:112) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPosting(TimelineWriter.java:112) at org.apache.hadoop.yarn.client.api.impl.TimelineWriter.putEntities(TimelineWriter.java:92) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:348) at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.putEntity(SystemMetricsPublisher.java:536) at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.publishApplicationACLsUpdatedEvent(SystemMetricsPublisher.java:392) at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.handleSystemMetricsEvent(SystemMetricsPublisher.java:257) at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher$ForwardingEventHandler.handle(SystemMetricsPublisher.java:564) at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher$ForwardingEventHandler.handle(SystemMetricsPublisher.java:559) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Stream closed. at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:470) at java.net.SocketInputStream.available(SocketInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:353) at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:552) at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609) still not clearly from the above logs why resources manager are with the alerts - `Connection failed to http://master2.start.com:8088 (timed out)` ... View more

hi all   we have HDP cluster with 528 data nodes machines    in Ambari HDFS Configs , we configured 3 config group because the following:    1) 212 data nodes machine are with 32G 2) 119 data nodes machines are with 64G 3) 197 data nodes machines are with 128G   so in Ambari we have following config group settings        now we need to configure the parameter - " DataNode maximum Java heap size  ( dtnode_heapsize )" according to the machines memory   so we want to set the following:   on first 212 data nodes machine are with 32G ,  DataNode maximum Java heap size will set to 10G on machines - 119 data nodes machines are with 64G ,  DataNode maximum Java heap size will set to 15G on machines - 197 data nodes machines are with 128G ,  DataNode maximum Java heap size will set to 20G   so in order to configure the parameter - DataNode maximum Java heap size , on each config group , we try to use the following tool - config.py   /var/lib/ambari-server/resources/scripts/configs.py -user=admin --password=admin  --port=8080 --action=set --host=ambari_server_node --cluster=hdp_cluster7 --config-type=hadoop-env -k "dtnode_heapsize" -v "10000"   the above cli will configure the parameter - dtnode_heapsize to 10G ( 10000M )    when we run above cli , the parameter -  dtnode_heapsize was update but not on the groups !   what was update is the parameter in the default group - "Default"   so how to set the parameter - dtnode_heapsize , according to the relevant config group?    I we not sure that config.py support configuration on config group , in that case we need maybe other approach    Note - the target is to automate the settings in Ambari by API/REST API or SCRIPTS, so manual changing isn't acceptable   ... View more

we have *HDP* cluster , version 2.6.5 , and *Ambari* we have in cluster ***436 datanode*** linux machines we are trying to perform decommission and recommission on the *datanode* decommission , is working fine VIA API but when we did the recommission , then we not succeeded example: machine_name=datanode01 CLUSTER_NAME=hdp_prod5 AMBARI_MACHINE=master_02 curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{ "RequestInfo":{ "context":"Recommission DataNodes", "command":"RECOMMISSION", "parameters":{ "slave_type":"DATANODE", "excluded_hosts":"'"machine_name"'" }, "operation_level":{ "level":"HOST_COMPONENT", "cluster_name":"'"$CLUSTER_NAME"'" } }, "Requests/resource_filters":[ { "service_name":"HDFS", "component_name":"NAMENODE" } ] }' http://$AMBARI_MACHINE:8080/api/v1/clusters/$CLUSTER_NAME/requests we get the following exception:   % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1798 100 1798 0 0 126k 0 --:--:-- --:--:-- --:--:-- 159k HTTP/1.1 500 Internal Server Error X-Frame-Options: DENY X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Cache-Control: no-store Pragma: no-cache Set-Cookie: AMBARISESSIONID=147rwl7q2x4otjuh7tle4ee6y;Path=/;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT User: admin Content-Type: text/plain Content-Length: 150 { "status" : 500, "message" : "An internal system exception occurred: Unsupported action RECOMMISSION for Service: HDFS and Component: NAMENODE" any idea why we get - `Unsupported action RECOMMISSION` ? ... View more

we are running this to get statistics missing or corrupt blocks or under replica blocks   su hdfs hdfs dfsadmin -report | grep "Under replicated blocks"   and we get   Under replicated blocks: 837326   so now we are trying to fix the under replica blocks by the following procedure   hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done   and now we are runs the first command   hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files   after commands run 60 second , we noticed that `/tmp/under_replicated_files` file is empty so what is going here??   why `hdfs dfsadmin -report | grep "Under replicated blocks"` , gives `Under replicated blocks: 837326` , but the file - `/tmp/under_replicated_files` is empty   how it can be ?   is it a bug from `hdfs dfsadmin -report | grep "Under replicated blocks"` , that gives wrongs results ? ... View more

we have hadoop cluster ( Ambari platform with HDP version - 2.6.4 ) and we performed verification step in order to understand if we have *under replica blocks* the first verification was with: su hdfs hdfs fsck / - --> its gives the results: Total size: 17653549013347 B (Total open files size: 854433698229 B) Total dirs: 843714 Total files: 11752836 Total symlinks: 0 (Files currently being written: 16) Total blocks (validated): 11792203 (avg. block size 1497052 B) (Total open file blocks (not validated): 6381) Minimally replicated blocks: 11792203 (100.00001 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 6 Number of racks: 1 so as we can see above `Under-replicated blocks` is 0 **BUT** when we perform the next verification: hdfs dfsadmin -report then we get Configured Capacity: 141275429535744 (128.49 TB) Present Capacity: 140886991802565 (128.14 TB) DFS Remaining: 84748655941292 (77.08 TB) DFS Used: 56138335861273 (51.06 TB) DFS Used%: 39.85% Under replicated blocks: 4212067 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 so from above we can see that Under replicated blocks is --> `4212067` about to know what is the right under replica number: why we get differences between `hdfs fsck /` and `hdfs dfsadmin -report` ? ... View more

We have Hadoop cluster that include `datanode` machines and `5 kafka` machines Kafka machines are installed as part of hortonworks packages , `kafka` version is 0.1X We run on `datanode` the `deeg_data` applications as executors that consuming data from `kafka` topics So applications `deeg_data` are consuming data from topics partitions that exists on kafka cluster ( `deeg_data` used `kafka` client for consuming ) On last days we saw that our application – `deeg_data` are failed and we start to find the root cause On `kafka` cluster we see the following behavior   /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --group deeg_data --describe --bootstrap-server kafka1:6667 To enable GC log rotation, use -Xloggc:<filename> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=<num_of_files> where num_of_file > 0 GC log rotation is turned off Consumer group ‘deeg_data’ is rebalancing from `kafka` side `kafka` cluster is healthy and all topics are balanced and all kafka brokers are up and signed correctly to zookeeper After some time ( couple hours ) , we run again the following , but without the errors about - `Consumer group ‘deeg_data’ is rebalancing` And we get the following correctly results /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --group deeg_data --describe --bootstrap-server kafka1:6667 To enable GC log rotation, use -Xloggc:<filename> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=<num_of_files> where num_of_file > 0 GC log rotation is turned off GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER deeg_data pot.sdr.proccess 0 6397256247 6403318505 6062258 consumer-1_/10.3.6.237 deeg_data pot.sdr.proccess 1 6397329465 6403390955 6061490 consumer-1_/10.3.6.237 deeg_data pot.sdr.proccess 2 6397314633 6403375153 6060520 consumer-1_/10.3.6.237 deeg_data pot.sdr.proccess 3 6397258695 6403320788 6062093 consumer-1_/10.3.6.237 deeg_data pot.sdr.proccess 4 6397316230 6403378448 6062218 consumer-1_/10.3.6.237 deeg_data pot.sdr.proccess 5 6397325820 6403388053 6062233 consumer-1_/10.3.6.237. . . .   So we want to understand why we get: Consumer group ‘deeg_data’ is rebalancing What is the reason for above state , and why we get `rebalancing` ... View more

we have 3 Kafka brokers on Linux RHEL 7.6 ( 3 linux machines ) kafka version is 2.7.X brokers ID's are - `1010,1011,1012` from kafka described we can see the following Topic: __consumer_offsets Partition: 0 Leader: none Replicas: 1011,1010,1012 Isr: 1010 Topic: __consumer_offsets Partition: 1 Leader: 1012 Replicas: 1012,1011,1010 Isr: 1012,1011 Topic: __consumer_offsets Partition: 2 Leader: 1011 Replicas: 1010,1012,1011 Isr: 1011,1012 Topic: __consumer_offsets Partition: 3 Leader: none Replicas: 1011,1012,1010 Isr: 1010 Topic: __consumer_offsets Partition: 4 Leader: 1011 Replicas: 1012,1010,1011 Isr: 1011 Topic: __consumer_offsets Partition: 5 Leader: none Replicas: 1010,1011,1012 Isr: 1010 from Zookeeper cli we can see that broker `id 1010` not defined [zk: localhost:2181(CONNECTED) 10] ls /brokers/ids [1011, 1012] and from the log - `state-change.log` we can see the following [2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-6 as the local replica for the partition is in an offline log directory (state.change.logger) [2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-9 as the local replica for the partition is in an offline log directory (state.change.logger) [2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-8 as the local replica for the partition is in an offline log directory (state.change.logger) [2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-11 as the local replica for the partition is in an offline log directory (state.change.logger) [2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-10 as the local replica for the partition is in an offline log directory (state.change.logger) [2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-46 as the local replica for the partition is in an offline log directory (state.change.logger) [2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-45 as the local replica for the partition is in an offline log directory (state.change.logger) [2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-48 as the local replica for the partition is in an offline log directory (state.change.logger) [2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-47 as the local replica for the partition is in an offline log directory (state.change.logger) [2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 epoch 323 for partition __consumer_offsets-49 as the local replica for the partition is in an offline log directory (state.change.logger) by ls -ltr , we can see that `controller.log` and `state-change.log` are not update from `Dec 16` -rwxr-xr-x 1 root kafka 343477146 Dec 16 14:15 controller.log -rwxr-xr-x 1 root kafka 207911766 Dec 16 14:15 state-change.log -rw-r--r-- 1 root kafka 68759461 Dec 16 14:15 kafkaServer-gc.log.6.current -rwxr-xr-x 1 root kafka 6570543 Dec 17 09:42 log-cleaner.log -rw-r--r-- 1 root kafka 524288242 Dec 20 00:39 server.log.10 -rw-r--r-- 1 root kafka 524289332 Dec 20 01:37 server.log.9 -rw-r--r-- 1 root kafka 524288452 Dec 20 02:35 server.log.8 -rw-r--r-- 1 root kafka 524288625 Dec 20 03:33 server.log.7 -rw-r--r-- 1 root kafka 524288395 Dec 20 04:30 server.log.6 -rw-r--r-- 1 root kafka 524288237 Dec 20 05:27 server.log.5 -rw-r--r-- 1 root kafka 524289136 Dec 20 06:25 server.log.4 -rw-r--r-- 1 root kafka 524288142 Dec 20 07:25 server.log.3 -rw-r--r-- 1 root kafka 524288187 Dec 20 08:21 server.log.2 -rw-r--r-- 1 root kafka 524288094 Dec 20 10:52 server.log.1 -rw-r--r-- 1 root kafka 323361 Dec 20 19:50 kafkaServer-gc.log.0.current -rw-r--r-- 1 root kafka 323132219 Dec 20 19:50 server.log -rwxr-xr-x 1 root kafka 15669106 Dec 20 19:50 kafkaServer.out what we did until now is that: we restart all 3 zookeeper servers we restart all kafka brokers but still kafka broker `1010` appears as `leader none` , and not in zookeeper data   **additional info** [zk: localhost:2181(CONNECTED) 11] get /controller {"version":1,"brokerid":1011,"timestamp":"1640003679634"} cZxid = 0x4900000b0c ctime = Mon Dec 20 12:34:39 UTC 2021 mZxid = 0x4900000b0c mtime = Mon Dec 20 12:34:39 UTC 2021 pZxid = 0x4900000b0c cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x27dd7cf43350080 dataLength = 57 numChildren = 0 **from kafka01** more meta.properties # #Tue Nov 16 07:45:36 UTC 2021 cluster.id=D3KpekCETmaNveBJzE6PZg version=0 broker.id=1010 **relevant ideas** in topics disk we have the following files ( additionally to topics partitions ) -rw-r--r-- 1 root kafka 91 Nov 16 07:45 meta.properties -rw-r--r-- 1 root kafka 161 Dec 15 16:04 cleaner-offset-checkpoint -rw-r--r-- 1 root kafka 13010 Dec 15 16:20 replication-offset-checkpoint -rw-r--r-- 1 root kafka 1928 Dec 17 09:42 recovery-point-offset-checkpoint -rw-r--r-- 1 root kafka 80 Dec 17 09:42 log-start-offset-checkpoint any idea if deletion of one or more of above files can help with our issue? ... View more

we are using HDP cluster with *182* `data node` machines   *HDP version - 2.6.4 Ambari version 2.6.1*   we notes the following behavior on the `data nodes` machines ( its happens on all `data-node` machines and on all disks )   when we perform the command as above example:   ps -eo s,user,cmd | grep ^[RD] D hdfs du -sk /grid/sdj/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990 D hdfs du -sk /grid/sdm/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990 R root ps -eo s,user,cmd   Note - each disk in the `data node` is `5.4 T Bytes`   we can see that `HDFS` is running the "du -sk" on the `data node` disks we don't so like this , because the meaning of that is consuming high load CPU Avrg and sometimes even bad performance   we are understand that `HDFS` need to run the "`du -sk`" in order to verify the disks space , **but** on other hand its cost - high CPU load avrg and sometimes even poor performance so - is it possible to tell `HDFS` in some way to disable this verification ?   ... View more

we can see from our `RHEL 7.6` server ( kernel version - `3.10.0-957.el7.x86_64` ) that following process are with `D` state ( they runs from `HDFS` user )   Note - *D state code means that process is in uninterruptible sleep*   ps -eo s,user,cmd | grep ^[RD] D hdfs du -sk /grid/sdj/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990 D hdfs du -sk /grid/sdm/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990 R root ps -eo s,user,cmd   note's - the disks `sdj` and `sdm` are `3T` byte size , also `"du -sk`" happens on other disks as `sdd , sdf` etc   we are suspect that the fact that we have high CPU load avrg is because the "du -sk" that actually run on the disks   so I was thinking what we can do regarding to below behavior   one option is maybe to disable the "`du -sk`" verification from `HDFS` , but no clue how to do that second option is to think what actually cause the `D` state ?   I don't sure ... but maybe upgrade the kernel version will help to avoid D state? or else? ( like disable the CPU Thread(s) ) , etc ?   more details lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 48 On-line CPU(s) list: 0-47 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 and CPU LOAD AVRG is around ~ 42-45 ( for 15min avrg )   Reference :   https://community.cloudera.com/t5/Support-Questions/Does-hadoop-run-dfs-du-automatically-when-a-new-job-starts/td-p/231297 https://community.cloudera.com/t5/Support-Questions/Can-hdfs-dfsadmin-and-hdfs-dsfs-du-be-taxing-on-my-cluster/m-p/182402 https://community.pivotal.io/s/article/Dealing-with-Processes-in-State-D---Uninterruptible-Sleep-Usually-IO?language=en_US ... View more

we have HDP cluster ( Hadoop cluster version 2.6.5 ) and we add the ranger and ranger KMS to the cluster as services after adding the service - ranger KMS and do some settings we performed the following   [hdfs@worker01 tmp]$ hdfs dfs -mkdir /zone_encr_1 [hdfs@worker01 tmp]$ hdfs crypto -createZone -keyName secret_hdp1 -path /zone_encr_1 Added encryption zone /zone_encr_1 [hdfs@worker01 tmp]$ hdfs dfs -copyFromLocal file.txt /zone_encr_1 [hdfs@worker01 tmp]$ hdfs dfs -cat /zone_encr_1/file.txt hello every one [hdfs@worker01 tmp]$ hdfs dfs -ls /zone_encr_1/file.txt -rw-r--r-- 2 hdfs hdfs 23 2021-11-24 20:19 /zone_encr_1/file.txt [hdfs@worker01 tmp]$ hdfs crypto -listZones /zone_encr secret_hdp1 /zone_encr_new secret_hdp1 /zone_encr_1 secret_hdp1 as we can see above first we create folder - /zone_encr_1 under hdfs then we add encryption to folder - /zone_encr_1 then we copy from local folder the file - file.txt that include the text - "hello every one" to hdfs folder - /zone_encr_1 then we do the test with `hdfs dfs -cat /zone_encr_1/file.txt` and we expect to get encrypted file , but we not   we still get the file as   hello every one since I just to learn the ranger KMS capabilities , I am not sure if I missed something       https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/copy-to-from-encr-zone.html ... View more

hi all   we have Kafka server with version 2.6 on Linux rhel machine    what is wrong with the following Kafka cli   kafka-topics.sh --bootstrap-server="kafka1:6667" --describe|more Error while executing topic command : The broker does not support DESCRIBE_CONFIGS [2021-09-02 11:19:02,486] ERROR org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support DESCRIBE_CONFIGS (kafka.admin.TopicCommand$)   is above cli - is valid approach ?  ... View more

we have Hadoop cluster ( HDP 2.6.5 cluster with ambari , with 25 datanodes & nodemanager machines ) we are using spark streaming application (spark 2.1 run over Hortonworks 2.6.x ) the current situation is that spark streaming applications runs on all datanodes & node-manager machines but from the   resources manager   logs we see the following INFO warning:   2021-06-27 14:07:01,456 INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes: app_id=application_1624802728037_0004 existingReservations=10 totalAvailableNodes=181 reservableNodesRatio=0.05 numAllowedReservations=10 2021-06-27 14:07:01,456 INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes: app_id=application_1624802728037_0003 existingReservations=10 totalAvailableNodes=181 reservableNodesRatio=0.05 numAllowedReservations=10 2021-06-27 14:07:01,456 INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes: app_id=application_1624802728037_0009 existingReservations=10 totalAvailableNodes=181 reservableNodesRatio=0.05 numAllowedReservations=10 regarding to   Reservation Exceeds Allowed number of nodes what is the customization that we should to do in order to avoid above those messages? ... View more

we have 2 resource managers that are working as part of HDP cluster the first resource manager is failed after couple minutes from the log of the resource manager we can see the following lines that returned many times   2021-06-27 14:07:16,022 INFO scheduler.SchedulerNode (SchedulerNode.java:allocateContainer(152)) - Assigned container container_e83_1624802728037_0001_01_004355 of capacity <memory:86016, vCores:5> on host datanode23.fgtf.com:45454, which has 5 containers, <memory:199680, vCores:15> used and <memory:27708, vCores:75> available after allocation 2021-06-27 14:07:01,279 INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes: app_id=application_1624802728037_0001 existingReservations=10 totalAvailableNodes=181 reservableNodesRatio=0.05 numAllowedReservations=10 2021-06-27 14:07:01,279 INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes: app_id=application_1624802728037_0004 existingReservations=10 totalAvailableNodes=181 reservableNodesRatio=0.05 numAllowedReservations=10 2021-06-27 14:07:01,279 INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes: app_id=application_1624802728037_0003 existingReservations=10 totalAvailableNodes=181 reservableNodesRatio=0.05 numAllowedReservations=10 2021-06-27 14:07:01,279 INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes:     AND   2021-06-27 14:07:01,279 INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes: app_id=application_1624802728037_0001 existingReservations=10 totalAvailableNodes=181 reservableNodesRatio=0.05 numAllowedReservations=10 2021-06-27 14:07:01,279 INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes: app_id=application_1624802728037_0004 existingReservations=10 totalAvailableNodes=181 reservableNodesRatio=0.05 numAllowedReservations=10 2021-06-27 14:07:01,279 INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes: app_id=application_1624802728037_0003 existingReservations=10 totalAvailableNodes=181 reservableNodesRatio=0.05 numAllowedReservations=10 2021-06-27 14:07:01,279 INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes: and 2021-06-27 14:07:01,282 INFO webproxy.WebAppProxyServlet (WebAppProxyServlet.java:doGet(382)) - dr.who is accessing unchecked http://22.20.101.13:53092/api/v1/applications/application_1624802728037_0009/executors which is the app master GUI of application_1624802728037_0009 owned by hdfs 2021-06-27 14:07:01,282 INFO webproxy.WebAppProxyServlet (WebAppProxyServlet.java:doGet(382)) - dr.who is accessing unchecked http://22.20.14.5:36198/api/v1/applications/application_1624801538018_0011/executors which is the app master GUI of application_1624801538018_0011 owned by hdfs any idea what is the meaning of the INFO about? INFO fair.FSAppAttempt (FSAppAttempt.java:reservationExceedsThreshold(495)) - Reservation Exceeds Allowed number of nodes: app_id=application_1624802728037_0001 existingReservations=10 totalAvailableNodes=181 reservableNodesRatio=0.05 numAllowedReservations=10   ... View more

  in Ambari we have the following feature that is disable , under yarn Configs what is the relevant ambari rest api in order to change the state of CPU Scheduling from disable to enable ? ... View more