About rohit_r_sharma

rohit_r_sharma · ‎12-24-2020

Dear Team, Problem Statement: While creating new topics ISR for all the topics getting shrink automatically, even though we created topic it was throwing below errors in kafka server.log. VERSIONS: Kafka version (0.10.0), HDP-2.6.3.0 No. of Brokers 3 with 16GB JVM Heap. We have tried below steps as per the URL but no luck 😞 https://medium.com/@nblaye/reset-consumer-offsets-topic-in-kafka-with-zookeeper-5910213284a2 We stop all brokers. Removed below files from all brokers. cleaner-offset-checkpoint , .lock , recovery-point-offset-checkpoint , replication-offset-checkpoint Restarted all brokers. Logs for reference found in server.log [2020-12-24 00:22:23,481] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,9] to broker 1002:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2020-12-24 00:18:59,924] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,36] to broker 1002:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread) [2020-12-24 00:18:59,951] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,45] to broker 1002:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread) [2020-12-24 01:14:20,923] INFO Partition [__consumer_offsets,14] on broker 1003: Shrinking ISR for partition [__consumer_offsets,14] from 1002,1003,1001 to 1002,1003 (kafka.cluster.Partition) [2020-12-24 01:14:20,925] INFO Partition [__consumer_offsets,32] on broker 1003: Shrinking ISR for partition [__consumer_offsets,32] from 1002,1003,1001 to 1002,1003 (kafka.cluster.Partition) [2020-12-24 01:14:20,927] INFO Partition [__consumer_offsets,29] on broker 1003: Shrinking ISR for partition [__consumer_offsets,29] from 1003,1002,1001 to 1003,1002 (kafka.cluster.Partition) [2020-12-24 01:14:20,928] INFO Partition [__consumer_offsets,44] on broker 1003: Shrinking ISR for partition [__consumer_offsets,44] from 1003,1002,1001 to 1003,1002 (kafka.cluster.Partition) Please suggest on this and Thanks in advance.

rohit_r_sharma · ‎09-05-2019

Hi All, Resolved using below steps: 1) To observe the Datanode threads: Created a widget in Ambari under HDFS for DataNode Threads (Runnable, Waited, Blocked) Monitored that from a particular date the threads went in wait stage. Exported the graph widget CSV file to view the exact time of wait threads. 2) Restart all Datanodes manually and observed that the wait threads were released. 3) With default 4096 threads the DataNode is running properly. Still unable to understand: 1) How to check the wait threads are in which DataNode? 2) Which task or process tend to threads in the wait stage? Would like to know if anyone comes across this and able to find in detail. Else the above steps are the only solution for wait threads.

rohit_r_sharma · ‎08-29-2019

Ambari 2.6 and HDP 2.6.3. The error is displayed while performing the following error: 1) HDFS get operation. 2) While aggregating and writing file on HDFS using pyspark. Error: "19/08/29 15:53:02 WARN hdfs.DFSClient: Failed to connect to /DN_IP:1019 for block, add to deadNodes and continue. java.io.EOFException: Premature EOF: no length prefix available " We found the following links to resolve the above error. => To set dfs.datanode.max.transfer.threads=8196 1) https://www.netiq.com/documentation/sentinel-82/admin/data/b1nbq4if.html (Performance Tuning Guidelines) 2) https://github.com/hortonworks/structor/issues/7 (jmaron commented on Jul 28, 2014) Could you all please suggest shall i go ahead with this resolution? Does this configuration affects any other services? Thankyou

rohit_r_sharma · ‎06-11-2019

@Jay Kumar SenSharma, Thanks for the support!!! Yeah, there was inconsistency in Ambari-Server DB which was not allowing Alert to function on Ambari-UI. The Ambari-server DB size was grown to 294 MB. By purging the last 6 months from DB and restarting the ambari functioned the Alerts back on Ambari-UI. Would like to know in detail if this happens on PROD env what measures should be taken as an admin.

rohit_r_sharma · ‎06-11-2019

@Geoffrey Shelton Okot, The purging saved me. Thanks a lot for the support. It would be great if I can automate purging every 6 months, please let me know if there is any way to do it.

rohit_r_sharma · ‎06-10-2019

@Geoffrey Shelton Okot, The ambari-server DB was created on 2018-09-04 20:10:19. The size of table schema ambari which belongs to ambari-server is 294 MB. Do you really think I need to purge the schema? Should the ambari-server schema size have a limit? If it is so How to limit it? OR automate purging?

rohit_r_sharma · ‎06-10-2019

@Jay Kumar SenSharma 1st Point ambari-server.log : Error Processing URI: /api/v1/clusters/cluster-name/alerts - (java.lang.NullPointerException) null 2nd Incognito Browser- Console (Tab) 3rd There is enough memory for Ambari-Server to function approx 166GB is Available. 4th With the help of your link above there is no log in ambari server saying exceeded Java Heap Size. The error in ambari-server is Java Null Pointer Exception null running alerts tab. Please Suggest.

rohit_r_sharma · ‎05-31-2019

Hi All, The configured HDP 2.6.3 cluster with Ambari 2.6 is unable to show contents inside the Alerts page. Can anyone help me, how to find the RCA for the same? Attaching screenshot of the Ambari Alerts Page: How to monitor log ? for this Alert page.

rohit_r_sharma · ‎09-26-2018

@Gitanjali Bare Even i have faced the timeout error. There are two ways to solve this error: 1) Validate the KDC port 88 is allowed to ESTABLISH for both TCP and UDP connections. netstat -an | grep 88 2) If the UDP is not allowed to used, add the following entry in krb5.conf under [libdefaults] udp_preference_limit = 1 This worked for me . Hope this will help you also. Thanks...

rohit_r_sharma · ‎09-18-2018

@Gonçalo Cunha, Thanks for your response. the current krb5.conf looks like: [libdefaults] renew_lifetime = 7d forwardable = true default_realm = XYZ.COM ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false default_ccache_name = /tmp/krb5cc_%{uid} #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 [domain_realm] .example.com = XYZ.COM example.com = XYZ.COM [logging] default = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log kdc = FILE:/var/log/krb5kdc.log [realms] XYZ.COM = { admin_server = FQDN kdc = FQDN } The principal pointing to this is like: username/hostname@XYZ.COM Keytab: username.service.keytab

Online	Offline
Last Visited	‎01-13-2021 12:43 AM

Member Since	‎06-15-2016 10:46 AM
Last Visited	‎01-13-2021 12:43 AM
Posts	45
Kudos received	1

Cloudera Community

Re: Premature EOF: Error while reading and writing...

Unable to create topic in kafka cluster.

Re: Premature EOF: Error while reading and writing...

Premature EOF: Error while reading and writing dat...

Re: Unable to view Ambari Alerts page content

Re: Unable to view Ambari Alerts page content

Re: Unable to view Ambari Alerts page content

Re: Unable to view Ambari Alerts page content

Unable to view Ambari Alerts page content

Re: Publish Kafka Error --- Time out

Re: Caused by: sun.security.krb5.KrbException: Gen...