Member since
β06-15-2016
45
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
8583 | β09-05-2019 04:46 AM |
β12-24-2020
02:20 AM
Dear Team, Problem Statement: While creating new topics ISR for all the topics getting shrink automatically, even though we created topic it was throwing below errors in kafka server.log. VERSIONS: Kafka version (0.10.0), HDP-2.6.3.0 No. of Brokers 3 with 16GB JVM Heap. We have tried below steps as per the URL but no luck π https://medium.com/@nblaye/reset-consumer-offsets-topic-in-kafka-with-zookeeper-5910213284a2 We stop all brokers. Removed below files from all brokers. cleaner-offset-checkpoint , .lock , recovery-point-offset-checkpoint , replication-offset-checkpoint Restarted all brokers. Logs for reference found in server.log [2020-12-24 00:22:23,481] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,9] to broker 1002:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2020-12-24 00:18:59,924] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,36] to broker 1002:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread) [2020-12-24 00:18:59,951] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,45] to broker 1002:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread) [2020-12-24 01:14:20,923] INFO Partition [__consumer_offsets,14] on broker 1003: Shrinking ISR for partition [__consumer_offsets,14] from 1002,1003,1001 to 1002,1003 (kafka.cluster.Partition) [2020-12-24 01:14:20,925] INFO Partition [__consumer_offsets,32] on broker 1003: Shrinking ISR for partition [__consumer_offsets,32] from 1002,1003,1001 to 1002,1003 (kafka.cluster.Partition) [2020-12-24 01:14:20,927] INFO Partition [__consumer_offsets,29] on broker 1003: Shrinking ISR for partition [__consumer_offsets,29] from 1003,1002,1001 to 1003,1002 (kafka.cluster.Partition) [2020-12-24 01:14:20,928] INFO Partition [__consumer_offsets,44] on broker 1003: Shrinking ISR for partition [__consumer_offsets,44] from 1003,1002,1001 to 1003,1002 (kafka.cluster.Partition) Please suggest on this and Thanks in advance.
... View more
Labels:
- Labels:
-
Apache Kafka
-
Kerberos
β09-05-2019
04:46 AM
Hi All, Resolved using below steps: 1) To observe the Datanode threads: Created a widget in Ambari under HDFS for DataNode Threads (Runnable, Waited, Blocked) Monitored that from a particular date the threads went in wait stage. Exported the graph widget CSV file to view the exact time of wait threads. 2) Restart all Datanodes manually and observed that the wait threads were released. 3) With default 4096 threads the DataNode is running properly. Still unable to understand: 1) How to check the wait threads are in which DataNode? 2) Which task or process tend to threads in the wait stage? Would like to know if anyone comes across this and able to find in detail. Else the above steps are the only solution for wait threads.
... View more
β08-29-2019
04:20 AM
Ambari 2.6 and HDP 2.6.3. The error is displayed while performing the following error: 1) HDFS get operation. 2) While aggregating and writing file on HDFS using pyspark. Error: "19/08/29 15:53:02 WARN hdfs.DFSClient: Failed to connect to /DN_IP:1019 for block, add to deadNodes and continue. java.io.EOFException: Premature EOF: no length prefix available " We found the following links to resolve the above error. => To set dfs.datanode.max.transfer.threads=8196 1) https://www.netiq.com/documentation/sentinel-82/admin/data/b1nbq4if.html (Performance Tuning Guidelines) 2) https://github.com/hortonworks/structor/issues/7 (jmaron commented on Jul 28, 2014) Could you all please suggest shall i go ahead with this resolution? Does this configuration affects any other services? Thankyou
... View more
Labels:
β06-11-2019
11:37 AM
@Jay Kumar SenSharma, Thanks for the support!!! Yeah, there was inconsistency in Ambari-Server DB which was not allowing Alert to function on Ambari-UI. The Ambari-server DB size was grown to 294 MB. By purging the last 6 months from DB and restarting the ambari functioned the Alerts back on Ambari-UI. Would like to know in detail if this happens on PROD env what measures should be taken as an admin.
... View more
β06-11-2019
11:28 AM
@Geoffrey Shelton Okot, The purging saved me. Thanks a lot for the support. It would be great if I can automate purging every 6 months, please let me know if there is any way to do it.
... View more
β06-10-2019
01:06 PM
@Geoffrey Shelton Okot, The ambari-server DB was created on 2018-09-04 20:10:19. The size of table schema ambari which belongs to ambari-server is 294 MB. Do you really think I need to purge the schema? Should the ambari-server schema size have a limit? If it is so How to limit it? OR automate purging?
... View more
β06-10-2019
12:20 PM
@Jay Kumar SenSharma 1st Point ambari-server.log : Error Processing URI: /api/v1/clusters/cluster-name/alerts - (java.lang.NullPointerException) null 2nd Incognito Browser- Console (Tab) 3rd There is enough memory for Ambari-Server to function approx 166GB is Available. 4th With the help of your link above there is no log in ambari server saying exceeded Java Heap Size. The error in ambari-server is Java Null Pointer Exception null running alerts tab. Please Suggest.
... View more
β05-31-2019
08:45 AM
Hi All, The configured HDP 2.6.3 cluster with Ambari 2.6 is unable to show contents inside the Alerts page. Can anyone help me, how to find the RCA for the same? Attaching screenshot of the Ambari Alerts Page: How to monitor log ? for this Alert page.
... View more
Labels:
- Labels:
-
Apache Ambari
β09-26-2018
01:32 PM
@Gitanjali Bare Even i have faced the timeout error. There are two ways to solve this error: 1) Validate the KDC port 88 is allowed to ESTABLISH for both TCP and UDP connections. netstat -an | grep 88 2) If the UDP is not allowed to used, add the following entry in krb5.conf under [libdefaults] udp_preference_limit = 1 This worked for me . Hope this will help you also. Thanks...
... View more
β09-18-2018
10:45 AM
@Gonçalo Cunha, Thanks for your response. the current krb5.conf looks like: [libdefaults]
renew_lifetime = 7d
forwardable = true
default_realm = XYZ.COM
ticket_lifetime = 24h
dns_lookup_realm = false
dns_lookup_kdc = false
default_ccache_name = /tmp/krb5cc_%{uid}
#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
.example.com = XYZ.COM
example.com = XYZ.COM
[logging]
default = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
kdc = FILE:/var/log/krb5kdc.log
[realms]
XYZ.COM = {
admin_server = FQDN
kdc = FQDN
} The principal pointing to this is like: username/hostname@XYZ.COM Keytab: username.service.keytab
... View more