Member since
05-15-2019
42
Posts
20
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3151 | 11-21-2017 04:48 AM | |
1135 | 05-02-2017 02:55 AM |
05-06-2022
02:42 AM
@arunpoy If we are using the CDH/CDP both timeout parameters(hbase.rpc.timeout & hbase.client.scanner.timeout.period) need to be added in both server-side and client-side in the below paths from the HBase configuration. HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml Bothe the time-out parameters need to be added on the server and client-side and RPC(hbase.rpc.timeout) time-out needs to be set a bit higher than the client scanner time-out (hbase.client.scanner.timeout.period).
... View more
05-01-2020
06:29 AM
Hi , where did you found the property he talked about to modify it ? because I'm facing the same issue ? thank in advance
... View more
10-08-2018
12:38 PM
Is there any configuration parameter related to heap configuration for n number of watches and znode count ?
... View more
02-28-2019
08:25 AM
Hi @Rodrigo Hjort , did you solve this problem and if yes, how ?
... View more
10-03-2018
10:58 AM
Hi, Please find the below steps for moving zookeeper data directory. change dataDir conf in ambari ( Go to Ambari -> ZooKeeper -> Configs -> ZooKeeper Server -> ZooKeeper directory /mnt/scratch/zookeeper) Stop
all zookeeper servers ( Zookeeper -> service actions -> stop ) copy contents to new dir, change permission of folder (myid and version-2/ ) . Login to zookeeper1 node $ cp -r /mnt/sda/zookeeper/* /mnt/scratch/zookeeper/ $ chown -R zookeeper:hadoop /mnt/scratch/zookeeper/ start only zookeeper1 node zookeeper server from ambari UI repeat 2-4 for other two zookeeper servers (zookeeper2 and zookeeper3) Restart all services if require.
... View more
01-02-2017
11:58 AM
4 Kudos
PROBLEM
Zookeeper
transaction logs and snapshot files are created very frequently (multiple files
in every minute) and that fills up the FileSystem in a very short time.
ROOT CAUSE
One or more
application are creating or modifying the znodes too frequently, causing too
many transactions in a short duration. This leads to the creation of too many
transactional log files and snapshot files since they get rolled over after
100,000 entries by default (as defined by zookeeper property 'snapCount')
RESOLUTION
The resolution for
such cases involves reviewing the zookeeper transaction logs to find the znodes
that are updated/created most frequently using the following command on one of
the zookeeper servers:
# cd /usr/hdp/current/zookeeper-server
# java -cp zookeeper.jar:lib/* org.apache.zookeeper.server.LogFormatter /hadoop/zookeeper/version-2/logxxx
(where 'dataDir' is set to '/hadoop/zookeeper' within zookeeper configuration)
Once the frequently
updating znodes are identified using the above command, one should continue
with fixing the related application that is creating such a large number of
updates on zookeeper.
An example of such
an application that can cause this problem is Hbase, when there are very large
number of regions stuck in transition and they repeatedly fail to
become online.
... View more
Labels:
12-12-2015
05:05 PM
@rmaruthiyodan, @Jonas Straub Agreed! We might need to run hive metatool to update existing nameservice URI.
... View more
11-05-2015
03:36 PM
1 Kudo
@rmaruthiyodan@hortonworks.com Please see the following information and see if your nodes are healthy , connectivity between them is stable. It is desirable for correctness of the system that only one NameNode be in the Active state at any given time. Importantly, when using the Quorum Journal Manager, only one NameNode will ever be allowed to write to the JournalNodes, so there is no potential for corrupting the file system metadata from a split-brain scenario. However, when a failover occurs, it is still possible that the previous Active NameNode could serve read requests to clients, which may be out of date until that NameNode shuts down when trying to write to the JournalNodes. For this reason, it is still desirable to configure some fencing methods even when using the Quorum Journal Manager.'
Further reading can be found here: http://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#Automatic_Failover
If NameNode remains unresponsive for long enough, ZooKeeper notices, and gives control to the HA backup NameNode. The backup NameNode increments the epoch-count in the journal nodes (as it should) and takes over control of HDFS. Eventually, the AD call returns, and the former NameNode wakes up, notices that the epoch-count in the journal nodes has inexplicably increased by one, and shuts itself down, as it should do in response to this condition. (It is designed to do this to avoid two NameNodes in a split-brain situation.)
... View more
10-28-2015
10:53 AM
@rmaruthiyodan@hortonworks.com - It's supported as far as I know. You are using zookprusr (example) for zookepper , as long as zookeeper service is up , we are good. Kafka Kerberos Doc Client { // used for zookeeper connection
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
storeKey=true
useTicketCache=false
serviceName="zookeeper"
principal="kafka/c6401.ambari.apache.org@EXAMPLE.COM";
};
... View more
10-07-2015
03:27 AM
@aagarwal@hortonworks.com Thanks for confirming and the details.
... View more