Member since
05-15-2019
42
Posts
20
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3069 | 11-21-2017 04:48 AM | |
1109 | 05-02-2017 02:55 AM |
10-05-2018
05:41 PM
This could be KDC server issue. Please check if you are hitting this: https://bugzilla.redhat.com/show_bug.cgi?id=1560951
... View more
11-21-2017
05:42 AM
@suresh krish As the referred article too suggests, if you were to attempt recovering accidentally deleted files in Production cluster's HDFS, then it is recommended to immediately stop all the DataNodes in the cluster and seek support's help to go through the process. When it is about recovering production data, it is very important that one has very clear understanding of the recovery procedure, knows all the precautions and checks to be taken care of and is confident on how to proceed if any of the steps fail.
... View more
11-21-2017
05:17 AM
If you have 3 or more zookeeper servers, then you could carry out these steps on each Zookeeper one by one, in a rolling fashion, thereby keeping the zookeeper quorum intact. Otherwise, the mentioned steps are fine. Copy all files inside current dataDir (myid , version-2) to the new directory, update 'dataDir' and 'dataLogDir' (if separately configured) properties inside zoo.cfg, set the directory ownership (recursive) to zk service user and restart the zookeeper server. During the startup, zookeeper loads the latest 'snapshot' file and replays the transaction file to load the state. Follower zookeepers also syncs with the leader for the current state.
... View more
11-21-2017
04:48 AM
1 Kudo
Verify whether message.max.bytes (or max.message.bytes for the topic) is set to an appropriate value to support the large messages. And to enable the consumers to read from this topic, set fetch.message.max.bytes as well.
... View more
05-26-2017
04:55 PM
7 Kudos
Zookeeper is one of the most critical components in an HDP cluster, but it is also one that is given least importance usually when tuning cluster for performance and while troubleshooting slowness in a cluster. Here is a basic checklist for zookeeper health check that one must go through to ensure that Zookeeper is running fine. Let's keep the zookeeper happy to be able to better manage the occupants of the zoo 🙂
1. Are all the Zookeeper servers given dedicated disks for transaction log directory ('dataDir' / 'dataLogDir') ? It is very important to have fast disks to complete 'fsync' of new transactions to the log, where zookeeper writes before any update takes place and before sending a response back to the client. Slower 'fsync' for transaction log is one of the most common reasons seen in the past for slower zookeeper response. Yes, the disk space requirement is usually not very high by the zookeeper and one might wonder if its worth to dedicate a complete disk to zookeeper log directory, but its required to prevent I/O operations by other applications/processes from keeping the disk busier. Some of the common symptoms to be noticed if zookeeper finds slower writes to transactional log are:
Services such as NameNode zkfc and HBase Region servers, that uses ephemeral znodes to track its liveliness, shuts down after repeated zookeeper server connection timeouts. The zookeeper server log frequently reports errors such as: WARN [SyncThread:2:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:2 took 7050ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2. Is the zookeeper process given enough heap memory, according to the number of znodes, clients and watchers connecting the zookeepers. To arrive at the right zookeeper heap size, one has to run load tests and find the estimate on required heap size. Insufficient memory allocation for zookeepers can affect its performance once it goes through very frequent GC cycles when the heap usage reaches close to 100% of its total heap size allocation. The following four letter zookeeper commands provide many useful information about the running zookeeper instances: # echo 'stat' | nc <ZK_HOST> 2181
# echo 'mntr' | nc <ZK_HOST> 2181 In the above command output, watch for numbers against the stats such as znode count, number of watchers, number of client connections and max/avg latency among other things. In most cases a heap size between 2GB and 4GB should be a good, but as mentioned above, this depends on the kind of load on the zookeeper. In addition to the above mentioned 'four letter' commands, it is also recommended to keep an eye on the increasing heap size and the GCs, especially during the time of slowness, using tools such as: # sudo su - zookeeper ; jmap -heap <ZK_PID>
# sudo su - zookeeper ; jstat -gcutil 2000 10 <ZK_PID>
3. Are there too many zookeepers in the ensemble ? Three ZooKeeper servers is the minimum recommended size for an ensemble. And in most cases, three zookeepers are good enough too. Increased number of zookeeper servers, although gives more reliability (a 7 node ensemble can withstand loss of 3 nodes compared to the tolerance of 1 node loss in case of a 3 three node ensemble), and better read throughput when there are large number of concurrent clients connected, it can lead to slower write operations since every update/write operation is required to be committed by atleast half of the nodes in an ensemble. Some alternatives to prevent the slower writes arising due to larger ensembles are:
Use dedicated zookeeper ensemble for certain workloads in the cluster For larger ensemble, use zookeeper observers - Ref. http://zookeeper.apache.org/doc/trunk/zookeeperObservers.html (although configuration of zookeeper observer is not supported in the current Ambari version as of this writing).
4. Are the 'dataDir' / 'dataLogDir' filling up too fast ? As mentioned above, every transaction to zookeepers are written to the transaction log file. When a large number of concurrent ZK clients continuously connects and does very frequent updates, possibly due to an error condition at the client, it can lead to the transaction logs getting rolled over multiple times in a minute due to its steadily increasing size and thus resulting in a large number of Snapshot files as well. This can further cause disks running out of free space. For such issues, one has to identify and fix the client application. Review the stats from above in addition to zookeeper logs and/or the latest transaction log, to find the latest updates on the znodes using 'logFormatter' tool: # java -cp /usr/hdp/current/zookeeper-server/*:/usr/hdp/current/zookeeper-server/lib/* org.apache.zookeeper.server.LogFormatter /hadoop/zookeeper/version-2/log.xxxxx Further, the zookeeper properties - 'autopurge.snapRetainCount' and 'autopurge.purgeInterval' have to be tuned according to the required retention count and the frequency to limit the increasing number of transaction log and snapshot files.
... View more
05-02-2017
02:55 AM
1 Kudo
Although this was answered earlier over a support ticket, updating the details here for any future visitor. Phoenix in HDP 2.5 and above includes PHOENIX-1734 - where the Phoenix local indexes are co-located in the same region where the corresponding data exists, although on a different column family. In the above explain plan, the output actually tells that the local index is in use: "RANGE SCAN OVER TEST_TABLE [1,'v1-2']" --> Means its a range scan (instead of a full table scan otherwise), on the data table using the local index (type 1 = local index) and the given value of "v1-2".
... View more
04-24-2017
06:27 PM
Although an old post, updating here for any future visitors: This issue occurs in most cases due to one or more of the brokers being unavailable, if not any of the above mentioned ones. The controller node completes a topic deletion only after all the topic's partition replicas are removed from all the brokers. So validate that brokers that are currently online by checking the following znode, since sometimes although the broker processes are in running status, they may not be actually part of the cluster because of various reasons such as memory contention and continuous GC cycles: zk> ls /brokers/ids
... View more
02-08-2017
06:37 PM
A quick check would be to ensure that you've used the right parent znode (as you've used /hbase) in the connection string, defined by 'zookeeper.znode.parent' inside hbase-site.xml. It usually looks like /hbase-secure or /hbase-unsecure, based on whether or not kerberos is enabled in the cluster. And if that looks good, validate the connection string by using sqlline.py: /usr/hdp/current/phoenix-client/bin/sqlline.py 10.40.17.183,10.40.17.155,10.40.17.129:2181:/hbase
... View more
01-02-2017
11:58 AM
4 Kudos
PROBLEM
Zookeeper
transaction logs and snapshot files are created very frequently (multiple files
in every minute) and that fills up the FileSystem in a very short time.
ROOT CAUSE
One or more
application are creating or modifying the znodes too frequently, causing too
many transactions in a short duration. This leads to the creation of too many
transactional log files and snapshot files since they get rolled over after
100,000 entries by default (as defined by zookeeper property 'snapCount')
RESOLUTION
The resolution for
such cases involves reviewing the zookeeper transaction logs to find the znodes
that are updated/created most frequently using the following command on one of
the zookeeper servers:
# cd /usr/hdp/current/zookeeper-server
# java -cp zookeeper.jar:lib/* org.apache.zookeeper.server.LogFormatter /hadoop/zookeeper/version-2/logxxx
(where 'dataDir' is set to '/hadoop/zookeeper' within zookeeper configuration)
Once the frequently
updating znodes are identified using the above command, one should continue
with fixing the related application that is creating such a large number of
updates on zookeeper.
An example of such
an application that can cause this problem is Hbase, when there are very large
number of regions stuck in transition and they repeatedly fail to
become online.
... View more
Labels:
08-31-2016
06:10 AM
Looks like the hbase configuration is not picked by the phoenix client. Please check by setting either of the following environment variables: export HBASE_CONF_DIR=/etc/hbase/conf Or export HBASE_CONF_PATH=/etc/hbase/conf
... View more