CVE-2018-8012 vulnerability is reported here, https://nvd.nist.gov/vuln/detail/CVE-2018-8012   o Affected Versions: Apache ZooKeeper before 3.4.10, and 3.5.0-alpha through 3.5.3-beta. o Mitigations: - The Apache JIRA (ZOOKEEPER-1045) is fixed @ Zookeeper 3.4.10. - Currently, the Fix for ZOOKEEPER-1045 (Zookeeper 3.4.10) isn't planned for any HDP release as the vulnerability is less of a vulnerability and more of a lack of a feature - Zookeeper version 3.4.6 is present from HDP 2.3.0 to HDP 2.6.x   o Workaround: - The workaround for this is to implement firewall rules to disallow connections from ports other than 2181. - Customer may try to move the ZooKeeper Election and Follower ports into the privileged port range, but this would require additional logic to start ZK as root and then drop privileges which does not exist as of now. - As such, the feasible workaround for this CVE is to configure firewall to allow communication [for Leader Election] within cluster. A possible way is: (01) Keep the ACL of port 2181 unchanged (Any Server is allowed to access), (02) Forbid all access to both port 2888 and 3888 as defined in the zookeeper config file zoo.cfg except for zookeeper servers (To ensure they can communicate with each other). o Will this issue be addressed in the future versions? As the issue has been addressed with the workaround, We have no immediate plans to backport any major features into ZooKeeper and Product Management team working on getting this added on roadmap. This vulnerability had been reported earlier and the Jira for same had also been raised to keep a track of it. So far, the only solution for this CVE is to configure firewall to allow communication [for leader election] within cluster. ... View more

You can use trace command for tracing requests from HBase Shell. trace 'start' command turns on tracing and trace 'stop' command turns off tracing. hbase(main):001:0> trace 'start' hbase(main):002:0> put 'test', 'row1', 'f:', 'val1' # traced commands hbase(main):003:0> trace 'stop' trace 'start' and trace 'stop' always returns boolean value representing if or not there is ongoing tracing. As a result, trace 'stop' returns false on success. trace 'status' just returns if or not tracing is turned on. hbase(main):001:0> trace 'start' => true hbase(main):002:0> trace 'status' => true hbase(main):003:0> trace 'stop' => false hbase(main):004:0> trace 'status' => false REFERECE: https://hbase.apache.org/book.html ... View more

HBaseFsck (hbck) is a tool for checking for region consistency and table integrity problems and repairing a corrupted HBase. It works in two basic modes — a read-only inconsistency identifying mode and a multi-phase read-write repair mode. To check to see if your HBase cluster has corruptions, run hbck against your HBase cluster: $ ./bin/hbase hbck At the end of the commands output it prints OK or tells you the number of INCONSISTENCIES present. You may also want to run hbck a few times because some inconsistencies can be transient (e.g. cluster is starting up or a region is splitting). Operationally you may want to run hbck regularly and setup alert if it repeatedly reports inconsistencies . A run of hbck will report a list of inconsistencies along with a brief description of the regions and tables affected. The using the -details option will report more details including a representative listing of all the splits present in all the tables. $ ./bin/hbase hbck -details If you just want to know if some tables are corrupted, you can limit hbck to identify inconsistencies in only specific tables. For example the following command would only attempt to check table TableFoo and TableBar. The benefit is that hbck will run in less time. $ ./bin/hbase hbck TableFoo TableBar Best Practices: ● Redirect output to a file hbase hbck >>/tmp/hbck.txt ● Larger clusters run table specific hbck fixes hbase hbck –fixMeta mytable ● Avoid running hbck with –repair flag. ... View more

To avoid full table scan while running hbase copytable job make use of --startrow and --stoprow parameters instead of --starttime and --endtime. Example: hbase org.apache.hadoop.hbase.mapreduce.CopyTable --startrow r1 --stoprow r12 --peer.adr=zk1,zk2,zk3:2181:/hbase-unsecure TestTable hbase -Dhbase.client.scanner.caching=1 -Dmapred.map.tasks.speculative.execution=false org.apache.hadoop.hbase.mapreduce.CopyTable --peer.adr=phbaiskdc2000.phx.qa.xyz.com:2181:/hbase-unsecure --new.name=REVIEW --startrow="BnTQA.AR.8aa1b29c-4613-4c40-924b-2294759854c4" --stoprow="BnTQA.AR.8aa1b29c-4613-4c40-924b-2294759854c4" REVIEW Reference: https://hbase.apache.org/book.html ... View more

PROBLEM: Security scan reporting clickjacking vulnerabilities for HBase ports (16010 and 16030) and AMS ports (61330 and 61310). ROOT CAUSE: HBASE-14148 RESOLUTION: Upgrade to the HDP 2.5.X ... View more

PROBLEM: HBase Master failing due to following exception. 2017-03-30 20:51:13,130 ERROR [<HOST NAME>,33733,1490218864063_ChoreService_3] normalizer.RegionNormalizerChore: Caught error java.lang.NullPointerException at org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer.getRegionSize(SimpleRegionNormalizer.java:172) at org.apache.hadoop.hbase.master.normalizer.SimpleRegionNormalizer.computePlanForTable(SimpleRegionNormalizer.java:99) at org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1417) at org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:49) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ROOT CASUE: https://issues.apache.org/jira/browse/HBASE-15933 WORKAROUND: Move Splitting WAL files from hbase.root.dir (by default it set to /apps/hbase/data/WALs) to some tmp directory. ... View more

Append following properties in /usr/hdp/current/phoenix-client/bin/log4j.properties and change log4j.threshold=TRACE log4j.logger.org.apache.phoenix=TRACE log4j.logger.org.apache.hadoop.hbase.ipc=TRACE log4j.logger.org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel=DEBUG ... View more

Periodically, and when there are no regions in transition, a load balancer will run and move regions around to balance the cluster’s load. See Balancer for configuring this property. The Load Balancer ensures that the region replicas are not co-hosted in the same region servers and also in the same rack (if possible). The HDFS balancer attempts to spread HDFS blocks evenly among DataNodes. HBase relies on compactions to restore locality after a region split or failure. These two types of balancing do not work well together. In the past, the generally accepted advice was to turn off the HDFS load balancer and rely on the HBase balancer, since the HDFS balancer would degrade locality. HDFS-6133 provides the ability to exclude favored-nodes (pinned) blocks from the HDFS load balancer, by setting the dfs.datanode.block-pinning.enabled property to true in the HDFS service configuration. HBase can be enabled to use the HDFS favored-nodes feature by switching the HBase balancer class (conf: hbase.master.loadbalancer.class ) to org.apache.hadoop.hbase.favored.FavoredNodeLoadBalancer which is documented here. HDFS-6133 is available in HDFS 2.7.0 and higher, but HBase does not support running on HDFS 2.7.0, so you must be using HDFS 2.7.1 or higher to use this feature with HBase. Reference: https://hbase.apache.org/book.html ... View more