129793ms is a very long time indeed. I would suspect that the kernel is blocking the process and not scheduling it for some reason. I would look into kernel messages (like dmsg, /var/log/messages) etc for clues. ... View more

You can do this: - Manually kill the server server05 by issuing kill -9. This will cause the master to recognize that the server is dead, and will re-assign the regions that were hosted there. Also you can safely restart the master in a production env. Nothing in Hbase client depends on master being available, in regular read / write paths (only DDL statements). Master is pretty light and will come up quikly, so you can restart masters safely. ... View more

Maybe you don't have the patch. Which version of HDP is this? You can try something like this: export HBASE_OPTS="$HBASE_OPTS -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.security.auth.login.config=/usr/hdp/current/hbase-client/conf/hbase_client_jaas.conf" Then run the hbase shell command. ... View more

The problem is that by default, when you launch hbase shell, it does not authenticate to zookeeper. For doing replication-related operations, you should be authenticating as the hbase server-side user. The easiest way, would be to launch the shell like this: hbase --auth-as-server shell See https://issues.apache.org/jira/browse/HBASE-15145 ... View more

How many disks you have per node? Can you please check the region server logs about compaction related logging? Everytime a compaction runs and finishes, there should be some log saying how much data is compacted and how long it took etc. There seems to be something misconfigured in your environment. ... View more

You should close the Connection / Admin / Table WHEN you are done. According to #3, Connection creation is very costly, so you would want to share the connection as much as possible, which means that you won't be "done" with the Connection until the application shuts down. Close the Connection when you know that you won't be doing any more requests. However, Table and Admin is relatively cheap, so you should open / close those per request. ... View more

Thanks. Fixed the link. ... View more

What do you mean by "When HBase Coprocessor registers this added data"? Are you doing puts against another table, or creating a table? What is your coprocessor code doing exactly? ... View more

Thanks Artem. This is still a little bit low level. You can use org.apache.hadoop.hbase.security.access.AccessControlClient as a client-facing API to grant / revoke at a higher level. ... View more

You should run HBCK with -fix argument(s). ... View more

Did you enable the backup configuration? It should be there in the HDP2-5. documentation. @mqureshi you can read this: http://hortonworks.com/blog/coming-hdp-2-5-incremental-backup-restore-apache-hbase-apache-phoenix/ ... View more

Also, deploying the region observer like this is not the correct way to go. It will make it so that ALL tables in HBase are deployed with this coprocessor (including the system tables). Since they do not have the "cf1" column, operations will fail for system table and you cannot get a cluster up and running. Please use the table property for the coproccessor for selectively enabling the coprocessor only for the tables that you need. See https://hbase.apache.org/book.html#_guidelines_for_deploying_a_coprocessor ... View more

You should use the Hive -> Phoenix connector instead of the Hive -> HBase connector since the table is a Phoenix table. More information here: https://phoenix.apache.org/hive_storage_handler.html ... View more

You should have hbase-site.xml in your classpath for the java application. Please read: https://community.hortonworks.com/articles/4091/hbase-client-application-best-practices.html ... View more

Most of the HBase features are cell-oriented rather than row-oriented unlike RDBMSs. For example the TTL is decided based on each individual cell, rather than a given row. Compactions (which is how HBase expires data) will also work for column families separately. They will never see the whole data for a given row. However, you can still implement what you want with some amount of code. As Josh suggests, you can actually implement a Filter that will only return rows that match your TTL criteria. Then you can issue deletes for those rows periodically. ... View more

You cannot use the Scan's time range filters because as you have guessed HBase is not a row-oriented engine, but a cell-oriented one. The correct approach is to write a Filter which will decide whether to include the whole row or not. For doing that, you can set the timerange in your filter, and override Filter.filterRow() and filterKeyValue() methods and keep the state within the row, and decide to include the row or not based on the Cells matching the timerange or not. You can find example filters to look at in the source code or elsewhere. ... View more

HBase client running inside the mapper has to be able to connect to HBase with kerberos authentication. Normally if this was not a MR job, the client application is usually run within a login context where the keytabs are deployed and kinit has been run. When the hbase client is run inside a task (mapper or reducer) in MR or in Spark, the client still has to authenticate. Since these tasks are run in a different node and a different process, the original TGT the application has obtained is obviously not available. TableInputFormat solves this case automatically by obtaining "delegation tokens" and distributing these delegation tokens together with the MR job. Luckily, these are already built in MR framework, so that token distribution will be taken care of. You can call: TableMapReduceUtil.initCredentials(job) before you submit your job, so that the Connection that you create in the Mapper side will automatically get the tokens and be able to connect to HBase. Notice that you should still kinit() or login with keytab in the application who submits the MR job. ... View more