The scenario you describe is not relevant for HBase. If you want to build some kind of non-Kerberos based authentication mechanism for HBase, you are welcome to do so. My previous answer is accurate given what authentication mechanisms currently exist in HBase. ... View more

Without enabling Kerberos authentication for HBase, any authorization checks you make are pointless. When you don't have Kerberos authentication enabled, there is no guarantee that the end user is who they say they are. This makes authorization pointless. I would focus on getting strong authentication setup before looking more into authorization. ... View more

I would start by assuming that no service which relies on HDFS can simply use S3 directly. S3Guard can likely bridge the gap for most systems (HBase is an exception), but I cannot tell you the requirements for every service in existence. ... View more

You are running against a version of Hadoop which does not have the expected classes that HBase wants to check. I find it very unlikely that you are using Hadoop 3.1.2 on the HBase classpath. HBase relies on very specific semantics from the underlying filesystem to guarantee no data loss. This warning is telling you that HBase failed to make this automatic check and that you should investigate this to make sure that you don't experience data loss going forward. ... View more

If you inspect the Mapper log files, you should be able to find mention of an unparseable row when one is processed. You may have to increase the log level from INFO to DEBUG. Each Mapper is assigned an InputSplit which will be a contiguous group of lines from the input files that you specified (e.g. fileA lines 50 through 200). You can also use this information to work backwards. ... View more

As my previous comment says, this is a benign warning message. It does not indicate any problem with the system. If you have RegionServers crashing, your problem lies elsewhere. Would suggest you contact support to help you identify this problem if you are having problems doing so. ... View more

This is not an error that will cause any kind of problem with your system. RegionServers are known to be reporting the wrong version string. They should give the appropriate HDP-suffixed version string, but do not. ... View more

Please share the code you are using to run this benchmark. If it is using some sensitive data, please reproduce it using non-sensitive data. Without seeing how you are executing the timings, it's near impossible to give any meaningful advice. ... View more

Increase the amount of direct memory you configure for the JVM by setting -XX:MaxDirectMemorySize=... If you are not using direct memory via the HBase BlockCache or something else completely, 1-2GB should be sufficient. Fun fact: The service you're referring to is not a "Regional Server", it is "RegionServer". It is a server that hosts "region". ... View more

It's just a bug in HDP that the correct version is not being reported. There is no HBase issue to be concerned about. ... View more

Servicing a request from a client requires memory. If you allow an unlimited number of clients to connect to your ZooKeeper, you are allowing an unbounded amount of memory to be consumed as a result of client connections. This can cause OutOfMemoryErrors and a loss of service if clients (intentionally or unintentionally) overwhelm your system. ... View more

You are running the application in a way that includes the directory containing hbase-site.xml on the classpath. ... View more

Use a Scanner. You can't use Gets, as a Get requires you to know a rowKey. ... View more

After looking closer at your stacktrace, I think I see what's going on. That PhoenixDriver.java:88 is in the ShutdownHook on the JVM that Phoenix installs to close internals. I'm guessing that Spark must also install a ShutdownHook to clean up these localized resources, and the Spark hook is running before the Phoenix hook. I see your other update -- must have hit a different codepath inside of Spark which implicitly fixed the problem. Glad you got it figured out. ... View more

"gk-journal-importer-phoenix-0.0.3h.jar" is not a JAR file that HBase or Phoenix own -- this is a JAR file that your configuration is defines. You likely need to localize this JAR with your Spark application and set the proper localized path in the configuration. ... View more

HDP certifies HBase using HDFS encryption zones, not the HBase-provided encryption feature. We recommend you use HDFS encryption zones for encryption of HBase data-at-rest. ... View more

Do you have security enabled? Usually clients see this error but the server rejects the authenticated RPC. Turn on DEBUG logging for HBase and look at the RegionServer log for the hostname that you have configured. Most of the time, this is a result of a impersonation-related configuration error. The DEBUG message in the RegionServer log will inform you what the "real" user is (who is providing kerberos credentials) and who they are trying to impersonate (who the real user "says" they are). In your case here, "oozie" would be saying that it is "you" (or however you are running this application as). From this, you can amend your `hadoop.proxyuser...` configuration properties in core-site.xml, restart HBase, and try again. ... View more

You should look at the HBase Master log to understand why it has not yet become initialized on its own. Additionally, you may have to look at a JStack of the HBase Master process to help understand why it's stuck, if that's now obvious from the log itself. ... View more

Please be aware that the above may cause data loss, and may not be something you want to do on any system. You'll have to look at the rest of the Master log, and potentially a RegionServer log, to understand why the log splitting failed. Turning on DEBUG logging may help shed some more light on the situation, but would likely not be required. It can't be said what went wrong with only the provided information. ... View more

You would have to develop a custom ReplicationEndpoint in which you add this new column on every row being replicated. https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/replication/ReplicationEndpoint.html ... View more

You do not need to create a new table. You can use the existing table if you alter it to add the new column family. "rewrit[ing] the data" means that you must read all data and write it again using the new column family. Whether you read it from HBase or from its original form is of no consequence. ... View more

No, you cannot "rename" a column family in one atomic operation. To "rename", you must rewrite the data with the new column family. Then, you can simply drop the old column family. ... View more

Please include the version of HDP in every question you ask. ... View more

Sounds like you did not properly configure HBase for Kerberos authentication. The org.apache.hadoop.hbase.security.token.TokenProvider coprocessor which you should have deployed as a Region coprocessor provides the AuthenticationService. ... View more

Please start by ensuring you have the latest Phoenix ODBC driver as available on https://hortonworks.com/downloads/ Without more information from you as to what is happening server-side, I'm not sure how you expect anyone to be able to help you. Turn on logging in the ODBC driver (instructions are in the user guide for the ODBC driver on the downloads page) and look at the logging inside of the Phoenix Query Server. ... View more

First, you should investigate in the log files to understand why this region became stuck in this PENDING_CLOSE state. Then, you can close and reassign this region. HBCK should be able to do this for you, or you can use the HBase shell commands to do it by hand. ... View more

It sounds like you are using the software incorrectly. The expectation is that you run a full backup more than once, not only once. You run full backups so that the number of WALs to be tracked for incremental backups is limited. Run a new full backup every couple of weeks. ... View more