Member since
07-17-2019
738
Posts
432
Kudos Received
111
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1381 | 08-06-2019 07:09 PM | |
1614 | 07-19-2019 01:57 PM | |
1846 | 02-25-2019 04:47 PM | |
2810 | 10-11-2018 02:47 PM | |
734 | 09-26-2018 02:49 PM |
09-05-2019
03:53 AM
The scenario you describe is not relevant for HBase. If you want to build some kind of non-Kerberos based authentication mechanism for HBase, you are welcome to do so. My previous answer is accurate given what authentication mechanisms currently exist in HBase.
... View more
08-08-2019
05:06 PM
Without enabling Kerberos authentication for HBase, any authorization checks you make are pointless. When you don't have Kerberos authentication enabled, there is no guarantee that the end user is who they say they are. This makes authorization pointless. I would focus on getting strong authentication setup before looking more into authorization.
... View more
08-07-2019
04:29 PM
I would start by assuming that no service which relies on HDFS can simply use S3 directly. S3Guard can likely bridge the gap for most systems (HBase is an exception), but I cannot tell you the requirements for every service in existence.
... View more
08-06-2019
07:09 PM
Blob stores do not have the same semantics as file systems. HBase relies on very specific semantics with respect to concurrency and atomic operations which most blob stores (including S3) do not provide. One example: a move of some "directory" in an S3 bucket is not atomic whereas this is atomic in HDFS. HBase will 100% not work correctly if you try to configure hbase.rootdir to use S3 via the S3A adapter in Hadoop. EMR has proprietary code in their S3 filesystem access layer, unique from S3A, which does not suffer from this issue somehow.
... View more
07-23-2019
01:42 PM
1 Kudo
You are running against a version of Hadoop which does not have the expected classes that HBase wants to check. I find it very unlikely that you are using Hadoop 3.1.2 on the HBase classpath. HBase relies on very specific semantics from the underlying filesystem to guarantee no data loss. This warning is telling you that HBase failed to make this automatic check and that you should investigate this to make sure that you don't experience data loss going forward.
... View more
07-19-2019
01:57 PM
HBase 0.94 and 0.95 are extremely old versions. You should not be using them any longer. In general, you should use the same exact version of client jars which match the HBase cluster version you are trying to interact with.
... View more
07-15-2019
01:25 PM
If you inspect the Mapper log files, you should be able to find mention of an unparseable row when one is processed. You may have to increase the log level from INFO to DEBUG. Each Mapper is assigned an InputSplit which will be a contiguous group of lines from the input files that you specified (e.g. fileA lines 50 through 200). You can also use this information to work backwards.
... View more
06-26-2019
02:51 PM
As my previous comment says, this is a benign warning message. It does not indicate any problem with the system. If you have RegionServers crashing, your problem lies elsewhere. Would suggest you contact support to help you identify this problem if you are having problems doing so.
... View more
06-18-2019
02:27 PM
This is not an error that will cause any kind of problem with your system. RegionServers are known to be reporting the wrong version string. They should give the appropriate HDP-suffixed version string, but do not.
... View more
06-06-2019
06:54 PM
Please share the code you are using to run this benchmark. If it is using some sensitive data, please reproduce it using non-sensitive data. Without seeing how you are executing the timings, it's near impossible to give any meaningful advice.
... View more
05-06-2019
10:25 PM
1 Kudo
Increase the amount of direct memory you configure for the JVM by setting -XX:MaxDirectMemorySize=... If you are not using direct memory via the HBase BlockCache or something else completely, 1-2GB should be sufficient. Fun fact: The service you're referring to is not a "Regional Server", it is "RegionServer". It is a server that hosts "region".
... View more
04-25-2019
02:41 PM
It's just a bug in HDP that the correct version is not being reported. There is no HBase issue to be concerned about.
... View more
04-11-2019
04:23 PM
Servicing a request from a client requires memory. If you allow an unlimited number of clients to connect to your ZooKeeper, you are allowing an unbounded amount of memory to be consumed as a result of client connections. This can cause OutOfMemoryErrors and a loss of service if clients (intentionally or unintentionally) overwhelm your system.
... View more
02-26-2019
03:46 PM
1 Kudo
You are running the application in a way that includes the directory containing hbase-site.xml on the classpath.
... View more
02-25-2019
05:01 PM
Use a Scanner. You can't use Gets, as a Get requires you to know a rowKey.
... View more
02-25-2019
04:47 PM
1 Kudo
hbase(main):001:0> help 'scan'
Scan a table; pass table name and optionally a dictionary of scanner
specifications. Scanner specifications may include one or more of:
TIMERANGE, FILTER, LIMIT, STARTROW, STOPROW, ROWPREFIXFILTER, TIMESTAMP,
MAXLENGTH or COLUMNS, CACHE or RAW, VERSIONS, ALL_METRICS or METRICS
...
Some examples:
...
hbase> scan 't1', {COLUMNS => 'c1', TIMERANGE => [1303668804000, 1303668904000]}
... View more
02-20-2019
02:48 PM
After looking closer at your stacktrace, I think I see what's going on. That PhoenixDriver.java:88 is in the ShutdownHook on the JVM that Phoenix installs to close internals. I'm guessing that Spark must also install a ShutdownHook to clean up these localized resources, and the Spark hook is running before the Phoenix hook. I see your other update -- must have hit a different codepath inside of Spark which implicitly fixed the problem. Glad you got it figured out.
... View more
02-19-2019
03:44 PM
"gk-journal-importer-phoenix-0.0.3h.jar" is not a JAR file that HBase or Phoenix own -- this is a JAR file that your configuration is defines. You likely need to localize this JAR with your Spark application and set the proper localized path in the configuration.
... View more
02-11-2019
03:38 PM
1 Kudo
HDP certifies HBase using HDFS encryption zones, not the HBase-provided encryption feature. We recommend you use HDFS encryption zones for encryption of HBase data-at-rest.
... View more
02-08-2019
03:25 PM
1 Kudo
Do you have security enabled? Usually clients see this error but the server rejects the authenticated RPC. Turn on DEBUG logging for HBase and look at the RegionServer log for the hostname that you have configured. Most of the time, this is a result of a impersonation-related configuration error. The DEBUG message in the RegionServer log will inform you what the "real" user is (who is providing kerberos credentials) and who they are trying to impersonate (who the real user "says" they are). In your case here, "oozie" would be saying that it is "you" (or however you are running this application as). From this, you can amend your `hadoop.proxyuser...` configuration properties in core-site.xml, restart HBase, and try again.
... View more
01-29-2019
02:39 PM
You should look at the HBase Master log to understand why it has not yet become initialized on its own. Additionally, you may have to look at a JStack of the HBase Master process to help understand why it's stuck, if that's now obvious from the log itself.
... View more
01-29-2019
02:37 PM
Please be aware that the above may cause data loss, and may not be something you want to do on any system. You'll have to look at the rest of the Master log, and potentially a RegionServer log, to understand why the log splitting failed. Turning on DEBUG logging may help shed some more light on the situation, but would likely not be required. It can't be said what went wrong with only the provided information.
... View more
01-07-2019
04:28 PM
You would have to develop a custom ReplicationEndpoint in which you add this new column on every row being replicated. https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/replication/ReplicationEndpoint.html
... View more
12-10-2018
03:11 AM
1 Kudo
You do not need to create a new table. You can use the existing table if you alter it to add the new column family. "rewrit[ing] the data" means that you must read all data and write it again using the new column family. Whether you read it from HBase or from its original form is of no consequence.
... View more
12-06-2018
11:33 PM
1 Kudo
No, you cannot "rename" a column family in one atomic operation. To "rename", you must rewrite the data with the new column family. Then, you can simply drop the old column family.
... View more
12-04-2018
04:46 PM
Please include the version of HDP in every question you ask.
... View more
11-29-2018
03:32 PM
Sounds like you did not properly configure HBase for Kerberos authentication. The org.apache.hadoop.hbase.security.token.TokenProvider coprocessor which you should have deployed as a Region coprocessor provides the AuthenticationService.
... View more
11-29-2018
03:30 PM
Please start by ensuring you have the latest Phoenix ODBC driver as available on https://hortonworks.com/downloads/ Without more information from you as to what is happening server-side, I'm not sure how you expect anyone to be able to help you. Turn on logging in the ODBC driver (instructions are in the user guide for the ODBC driver on the downloads page) and look at the logging inside of the Phoenix Query Server.
... View more
11-29-2018
03:27 PM
First, you should investigate in the log files to understand why this region became stuck in this PENDING_CLOSE state. Then, you can close and reassign this region. HBCK should be able to do this for you, or you can use the HBase shell commands to do it by hand.
... View more
11-12-2018
04:34 PM
It sounds like you are using the software incorrectly. The expectation is that you run a full backup more than once, not only once. You run full backups so that the number of WALs to be tracked for incremental backups is limited. Run a new full backup every couple of weeks.
... View more