Member since
09-24-2015
816
Posts
488
Kudos Received
189
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3173 | 12-25-2018 10:42 PM | |
| 14195 | 10-09-2018 03:52 AM | |
| 4764 | 02-23-2018 11:46 PM | |
| 2481 | 09-02-2017 01:49 AM | |
| 2914 | 06-21-2017 12:06 AM |
05-14-2016
02:37 AM
The error says that you have declared m columns in your Hive table, and n cf:column mappings in your hbase.columns.mapping string and that m!=n. Can you check which of these numbers is wrong? As I'm sure you know, you can declare Hive columns in a free text block using many lines and spaces but the hbase.columns.mapping string is very restrictive, it allows no "beautifying" spaces, only the key and cf:column parts separated by commas. Hive HBase Integration page doesn't mention any limit on the length of the string, though it admits that the string is somewhat cumbersome and restrictive. Alternatively, you can try to map columns from a HBase table column family "cf" using the ":key,cf:" string. They will map into a Hive map<...> element composed map(column,value) for each key, see an example here. You can then keep on working with the map, or explode it using Hive's explode(map) UDF.
... View more
05-14-2016
01:20 AM
1 Kudo
You can upgrade to any version, including Pivotal, by taking the so-called "data migration" approach: setup your new cluster, and transfer data from the old one to the new one. You can copy data directly from one cluster to another using distcp for HDFS files and CopyTable for HBase tables. You can copy Hive tables using export/import of tables. Obviously it's not practical for a large amount of data. For "in-place migration", upgrading binaries, keeping data as-is, I agree with Artem and Timothy that it's the best to engage Support, otherwise you can run into a lot of troubles, and even damage or lose your data.
... View more
05-14-2016
12:25 AM
The mappings you cited are the ones in Ambari. Based on settings of respective services Ambari will replace {{...}} parameters with real hostnames, ports etc. You can inspect the real values used by Knox by checking files under /etc/knox/conf, like gateway-site.xml and topologies/*.xml files. If some settings are not right you can try to fix them, for example by replace mappings in Ambari with specific values. For example, you can replace {{rm_host}}:{{jt_rpc_port}} with my-rm-fqdn.hadoop.com:{{jt_rpc_port}}, and you can replace the port as well.
... View more
05-13-2016
12:39 PM
Hi @mark doutre, I checked your files (2 days ago, but couldn't post sooner), and my conclusion is that Hive cannot handle Avro files without schema. From AvroSerDe page you can see which Avro versions are supported (1.5.3 to 1.7.5), and Avro specs say: Avro data is always serialized with its schema. Files that
store Avro data should always also include the schema for that
data in the same file. And it has been so from version 1. So, it's very clear that "standard" Avro files must include schema and Hive supports only such files. With schema-less files you are on your own, you would have to read "value" from HBase and apply your schema to read the data and store such records in Hive. You can also include schema, which will work, but you will waste some space in HBase by storing the same schema in each record. Hope this helps.
... View more
05-13-2016
04:16 AM
Well, not sure, if curl from cli works, it should work. Can you try to restart ambari-server and all ambari-agents.
... View more
05-13-2016
01:08 AM
Can you try to run just this command, make sure you get HTTP status 200 and no errors: curl -iv 'http://xxxx:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs If /ats/done doesn't exist, replace it with /tmp in the command. Last time I saw this issue the reason was http proxy, so make sure you have no proxy to servers in the cluster.
... View more
05-11-2016
02:03 PM
To repeat, 1 is the default, see defaults here, but just to be sure you can go ahead and set it directly in hdfs-site, and restart HDFS and dependent services.
... View more
05-11-2016
12:25 PM
After you restart HDFS and Yarn, Ambari will show you which other services to restart, like MapRed, Hive and Oozie.
... View more
05-10-2016
03:26 PM
Okay, then set "dfs.namenode.safemode.threshold-pct=0.999f" in Ambari, that's all you need to do.
... View more
05-10-2016
03:03 PM
Okay, please uplaod your files somewhere (one of your exisiting questions, or a new one), and I'll try to read them with Hive.
... View more