Created on 10-30-2015 08:53 AM - edited 09-16-2022 02:47 AM
I am using metastore manager to copy hdfs file and create a new table in hive. But, this table file (orders) has 3 different parquet files. I am using the "create a new table from a file" utility and I am getting an error message as shown below. I want to know if this is the process to create a hive table from this kind of a file. I want to know if this is more of permission issue to upload files. I would appreciate if someone responds to this post.
Mark
"
Failed to open file '/user/hive/warehouse/orders_new': [Errno 21] Is a directory: '/user/hive/warehouse/orders_new'
None
Created 11-05-2015 05:44 AM
Thanks very much for your reply. I will try to work on it. It could be that I dont have the necessary background to do this kind of task.
Mark
Created 10-30-2015 09:21 AM
Please note there are 3 parquet files that make up the orders_new table.
Mark
Created 10-30-2015 11:29 AM
I am also including the three files in the hdfs file I am trying to upload. Please provide me instructions if I should modify permissions on these files.
Mark
[ec2-user@cloudera1 ~]$ hadoop fs -ls /user/hive/warehouse/orders_new
Found 4 items
drwxrwxrwt - admin hive 0 2015-10-30 09:58 /user/hive/warehouse/orders_new/.metadata
-rw-r--r-- 2 admin hive 188686 2015-10-30 09:58 /user/hive/warehouse/orders_new/0ea8c3e9-493c-4eb7-9e8f-2e9a26b1620a .parquet
-rw-r--r-- 2 admin hive 184808 2015-10-30 09:58 /user/hive/warehouse/orders_new/37c29bc0-7528-4fed-b794-b7532dea7e92 .parquet
-rw-r--r-- 2 admin hive 184752 2015-10-30 09:58 /user/hive/warehouse/orders_new/785ec0d6-9d45-479b-9778-8a4121783c0d .parquet
Created 10-30-2015 04:43 PM
Created 10-30-2015 06:52 PM
Hi Romain,
The link you sent is a Google search page. Can you be more specific where I can get detailed instructions to upload three different parquet files? I can create the table but I will need help with regards to loading data. I would appreciate your help.
Mark
Created 10-31-2015 07:47 AM
Hi Romain,
I tried creating the table and I could not progress very far as I doesn't know the SerDe properties. That is where I got stuck. Can you help me with this issue?
Mark
Created 11-04-2015 06:54 AM
Created 11-04-2015 07:47 AM
Hi Mathieu,
Thanks for sending that information. This is what I am looking for. I created a copy of orders hdfs file in the same directory. I wanted to create a table using Hue Metastore Manager. The table file consists of three different parquet files. Is there a utility or set script to load these parquet files into the new table? I am not a programmer and I am new to Big Data. I would appreciate if you could send me a sample script that I could use. Thanks.
Mark
Created on 11-05-2015 02:49 AM - edited 11-05-2015 02:51 AM
As far as I'm aware of, I don't know of any utility which will automatically create the "right" table for your parquet data files.
You will need to create your table manualy (and according to the structure defined by your parquet files).
In order to do that, I think you should follow theses steps :
- 1 : Puts the 3 files inside the same folder on HDFS (only theses 3 files).
- 2 : Create an external tables :
---- pointing to that directory
---- specifying that the format is "parquet"
---- specifying the field expected by your existing parquet file (you need to know the structure of your parquet files for this)
By the way : does your 3 parquet files have the same structure ? (same number of field, same field name)
If no : this will not work.
regards
Created 11-05-2015 05:44 AM
Thanks very much for your reply. I will try to work on it. It could be that I dont have the necessary background to do this kind of task.
Mark