question Re: Problem with create new table in Hue in Support Questions

Problem with create new table in Hue

mark10120 — Fri, 16 Sep 2022 09:47:07 GMT

I am using metastore manager to copy hdfs file and create a new table in hive. But, this table file (orders) has 3 different parquet files. I am using the "create a new table from a file" utility and I am getting an error message as shown below. I want to know if this is the process to create a hive table from this kind of a file. I want to know if this is more of permission issue to upload files. I would appreciate if someone responds to this post.

Mark

Failed to open file '/user/hive/warehouse/orders_new': [Errno 21] Is a directory: '/user/hive/warehouse/orders_new'

None

Re: Problem with create new table in Hue

mark10120 — Fri, 30 Oct 2015 16:21:11 GMT

Please note there are 3 parquet files that make up the orders_new table.

Mark

Re: Problem with create new table in Hue

mark10120 — Fri, 30 Oct 2015 18:29:16 GMT

I am also including the three files in the hdfs file I am trying to upload. Please provide me instructions if I should modify permissions on these files.

Mark

[ec2-user@cloudera1 ~]$ hadoop fs -ls /user/hive/warehouse/orders_new
Found 4 items
drwxrwxrwt   - admin hive          0 2015-10-30 09:58 /user/hive/warehouse/orders_new/.metadata
-rw-r--r--   2 admin hive     188686 2015-10-30 09:58 /user/hive/warehouse/orders_new/0ea8c3e9-493c-4eb7-9e8f-2e9a26b1620a    .parquet
-rw-r--r--   2 admin hive     184808 2015-10-30 09:58 /user/hive/warehouse/orders_new/37c29bc0-7528-4fed-b794-b7532dea7e92    .parquet
-rw-r--r--   2 admin hive     184752 2015-10-30 09:58 /user/hive/warehouse/orders_new/785ec0d6-9d45-479b-9778-8a4121783c0d    .parquet

Re: Problem with create new table in Hue

Romainr — Fri, 30 Oct 2015 23:43:56 GMT

The create table metastore wizard does not currently support parquet files
in input.

You should be able to manually create an external table based on those
files:
https://www.google.com/search?q=hive+create+table+from+parquet+files&oq=hive+create+table+from+parquet+files&aqs=chrome..69i57.7586j0j7&sourceid=chrome&es_sm=93&ie=UTF-8

Re: Problem with create new table in Hue

mark10120 — Sat, 31 Oct 2015 01:52:58 GMT

Hi Romain,

The link you sent is a Google search page. Can you be more specific where I can get detailed instructions to upload three different parquet files? I can create the table but I will need help with regards to loading data. I would appreciate your help.

Mark

Re: Problem with create new table in Hue

mark10120 — Sat, 31 Oct 2015 14:47:59 GMT

Hi Romain,

I tried creating the table and I could not progress very far as I doesn't know the SerDe properties. That is where I got stuck. Can you help me with this issue?

Mark

Re: Problem with create new table in Hue

mathieu.d — Wed, 04 Nov 2015 14:54:03 GMT

Hi,

Here is a link that show you the HIVE Syntax for creating a table stored as parquet :
https://cwiki.apache.org/confluence/display/Hive/Parquet#Parquet-Hive0.13andlater

Re: Problem with create new table in Hue

mark10120 — Wed, 04 Nov 2015 15:47:59 GMT

Hi Mathieu,

Thanks for sending that information. This is what I am looking for. I created a copy of orders hdfs file in the same directory. I wanted to create a table using Hue Metastore Manager. The table file consists of three different parquet files. Is there a utility or set script to load these parquet files into the new table? I am not a programmer and I am new to Big Data. I would appreciate if you could send me a sample script that I could use. Thanks.

Mark

Re: Problem with create new table in Hue

mathieu.d — Thu, 05 Nov 2015 10:51:01 GMT

As far as I'm aware of, I don't know of any utility which will automatically create the "right" table for your parquet data files.

You will need to create your table manualy (and according to the structure defined by your parquet files).

In order to do that, I think you should follow theses steps :

- 1 : Puts the 3 files inside the same folder on HDFS (only theses 3 files).

- 2 : Create an external tables :

---- pointing to that directory

---- specifying that the format is "parquet"

---- specifying the field expected by your existing parquet file (you need to know the structure of your parquet files for this)

By the way : does your 3 parquet files have the same structure ? (same number of field, same field name)

If no : this will not work.

regards

Re: Problem with create new table in Hue

mark10120 — Thu, 05 Nov 2015 13:44:46 GMT

Thanks very much for your reply. I will try to work on it. It could be that I dont have the necessary background to do this kind of task.

Mark