Support Questions

Find answers, ask questions, and share your expertise

How to Load data from hdfs Multi level directory into Hive table?

avatar
Contributor

An error occur when i try to load data from hdfs directory with multy level directories?

Should change some properties in the hive-site.xml?

Thanks

1 ACCEPTED SOLUTION

avatar
Contributor

is that the only properties I need to add into the hive-site.xml?

<property>

<name>mapred.input.dir.recursive</name>

<value>true</value>

</property> <property>

<name>hive.mapred.supports.subdirectories</name>

<value>true</value>

</property>

View solution in original post

5 REPLIES 5

avatar
Expert Contributor

Can you give some details?

Saying multi-level, do you try to import files from multiple folders? Can you give sample directory or directories?

avatar
Contributor

it is a directory with many folders

/log/20170721

/log/20170720

/log/20170719

avatar
Contributor

is that the only properties I need to add into the hive-site.xml?

<property>

<name>mapred.input.dir.recursive</name>

<value>true</value>

</property> <property>

<name>hive.mapred.supports.subdirectories</name>

<value>true</value>

</property>

avatar
Super Collaborator

@Helmi Khalifa

Please use below snytax to load data from hdfs to hive tables

LOAD DATA INPATH '/hdfs/path' OVERWRITE INTO TABLE TABLE_NAME;

In case if you are trying to load to a specific partition of the table

LOAD DATA INPATH '/hdfs/path' OVERWRITE INTO TABLE TABLE_NAME PARTITION (ds='2008-08-15');

avatar
Contributor

I am using the same syntax as yours but it does'nt work.

there are some missing properties in the hive-site.xml file.

I added these properties in my comment below and it works now

mapred.input.dir.recursive

hive.mapred.supports.subdirectories

Thanks