Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to Load data from hdfs Multi level directory into Hive table?

avatar
Contributor

An error occur when i try to load data from hdfs directory with multy level directories?

Should change some properties in the hive-site.xml?

Thanks

1 ACCEPTED SOLUTION

avatar
Contributor

is that the only properties I need to add into the hive-site.xml?

<property>

<name>mapred.input.dir.recursive</name>

<value>true</value>

</property> <property>

<name>hive.mapred.supports.subdirectories</name>

<value>true</value>

</property>

View solution in original post

5 REPLIES 5

avatar
Expert Contributor

Can you give some details?

Saying multi-level, do you try to import files from multiple folders? Can you give sample directory or directories?

avatar
Contributor

it is a directory with many folders

/log/20170721

/log/20170720

/log/20170719

avatar
Contributor

is that the only properties I need to add into the hive-site.xml?

<property>

<name>mapred.input.dir.recursive</name>

<value>true</value>

</property> <property>

<name>hive.mapred.supports.subdirectories</name>

<value>true</value>

</property>

avatar
Super Collaborator

@Helmi Khalifa

Please use below snytax to load data from hdfs to hive tables

LOAD DATA INPATH '/hdfs/path' OVERWRITE INTO TABLE TABLE_NAME;

In case if you are trying to load to a specific partition of the table

LOAD DATA INPATH '/hdfs/path' OVERWRITE INTO TABLE TABLE_NAME PARTITION (ds='2008-08-15');

avatar
Contributor

I am using the same syntax as yours but it does'nt work.

there are some missing properties in the hive-site.xml file.

I added these properties in my comment below and it works now

mapred.input.dir.recursive

hive.mapred.supports.subdirectories

Thanks