- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How to Load data from hdfs Multi level directory into Hive table?
- Labels:
-
Apache Hive
Created ‎07-21-2017 08:49 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
An error occur when i try to load data from hdfs directory with multy level directories?
Should change some properties in the hive-site.xml?
Thanks
Created ‎07-21-2017 12:22 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
is that the only properties I need to add into the hive-site.xml?
<property>
<name>mapred.input.dir.recursive</name>
<value>true</value>
</property> <property>
<name>hive.mapred.supports.subdirectories</name>
<value>true</value>
</property>
Created ‎07-21-2017 09:02 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you give some details?
Saying multi-level, do you try to import files from multiple folders? Can you give sample directory or directories?
Created ‎07-21-2017 11:48 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
it is a directory with many folders
/log/20170721
/log/20170720
/log/20170719
Created ‎07-21-2017 12:22 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
is that the only properties I need to add into the hive-site.xml?
<property>
<name>mapred.input.dir.recursive</name>
<value>true</value>
</property> <property>
<name>hive.mapred.supports.subdirectories</name>
<value>true</value>
</property>
Created ‎07-21-2017 06:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Helmi Khalifa
Please use below snytax to load data from hdfs to hive tables
LOAD DATA INPATH '/hdfs/path' OVERWRITE INTO TABLE TABLE_NAME;
In case if you are trying to load to a specific partition of the table
LOAD DATA INPATH '/hdfs/path' OVERWRITE INTO TABLE TABLE_NAME PARTITION (ds='2008-08-15');
Created ‎07-22-2017 09:35 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am using the same syntax as yours but it does'nt work.
there are some missing properties in the hive-site.xml file.
I added these properties in my comment below and it works now
mapred.input.dir.recursive
hive.mapred.supports.subdirectories
Thanks
