Reply
Highlighted
div
New Contributor
Posts: 1
Registered: ‎11-08-2018

Hive partitioned table loading data is slow.

Hi our environment is 

 

A single server  ubuntu 14.04 with 16GB RAM and 1TB hard disk . We installed cloudera express edition on single node and trying to ingest data from local on to hive temporary table first( which is very fast)  and then into partitioned table (Which is taking lot of time ). each file we are ingesting is in csv format and 0.6GB size. 

The number of mappers are increasing each time we run the load query. 

Please suggest a solution for this .( Whether to change number of mappers and reducers or fine tune any of the servers resources like swap memory , physical memory etc ) . Please help.  

Announcements