I am doing some testing on a 10 node cluster (30GB memory each) using CDH5. I uploaded about 400GB of weather data across around 500 files, totaling about 4 billion lines of data into my HDFS. I'm trying to use Hive against this and just get a record count.   CREATE EXTERNAL TABLE weather.FP_MPE_GRID_SUB_IDW_RESULT_STEP1 ( STATION_ID STRING,     WX_DATE STRING,            HIGH_TMP_F DOUBLE,     LOW_TMP_F DOUBLE, TMP_F DOUBLE,     REL_HUM_PCT DOUBLE, WIND_SPEED_MPH DOUBLE,                 HIGHEST_WIND_GUST_MPH DOUBLE,           SOLAR_RAD_AVG_GLOBAL_WM2 DOUBLE,                        WATER_EQUIV_INCH DOUBLE, STATION_ID2 STRING, ID INT, DIST double, inv_dist_wght double ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES terminated by '\n' stored as textfile location '/user/bdanalytics/weather';   hive> describe weather.FP_MPE_GRID_SUB_IDW_RESULT_STEP1; OK station_id              string                  None wx_date                 string                  None high_tmp_f              double                  None low_tmp_f               double                  None tmp_f                   double                  None rel_hum_pct             double                  None wind_speed_mph          double                  None highest_wind_gust_mph   double                  None solar_rad_avg_global_wm2        double                  None water_equiv_inch        double                  None station_id2             string                  None id                      int                     None dist                    double                  None inv_dist_wght           double                  None Time taken: 0.201 seconds, Fetched: 14 row(s) hive>   hive> select count(id) from weather.FP_MPE_GRID_SUB_IDW_RESULT_STEP1; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes):   set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers:   set hive.exec.reducers.max=<number> In order to set a constant number of reducers:   set mapred.reduce.tasks=<number> Starting Job = job_1397147177898_0001, Tracking URL = http://hadoopg1:8088/proxy/application_1397147177898_0001/ Kill Command = /opt/cloudera/parcels/CDH-5.0.0-1.cdh5.0.0.p0.47/lib/hadoop/bin/hadoop job  -kill job_1397147177898_0001 Hadoop job information for Stage-1: number of mappers: 1548; number of reducers: 1 2014-04-10 16:43:50,346 Stage-1 map = 0%,  reduce = 0% 2014-04-10 16:44:04,361 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 49.42 sec 2014-04-10 16:44:05,519 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 125.63 sec 2014-04-10 16:44:06,580 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 144.91 sec 2014-04-10 16:44:07,673 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 179.89 sec 2014-04-10 16:44:08,733 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 213.98 sec 2014-04-10 16:44:09,791 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 245.45 sec 2014-04-10 16:44:10,852 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 240.17 sec 2014-04-10 16:44:11,905 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 241.82 sec 2014-04-10 16:44:13,009 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 241.82 sec 2014-04-10 16:44:14,091 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 250.38 sec 2014-04-10 16:44:15,151 Stage-1 map = 4%,  reduce = 0%, Cumulative CPU 255.41 sec 2014-04-10 16:44:16,235 Stage-1 map = 4%,  reduce = 0%, Cumulative CPU 277.35 sec 2014-04-10 16:44:17,370 Stage-1 map = 4%,  reduce = 0%, Cumulative CPU 301.2 sec 2014-04-10 16:44:18,411 Stage-1 map = 4%,  reduce = 0%, Cumulative CPU 314.53 sec 2014-04-10 16:44:19,472 Stage-1 map = 4%,  reduce = 0%, Cumulative CPU 329.22 sec 2014-04-10 16:44:20,545 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU 356.46 sec 2014-04-10 16:44:21,604 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU 387.29 sec 2014-04-10 16:44:22,705 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU 397.92 sec 2014-04-10 16:44:23,752 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 399.31 sec 2014-04-10 16:44:24,806 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 407.96 sec 2014-04-10 16:44:25,861 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 411.52 sec 2014-04-10 16:44:26,934 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 411.52 sec 2014-04-10 16:44:28,000 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 415.28 sec 2014-04-10 16:44:29,085 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 433.5 sec 2014-04-10 16:44:30,166 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 434.77 sec 2014-04-10 16:44:31,291 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 443.1 sec 2014-04-10 16:44:32,371 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 462.61 sec 2014-04-10 16:44:33,430 Stage-1 map = 7%,  reduce = 0%, Cumulative CPU 487.98 sec 2014-04-10 16:44:34,502 Stage-1 map = 7%,  reduce = 0%, Cumulative CPU 497.86 sec 2014-04-10 16:44:35,549 Stage-1 map = 7%,  reduce = 0%, Cumulative CPU 502.88 sec 2014-04-10 16:44:36,634 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 510.7 sec 2014-04-10 16:44:37,670 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 507.2 sec 2014-04-10 16:44:38,706 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 507.2 sec MapReduce Total cumulative CPU time: 8 minutes 27 seconds 200 msec Ended Job = job_1397147177898_0001 with errors Error during job, obtaining debugging information... Examining task ID: task_1397147177898_0001_m_000016 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000005 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000033 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000025 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000068 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000002 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000034 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000097 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000089 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000127 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000107 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000098 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000030 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000118 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000109 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000125 (and more) from job job_1397147177898_0001 Examining task ID: task_1397147177898_0001_m_000087 (and more) from job job_1397147177898_0001   Task with the most failures(4): ----- Task ID:   task_1397147177898_0001_m_000030   URL:   http://hadoopg1:8088/taskdetails.jsp?jobid=job_1397147177898_0001&tipid=task_1397147177898_0001_m_000030 ----- Diagnostic Messages for this Task: Error: Java heap space Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143     FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Job 0: Map: 1548  Reduce: 1   Cumulative CPU: 507.2 sec   HDFS Read: 39564410523 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 8 minutes 27 seconds 200 msec hive>     Looking for advice on maybe specific tuning parameters working with this size of data and what may be commonly needed to let this query run. Did some Googling and tried a number of parms but nothing has yet had any change to the error or the percent I make it through before blowing up.   ... View more