question Re: tpcds data loading to hive table failure in Archives of Support Questions (Read Only)

tpcds data loading to hive table failure

sgowda — Tue, 09 Aug 2016 00:26:42 GMT

Any idea what does below error mean ?

Job Commit failed with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams; Host Details : local host is: \"ctr-e20-1468887904486-0003-01-000002.hwx.site/172.27.27.0\"; destination host is: \"ctr-e20-1468887904486-0003-01-000003.hwx.site\":8020; )'\nFAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.tez.TezTask", "stdout": "Optimizing tables and computing stats\nSTART EXECUTE etl tpcds.realschema convert\n1470664 040.599955183\nFINISH EXECUTE etl tpcds.realschema convert\n1470673710.526750959\nSTART EXECUTE etl tpcds.realschema stats\n1470673710.539773402

Re: tpcds data loading to hive table failure

sgowda — Tue, 09 Aug 2016 00:30:41 GMT

Below is the ulimit set

ulimit -a

core file size (blocks, -c) unlimited

data seg size (kbytes, -d) unlimited

scheduling priority (-e) 0

file size (blocks, -f) unlimited

pending signals (-i) 1030387

max locked memory (kbytes, -l) 64

max memory size (kbytes, -m) unlimited

open files (-n) 1048576

pipe size (512 bytes, -p) 8

POSIX message queues (bytes, -q) 819200

real-time priority (-r) 0

stack size (kbytes, -s) 8192

cpu time (seconds, -t) unlimited

max user processes (-u) 1048576

virtual memory (kbytes, -v) unlimited

file locks (-x) unlimited

Re: tpcds data loading to hive table failure

sgowda — Fri, 12 Aug 2016 18:03:30 GMT

Found the hive shell logs at /tmp/{USER}/hive.log and figured out that the hive shell was OOMing. Increased the mem size and have restarted for now.