Support Questions
Find answers, ask questions, and share your expertise

Insufficient space for shared memory file

Insufficient space for shared memory file

Contributor
Hi

We are facing some serious issue while running some hive jobs which is leading to out of memory problem. This happens when impala queries are parallely fired along with hive inserts. Hence impala queries hanging in executing mode and eating away all our resources, which leads to failure of hive inserts.

Please any help to resolve this issue !!!!!


INSERT overwrite TABLE erer_dmn PARTITION(dw_cid = 1233) select coalesce(e.employee_key, e1.employee_key) as tlg_key, t.team_key as tlg_team_key_ref, t.team_id as tlg_team_id_ref, coalesce(e.emp_nm, e1.emp_nm) as tlg_team_leader_name, coalesce(e.emp_id, e1.emp_id) as tlg_team_leader_id, from_unixtime(unix_timestamp()) as dw_insert_datetime, 0 as dw_insert_datetime_ms, t.dw_change_datetime as dw_change_datetime, 0 as dw_change_datetime_ms, 1 as dw_version, -2 as dw_uid, -2 as dw_gid, 1 as dw_batch_id from erer1_dmn_1233_v t left join latest_v_1233 e on t.team_original_id = e.emp_role1 and e.dw_cid = 1233 join latest_v_1233 e1 on e1.emp_original_id = "Not Applicable" and e.dw_cid = 1233 where t.dw_cid = 1233 1743974 [main] INFO org.apache.hadoop.hive.ql.Driver - Query ID = yarn_20160502193838_04011034-0bdb-4e6c-a68a-9501f5a7e188 1743974 [main] INFO org.apache.hadoop.hive.ql.Driver - Total jobs = 1 1743974 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=TimeToSubmit start=1462217890947 end=1462217892746 duration=1799 from=org.apache.hadoop.hive.ql.Driver> 1743974 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver> 1743974 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=task.MAPREDLOCAL.Stage-8 from=org.apache.hadoop.hive.ql.Driver> 1743976 [main] INFO org.apache.hadoop.hive.ql.Driver - Starting task [Stage-8:MAPREDLOCAL] in serial mode 1743980 [main] INFO org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask - Generating plan file file:/vol6/yarn/nm/usercache/cdhadmin/appcache/application_1462074223095_24872/container_1462074223095_24872_01_000002/tmp/yarn/00512bb1-638f-4c1a-88c2-2db8a46cdf79/hive_2016-05-02_19-38-10_947_1506231246933844074-1/-local-10005/plan.xml 1743982 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities> 1743982 [main] INFO org.apache.hadoop.hive.ql.exec.Utilities - Serializing MapredLocalWork via kryo 1743986 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=serializePlan start=1462217892754 end=1462217892758 duration=4 from=org.apache.hadoop.hive.ql.exec.Utilities> 1744072 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 1744072 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.entity.capture.input.URI does not exist 1744144 [main] INFO org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask - Executing: /vol1/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/bin/hadoop jar /vol4/yarn/nm/filecache/35316/hive-exec.jar org.apache.hadoop.hive.ql.exec.mr.ExecDriver -localtask -plan file:/vol6/yarn/nm/usercache/cdhadmin/appcache/application_1462074223095_24872/container_1462074223095_24872_01_000002/tmp/yarn/00512bb1-638f-4c1a-88c2-2db8a46cdf79/hive_2016-05-02_19-38-10_947_1506231246933844074-1/-local-10005/plan.xml -jobconffile file:/vol6/yarn/nm/usercache/cdhadmin/appcache/application_1462074223095_24872/container_1462074223095_24872_01_000002/tmp/yarn/00512bb1-638f-4c1a-88c2-2db8a46cdf79/hive_2016-05-02_19-38-10_947_1506231246933844074-1/-local-10006/jobconf.xml Java HotSpot(TM) 64-Bit Server VM warning: Insufficient space for shared memory file: /tmp/hsperfdata_yarn/23070 Try using the -Djava.io.tmpdir= option to select an alternate temp location. 1744338 [main] ERROR org.apache.hadoop.hive.ql.exec.Task - Execution failed with exit status: 1 1744338 [main] ERROR org.apache.hadoop.hive.ql.exec.Task - Obtaining error information 1744338 [main] ERROR org.apache.hadoop.hive.ql.exec.Task - Task failed! Task ID: Stage-8 Logs: 1744338 [main] ERROR org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask - Execution failed with exit status: 1 1744349 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask 1744349 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=Driver.execute start=1462217892745 end=1462217893121 duration=376 from=org.apache.hadoop.hive.ql.Driver> 1744349 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 1744391 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=releaseLocks start=1462217893121 end=1462217893163 duration=42 from=org.apache.hadoop.hive.ql.Driver> 1744405 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 1744405 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=releaseLocks start=1462217893177 end=1462217893177 duration=0 from=org.apache.hadoop.hive.ql.Driver> <<< Invocation of Hive command completed <<< Hadoop Job IDs executed by Hive: job_1462074223095_24893,job_1462074223095_24917,job_1462074223095_24934,job_1462074223095_24944