Limit Oozie workflow or node execution time

robertoayuso · ‎10-09-2018

I am looking for a way to limit a oozie workflow or a node the execution time, to raise an error when that time is reached.

robertoayuso · ‎09-27-2018

Yarn memory is at 100% using command line too, but the task is faster.

robertoayuso · ‎09-27-2018

I am running a simple select insert sql with hive that selects from a csv table to insert in another parquet partitioned one. The task is ten times faster when run from command line using hive -f than sent from oozie using a hive action. And when run from oozie I see yarn memory at 100% but not from command line. Can someone tell me where do I have to look for the problem? Thanks

robertoayuso · ‎09-18-2018

Thanks Tarum, I will use the calc to configure it

robertoayuso · ‎09-17-2018

I am running that as admin user using spark-submit export PYTHONIOENCODING=utf8; time spark-submit -v --master yarn --deploy-mode cluster --driver-memory 8G --conf spark.network.timeout=10000000 --conf spark.executor.heartbeatInterval=1000000 --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.default.parallelism=2200 --conf spark.sql.shuffle.partitions=2200 --conf spark.driver.maxResultSize="4G" test.py When reduced yarn.scheduler.minimum-allocation-mb from 4G to 1G the error changed from 12Gb to is running beyond physical memory limits. Current usage: 10.3 GB of 9 GB physical memory used So... How are that limits calculated?

robertoayuso · ‎09-17-2018

Hi I am gettin the tipical error Container [pid=18542,containerID=container_e75_1537176390063_0001_01_000001] is running beyond physical memory limits. Current usage: 12.6 GB of 12 GB physical memory used; 19.0 GB of 25.2 GB virtual memory used. But I do not have 12GB configured anywere on Ambari, nor yarn nor mapreduce2 ¿where is that value? Thanks Roberto

Online	Offline
Last Visited	‎10-09-2018 08:56 AM

Date Registered	‎08-10-2018 11:40 AM
Last Visited	‎10-09-2018 08:56 AM
Total Messages Posted	8

Limit Oozie workflow or node execution time

Re: Simple hive select insert very slow launched f...

Simple hive select insert very slow launched from ...

Re: Yarn Container is running beyond physical memo...

Re: Yarn Container is running beyond physical memo...

Yarn Container is running beyond physical memory l...