About SK1

SK1 · ‎05-30-2016

@Sowmya Ramesh @Benjamin Leonhardi, I found the solution for this issue. Actually Before upgrade the value for "oozie.wf.rerun.failnodes" was "false". But after upgrade to HDP-2.3.4, value for "oozie.wf.rerun.failnodes" is "true",so that only failed action node in Oozie workflow instance run thus to prevent the rerun of successful action in Oozie. it is required to set following property in properties section in Process entity. <property name="oozie.wf.rerun.failnodes" value="false"/>

SK1 · ‎05-18-2016

@Ravi Mutyala: I checked and as I have give above we have this value comma separated for disk only. So not sure what is going wrong and why it is getting filled up even for a single big job.

SK1 · ‎05-18-2016

@Jitendra Yadav: We have following value for the above required properties. yarn.nodemanager.local-dirs=/grid01/hadoop/yarn/log,/grid03/hadoop/yarn/log,/grid04/hadoop/yarn/log,/grid05/hadoop/yarn/log,/grid06/hadoop/yarn/log,/grid07/hadoop/yarn/log,/grid08/hadoop/yarn/log,/grid09/hadoop/yarn/log,/grid10/hadoop/yarn/log And I could not find any value for hadoop.tmp.dir.

SK1 · ‎05-17-2016

@Jitendra Yadav: Let me explain my issue little bit more. We have total 52 worker nodes and each node has 100GB dedicated for /var/log. And there use to be a very big hive query(with 20 or more left join or right joins) which users run and during a single query run it create metadata(~100 GB) with many containers. This is the cause of issue and it trigger alerts. Once this job will fail or complete then immediately logs will clean.

SK1 · ‎05-17-2016

@Ravi Mutyala: Is there any hortonwork doc or reference where this recomanation is given ?

SK1 · ‎05-17-2016

@Jitendra Yadav: yes We have enabled 100GB for all worker nodes. And data is getting cleaned once job compete or failed. But my query is many users are running a very big query and their jobs are consuming whole 100 GB even more than that. Because of that jobs are failing.

SK1 · ‎05-17-2016

@Ravi Mutyala: Yes, I can do it but problem is whenever one of configured dirs will be 100% utilized imidiately we will keep on getting alerts email. And problem is we can not disable those alerts as we have to monitor.

SK1 · ‎05-17-2016

Thanks @Jitendra Yadav. I have done it but after that also it it getting 100% used.

SK1 · ‎05-17-2016

Team, Actually there are multiple jobs running on our servers and during running jobs are creating more staging data in local /var/log/yarn/log dir. I understand it is because of container and yarn.nodemanager.log-dirs property. We have 100GB for this location but still it is getting full, so is there anyway where we can redirect it to hdfs ? Thanks in advance.

SK1 · ‎05-13-2016

Thanks @Sowmya Ramesh, I will try and will let you know.

Online	Offline
Last Visited	‎01-15-2021 02:12 AM

Member Since	‎05-29-2017 06:13 AM
Last Visited	‎01-15-2021 02:12 AM
Posts	408
Kudos received	123

Cloudera Community

Re: select * from table is not returning any rows ...

Re: Falcon feed throwing an error during feed sche...

Re: we are getting below error/exception while exe...

Re: bin/solr status throwing an error "clusterstat...

Re: How to solve buffer memory issue in Solr

Re: How many times falcon retry for failed jobs in...

Re: Can we change yarn.nodemanager.log-dirs value ...

Re: Can we change yarn.nodemanager.log-dirs value ...

Re: Can we change yarn.nodemanager.log-dirs value ...

Re: Can we change yarn.nodemanager.log-dirs value ...

Re: Can we change yarn.nodemanager.log-dirs value ...

Re: Can we change yarn.nodemanager.log-dirs value ...

Re: Can we change yarn.nodemanager.log-dirs value ...

Can we change yarn.nodemanager.log-dirs value from...

Re: Can we use EL in falcon feed entity ?