Member since
06-11-2019
15
Posts
0
Kudos Received
0
Solutions
07-02-2020
10:49 PM
The command of the process- [testuser@dhubm02 ~]$ ps aux --sort=-%mem | head |grep "yarn-ats" yarn-ats 26561 0.7 4.9 6129012 1577496 ? Sl 05:03 3:49 /usr/lib/jvm/java-1.8.0-openjdk/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Dhdp.version=3.1.0.0-78 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop-yarn/embedded-yarn-ats-hbase/hs_err_pid%p.log -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/hadoop-yarn/embedded-yarn-ats-hbase/gc.log-202007010503 -XX:CMSInitiatingOccupancyFraction=70 -XX:ReservedCodeCacheSize=256m -Xms3276m -Xmx3276m -Djava.security.auth.login.config=/usr/hdp/3.1.0.0-78/hadoop/conf/embedded-yarn-ats-hbase/yarn_hbase_regionserver_jaas.conf -Dhbase.log.dir=/var/log/hadoop-yarn/embedded-yarn-ats-hbase -Dhbase.log.file=hbase-yarn-ats-regionserver-dhubm02.avios.dta.iag-gdp.net.log -Dhbase.home.dir=/usr/hdp/3.1.0.0-78/hbase/bin/.. -Dhbase.id.str=yarn-ats -Dhbase.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/3.1.0.0-78/hadoop/lib/native/Linux-amd64-64:/usr/hdp/3.1.0.0-78/hadoop/lib/native/Linux-amd64-64:/usr/hdp/3.1.0.0-78/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.regionserver.HRegionServer start Also noticed the below entry in - /var/log/hadoop-yarn/embedded-yarn-ats-hbase/gc.log-202007010503 2020-07-03T05:15:18.832+0000: 173484.604: [GC (Allocation Failure) 2020-07-03T05:15:18.833+0000: 173484.605: [ParNew: 901223K->19101K(996800K), 0.0230869 secs] 1054651K->173168K(3243904K), 0.0236997 secs] [Times: user=0.25 sys=0.00, real=0.03 secs] 2020-07-03T05:24:08.491+0000: 174014.262: [GC (Allocation Failure) 2020-07-03T05:24:08.491+0000: 174014.262: [ParNew: 905181K->15339K(996800K), 0.0224086 secs] 1059248K->170059K(3243904K), 0.0225202 secs] [Times: user=0.23 sys=0.00, real=0.03 secs] 2020-07-03T05:32:18.675+0000: 174504.447: [GC (Allocation Failure) 2020-07-03T05:32:18.676+0000: 174504.448: [ParNew: 901419K->16600K(996800K), 0.0228688 secs] 1056139K->171858K(3243904K), 0.0233860 secs] [Times: user=0.24 sys=0.00, real=0.02 secs] 2020-07-03T05:39:42.857+0000: 174948.629: [GC (Allocation Failure) 2020-07-03T05:39:42.857+0000: 174948.629: [ParNew: 902680K->17498K(996800K), 0.0220587 secs] 1057938K->173472K(3243904K), 0.0221793 secs] [Times: user=0.22 sys=0.00, real=0.02 secs]
... View more
07-01-2020
10:27 PM
It seems that the timeline server creates a process which is run by the yarn-ats user. The memory consumed by this process increases continuously till no free memory is available on the machine after which the service gets stopped and the process run by the yarn-ats user gets killed releasing the memory used by the process. On starting the timeline service, a new process gets created and the cycle gets repeated. It seems that the timeline server creates a process which is run by the yarn-ats user. The memory consumed by this process increases continuously till no free memory is available on the machine after which the service gets stopped and the process run by the yarn-ats user gets killed releasing the memory used by the process. Is there any configuration to prevent the process from consuming more memory? The Total memory available is 30GB and the process was using around 19-20% of the total memory.
... View more
Labels:
10-09-2019
06:24 AM
Hi, Did you get a solution for this? Does knox support mfa from third party vendors?
... View more
10-01-2019
11:10 AM
Even I am facing the same issue. As soon as I enable the controller service its creating livy sessions. I havent started any processors.
... View more
- Tags:
- ven i
08-21-2019
11:01 PM
While logging in using the AD user credentials into the YARN UI and HDFS UI it shows logged in as knox user. Because of this the logs are also not accessible. Am I supposed to make some changes in the configuration?
... View more
Labels:
- Labels:
-
Apache YARN
07-16-2019
09:54 AM
Lets say that the cluster and the AD is created in accountA and the s3 bucket is in accountB. I have used AWS microsoft active directory
... View more
06-11-2019
06:25 AM
After kerberization the spark jobs run as an admin user are not getting displayed on the yarn UI. The jobs are getting displayed on the spark history UI. You can see all the jobs by executing the yarn application -list -appStates ALL command. Is it because of some permission issue? Should i make some changes in the configurations? Attaching the screenshots of my spark ui before and after kerberization.
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN