Hello, We have got our Domain renamed from ABC to XYZ.com. Since then neither Cloudera Manager UI nor any other service UI's are accessible. As per my understanding, we would need to update the /etc/hosts file of each node in the cluster to reflect the new Domain name. however it would be great if anyone could advise on the below 2 things: 1. Do we really need to get /etc/host/file updated, OR is it taken care by Cloudera 2. Where all do we need to update the hostnames with new Domain name to have a fully functional cluster back to operational status Currently if I manually enter the complete address (ip+new doamin name) I can login to Cloudera Manager UI but from Cloudera Manager when I try to open any service UI's like NN UI it fails.   Any help is much appreciated   Regards Wert ... View more

Hello,   I was using Cloudera Enterprise 5.16 (Trial) which is now downgraded to Cloudera Express, since then I am facing issues starting Reports Manager. I am using the default embedded database.   CM / CDH - 5.16.2     2020-09-06 14:14:29,706 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist. 2020-09-06 14:14:29,711 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists. 2020-09-06 14:14:29,712 INFO com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 4100. 2020-09-06 14:14:29,712 INFO com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 4100 2020-09-06 14:14:29,725 INFO com.cloudera.enterprise.ssl.SSLFactory: Using default java truststore for verification of server certificates in HTTPS communication. 2020-09-06 14:14:29,761 WARN com.cloudera.cmf.BasicScmProxy: Exception while getting fetch configDefaults hash: none java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091) at com.cloudera.cmf.BasicScmProxy.authenticate(BasicScmProxy.java:276) at com.cloudera.cmf.BasicScmProxy.fetch(BasicScmProxy.java:596) at com.cloudera.cmf.BasicScmProxy.getFragmentAndHash(BasicScmProxy.java:686) at com.cloudera.cmf.DescriptorAndFragments.newDescriptorAndFragments(DescriptorAndFragments.java:64) at com.cloudera.headlamp.HeadlampServer.<init>(HeadlampServer.java:143) at com.cloudera.headlamp.HeadlampServer.main(HeadlampServer.java:250) 2020-09-06 14:14:29,772 WARN com.cloudera.headlamp.HeadlampServer: No descriptor fetched from http://master-1.asia-southeast1-b.c.seismic-kingdom-265805.internal:7180 on after 1 tries, sleeping for 2 secs 2020-09-06 14:14:31,793 WARN com.cloudera.headlamp.HeadlampServer: No descriptor fetched from http://master-1.asia-southeast1-b.c.seismic-kingdom-265805.internal:7180 on after 2 tries, sleeping for 2 secs 2020-09-06 14:14:33,794 WARN com.cloudera.headlamp.HeadlampServer: No descriptor fetched from http://master-1.asia-southeast1-b.c.seismic-kingdom-265805.internal:7180 on after 3 tries, sleeping for 2 secs 2020-09-06 14:14:35,795 WARN com.cloudera.headlamp.HeadlampServer: No descriptor fetched from http://master-1.asia-southeast1-b.c.seismic-kingdom-265805.internal:7180 on after 4 tries, sleeping for 2 secs 2020-09-06 14:14:37,797 WARN com.cloudera.headlamp.HeadlampServer: No descriptor fetched from http://master-1.asia-southeast1-b.c.seismic-kingdom-265805.internal:7180 on after 5 tries, sleeping for 2 secs 2020-09-06 14:14:39,797 ERROR com.cloudera.headlamp.HeadlampServer: Could not fetch descriptor after 5 tries, exiting.       Appreciate any help.   ... View more

Hi, Due to space constraints, I would like to reduce the following settings in CM: 1. Maximum Number of Events in the Event Server Store-5000000 To 4000000 2. Event Server Max Log Size - 200MB To 100MB   Would like to know if this is can be done or would hamper CM services. Appreciate any assistance.   Thanks Wert   ... View more

Hello, Is there a way to check the replication factor of a particular folder in HDFS? While we have default replication set to 3 in CM for some reason files being uploaded in a particular folder shows up with replication factor of 1.   Regards Wert ... View more

Hello,   I would like to understand the internal in / outs of Cloudera flow. How CM interacts  with other components in the Hadoop stack ( HDFS / YARN/ IMPALA etc.), would like to know if there are any references material / documents which I can go through to enhance my knowledge in this area.   Appreciate any information in this regard.   Regards ... View more

Hello All, Off late we have been getting concerning alerts for Reports Manager (The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.) occasionally reports manager service also goes to bad state. I believe, this has to do with Reports Manager Heap Memory, but am in a doubt hence wanted to get a confirmation before playing around with it. I have attached a snippet of the error, which I get. Also would like to know if there is any reference guide / document by which, we can calculate and periodically adjust such items. HDFS Used – 10TB / Total 90TB FSImage – 980M CM - 6.16.2 / CDH - 5.16.2 The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%. [root@MY Server cloudera-scm-headlamp]# tail -n 300 mgmt-cmf-mgmt-REPORTSMANAGER-MY Server.com.log.out 2019-11-20 19:03:43,443 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1363ms: GC pool 'ParNew' had collection(s): count=1 time=65ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1537ms 2019-11-20 19:03:48,933 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1965ms: GC pool 'ParNew' had collection(s): count=1 time=834ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1571ms 2019-11-20 19:04:10,282 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1228ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1280ms 2019-11-20 19:04:15,629 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1345ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1366ms 2019-11-20 19:04:26,295 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1321ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1343ms 2019-11-20 19:04:31,499 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1114ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms 2019-11-20 19:04:36,891 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1390ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1399ms 2019-11-20 19:04:52,016 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1138ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1158ms 2019-11-20 19:05:12,572 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1014ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1226ms 2019-11-20 19:05:28,066 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1051ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1311ms 2019-11-20 19:05:52,991 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1325ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1332ms 2019-11-20 19:06:35,411 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1104ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1395ms 2019-11-20 19:06:46,384 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1083ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1421ms 2019-11-20 19:06:51,484 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1098ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms 2019-11-20 19:07:07,179 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1249ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1273ms 2019-11-20 19:07:27,892 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1103ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1258ms 2019-11-20 19:07:33,073 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1179ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1294ms 2019-11-20 19:07:38,111 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1036ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1096ms 2019-11-20 19:07:44,198 INFO com.cloudera.cmf.BasicScmProxy: Failed request to SCM: 302 2019-11-20 19:07:45,318 INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required. 2019-11-20 19:07:45,394 INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM 2019-11-20 19:07:45,403 INFO com.cloudera.cmf.BasicScmProxy: Authenticated to SCM. 2019-11-20 19:08:07,837 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1157ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1218ms 2019-11-20 19:08:28,276 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1133ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1248ms 2019-11-20 19:08:33,105 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1227ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1251ms 2019-11-20 19:08:38,015 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1179ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1336ms 2019-11-20 19:08:40,247 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1175ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1312ms 2019-11-20 19:08:44,598 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1156ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1284ms 2019-11-20 19:08:46,743 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1094ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1312ms 2019-11-20 19:08:53,086 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1162ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1340ms 2019-11-20 19:08:55,207 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1120ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1306ms 2019-11-20 19:08:57,373 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1109ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1241ms 2019-11-20 19:08:59,443 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1069ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1203ms 2019-11-20 19:09:03,683 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1171ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1307ms 2019-11-20 19:09:05,929 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1191ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1355ms 2019-11-20 19:09:12,724 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1046ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1235ms 2019-11-20 19:09:14,848 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1123ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1292ms 2019-11-20 19:09:17,095 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1247ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1373ms 2019-11-20 19:09:20,294 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1142ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1283ms 2019-11-20 19:09:27,791 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1084ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1524ms 2019-11-20 19:09:30,897 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1050ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1358ms 2019-11-20 19:09:48,006 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1283ms 2019-11-20 19:09:50,625 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1118ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1155ms 2019-11-20 19:10:05,535 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1174ms 2019-11-20 19:10:12,452 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1164ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1268ms 2019-11-20 19:10:24,685 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1234ms 2019-11-20 19:10:34,427 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1163ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1312ms 2019-11-20 19:10:39,574 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1083ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1419ms 2019-11-20 19:10:49,650 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1198ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1413ms 2019-11-20 19:10:54,658 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1006ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1184ms 2019-11-20 19:10:59,817 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1081ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1249ms 2019-11-20 19:11:10,429 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1215ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1294ms 2019-11-20 19:11:20,762 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1194ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1237ms 2019-11-20 19:11:25,839 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1075ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1331ms 2019-11-20 19:11:31,441 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1057ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1378ms 2019-11-20 19:11:36,533 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1065ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1200ms 2019-11-20 19:11:51,468 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1064ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1352ms 2019-11-20 19:11:57,174 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1189ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1529ms 2019-11-20 19:12:02,353 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1162ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1340ms 2019-11-20 19:12:07,629 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1273ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1436ms 2019-11-20 19:12:18,074 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1127ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Wrote '/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/aggregates.tmp' (25361 bytes) 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting publishing HBase metrics 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Finished publishing HBase metrics 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampIndex: Temporarily shutting down search service 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Saving existing quotas for later merge 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Getting all files with quotas of type ALL 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) sortBy: null sortRerverse: false limit: 10000 2019-11-20 19:12:27,861 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) 2019-11-20 19:12:27,861 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for '+(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *})' took 1ms 2019-11-20 19:12:27,862 INFO com.cloudera.headlamp.HeadlampIndex: Swapping index and FS image files 2019-11-20 19:12:27,862 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old FS Image (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/fsimage) 2019-11-20 19:12:28,121 INFO com.cloudera.headlamp.HeadlampIndex: Moving new FS Image in place. 2019-11-20 19:12:28,121 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Index (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index) 2019-11-20 19:12:30,733 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Index in place. 2019-11-20 19:12:30,733 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Aggregate Summary (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/aggregates) 2019-11-20 19:12:30,734 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Aggregate Summary in place. 2019-11-20 19:12:30,734 INFO com.cloudera.headlamp.HeadlampIndex: Starting up search service 2019-11-20 19:12:30,803 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000 2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/ 2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 1ms 2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating DSQUOTA in quota cache for '/' from -1 to -1 2019-11-20 19:12:30,980 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000 2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/ 2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 1ms 2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating NSQUOTA in quota cache for '/' from 9223372036854775807 to 9223372036854775807 2019-11-20 19:12:32,272 INFO com.cloudera.headlamp.HeadlampIndexManager: Finished indexing HDFS services 2019-11-20 19:38:45,654 INFO com.cloudera.cmf.BasicScmProxy: Failed request to SCM: 302 2019-11-20 19:38:46,655 INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required. 2019-11-20 19:38:46,755 INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM 2019-11-20 19:38:46,874 INFO com.cloudera.cmf.BasicScmProxy: Authenticated to SCM.   Appreciate  any help in this regard   Regards Wert ... View more

Hi   I am running out of space on one of our DN, and found that Yarn container logs are consuming around 50GB, though there are 3 Spark jobs/applications running which can be seen under Yarn > Applications tab, but I see a lot of other application ids also listed, would like to know if these could be deleted to reclaim space (the ones which are not running) Secondly, any precautions to be taken into consideration while deleting these container logs. Any guidance would be of great help.   Regards Wert     ... View more

Hi, I am unable to start node manager on a node and get the attached error, any help is resolving this is much appreciated.  Error starting NodeManager org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1 missing files; e.g.: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/000042.sst at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:181) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:245) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609) Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1 missing files; e.g.: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/000042.sst at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:950) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:937) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:210) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 5 more   ... View more

Hello All, Due to memory issues we had to remove some components from our test cluster and one of them was oozie. Currently I am getting an error in Hue "Missing required value: Oozie Service", would appreciate any help/guide in getting this resolved.     ... View more