Member since
09-25-2018
41
Posts
1
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 425 | 08-20-2019 11:07 PM | |
| 1245 | 01-06-2019 07:32 PM |
03-20-2020
01:00 PM
@StevenOD Can rebalance be done when kudu service is operational? or do we need to put kudu service on maintenence?
... View more
02-27-2020
06:38 PM
Thanks @venkatsambath & @EricL this would certainly help in my hadoop journey. Regards
... View more
02-26-2020
08:37 PM
Hello, I would like to understand the internal in / outs of Cloudera flow. How CM interacts with other components in the Hadoop stack ( HDFS / YARN/ IMPALA etc.), would like to know if there are any references material / documents which I can go through to enhance my knowledge in this area. Appreciate any information in this regard. Regards
... View more
- Tags:
- cdh
- cloudera-manager
11-20-2019
07:51 PM
@bgooley Thanks for your reply and assistance. @EricL Current heap from RM is 1GB Regards Wert
... View more
11-19-2019
11:40 PM
Hello All, Off late we have been getting concerning alerts for Reports Manager (The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.) occasionally reports manager service also goes to bad state. I believe, this has to do with Reports Manager Heap Memory, but am in a doubt hence wanted to get a confirmation before playing around with it. I have attached a snippet of the error, which I get. Also would like to know if there is any reference guide / document by which, we can calculate and periodically adjust such items. HDFS Used – 10TB / Total 90TB FSImage – 980M CM - 6.16.2 / CDH - 5.16.2 The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.
[root@MY Server cloudera-scm-headlamp]# tail -n 300 mgmt-cmf-mgmt-REPORTSMANAGER-MY Server.com.log.out
2019-11-20 19:03:43,443 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1363ms: GC pool 'ParNew' had collection(s): count=1 time=65ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1537ms
2019-11-20 19:03:48,933 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1965ms: GC pool 'ParNew' had collection(s): count=1 time=834ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1571ms
2019-11-20 19:04:10,282 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1228ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1280ms
2019-11-20 19:04:15,629 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1345ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1366ms
2019-11-20 19:04:26,295 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1321ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1343ms
2019-11-20 19:04:31,499 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1114ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms
2019-11-20 19:04:36,891 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1390ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1399ms
2019-11-20 19:04:52,016 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1138ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1158ms
2019-11-20 19:05:12,572 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1014ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1226ms
2019-11-20 19:05:28,066 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1051ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1311ms
2019-11-20 19:05:52,991 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1325ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1332ms
2019-11-20 19:06:35,411 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1104ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1395ms
2019-11-20 19:06:46,384 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1083ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1421ms
2019-11-20 19:06:51,484 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1098ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms
2019-11-20 19:07:07,179 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1249ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1273ms
2019-11-20 19:07:27,892 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1103ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1258ms
2019-11-20 19:07:33,073 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1179ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1294ms
2019-11-20 19:07:38,111 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1036ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1096ms
2019-11-20 19:07:44,198 INFO com.cloudera.cmf.BasicScmProxy: Failed request to SCM: 302
2019-11-20 19:07:45,318 INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required.
2019-11-20 19:07:45,394 INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM
2019-11-20 19:07:45,403 INFO com.cloudera.cmf.BasicScmProxy: Authenticated to SCM.
2019-11-20 19:08:07,837 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1157ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1218ms
2019-11-20 19:08:28,276 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1133ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1248ms
2019-11-20 19:08:33,105 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1227ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1251ms
2019-11-20 19:08:38,015 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1179ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1336ms
2019-11-20 19:08:40,247 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1175ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1312ms
2019-11-20 19:08:44,598 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1156ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1284ms
2019-11-20 19:08:46,743 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1094ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1312ms
2019-11-20 19:08:53,086 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1162ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1340ms
2019-11-20 19:08:55,207 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1120ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1306ms
2019-11-20 19:08:57,373 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1109ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1241ms
2019-11-20 19:08:59,443 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1069ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1203ms
2019-11-20 19:09:03,683 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1171ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1307ms
2019-11-20 19:09:05,929 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1191ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1355ms
2019-11-20 19:09:12,724 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1046ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1235ms
2019-11-20 19:09:14,848 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1123ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1292ms
2019-11-20 19:09:17,095 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1247ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1373ms
2019-11-20 19:09:20,294 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1142ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1283ms
2019-11-20 19:09:27,791 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1084ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1524ms
2019-11-20 19:09:30,897 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1050ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1358ms
2019-11-20 19:09:48,006 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1283ms
2019-11-20 19:09:50,625 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1118ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1155ms
2019-11-20 19:10:05,535 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1174ms
2019-11-20 19:10:12,452 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1164ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1268ms
2019-11-20 19:10:24,685 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1234ms
2019-11-20 19:10:34,427 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1163ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1312ms
2019-11-20 19:10:39,574 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1083ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1419ms
2019-11-20 19:10:49,650 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1198ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1413ms
2019-11-20 19:10:54,658 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1006ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1184ms
2019-11-20 19:10:59,817 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1081ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1249ms
2019-11-20 19:11:10,429 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1215ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1294ms
2019-11-20 19:11:20,762 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1194ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1237ms
2019-11-20 19:11:25,839 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1075ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1331ms
2019-11-20 19:11:31,441 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1057ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1378ms
2019-11-20 19:11:36,533 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1065ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1200ms
2019-11-20 19:11:51,468 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1064ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1352ms
2019-11-20 19:11:57,174 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1189ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1529ms
2019-11-20 19:12:02,353 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1162ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1340ms
2019-11-20 19:12:07,629 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1273ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1436ms
2019-11-20 19:12:18,074 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1127ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Wrote '/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/aggregates.tmp' (25361 bytes)
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting publishing HBase metrics
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Finished publishing HBase metrics
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampIndex: Temporarily shutting down search service
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Saving existing quotas for later merge
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Getting all files with quotas of type ALL
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) sortBy: null sortRerverse: false limit: 10000
2019-11-20 19:12:27,861 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *})
2019-11-20 19:12:27,861 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for '+(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *})' took 1ms
2019-11-20 19:12:27,862 INFO com.cloudera.headlamp.HeadlampIndex: Swapping index and FS image files
2019-11-20 19:12:27,862 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old FS Image (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/fsimage)
2019-11-20 19:12:28,121 INFO com.cloudera.headlamp.HeadlampIndex: Moving new FS Image in place.
2019-11-20 19:12:28,121 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Index (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index)
2019-11-20 19:12:30,733 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Index in place.
2019-11-20 19:12:30,733 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Aggregate Summary (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/aggregates)
2019-11-20 19:12:30,734 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Aggregate Summary in place.
2019-11-20 19:12:30,734 INFO com.cloudera.headlamp.HeadlampIndex: Starting up search service
2019-11-20 19:12:30,803 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000
2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/
2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 1ms
2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating DSQUOTA in quota cache for '/' from -1 to -1
2019-11-20 19:12:30,980 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000
2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/
2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 1ms
2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating NSQUOTA in quota cache for '/' from 9223372036854775807 to 9223372036854775807
2019-11-20 19:12:32,272 INFO com.cloudera.headlamp.HeadlampIndexManager: Finished indexing HDFS services
2019-11-20 19:38:45,654 INFO com.cloudera.cmf.BasicScmProxy: Failed request to SCM: 302
2019-11-20 19:38:46,655 INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required.
2019-11-20 19:38:46,755 INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM
2019-11-20 19:38:46,874 INFO com.cloudera.cmf.BasicScmProxy: Authenticated to SCM. Appreciate any help in this regard Regards Wert
... View more
Labels:
11-05-2019
08:14 PM
Hi @gst I really appreciate all your time & assistance this topic. Thanks a lot. Regards Wert
... View more
11-05-2019
05:15 PM
Hi Gst, Appreciate your assistance on this so far, regarding stopping Yarn Service in our cluster it would be difficult as we have Spark streaming jobs running, stopping those would require a lot of approvals etc, any other way to get the contents inside /data/yarn/nm/usercache/my.application/filecache/* deleted. Regards Wert
... View more
11-01-2019
12:56 AM
@gst Thanks for your reply, post clean-up container logs are at comfortable at 660M. @EricL path for container logs are (/data/yarn/container-logs), since I have you guys with me on this I would also like to check something about yarn filecache. I see it’s consuming around 9.3G of space any way we can have this reduced? via an automated way? Following are the Yarn settings which I, think controls filecache…. However, you guys are the experts yarn.nodemanager.localizer.cache.target-size-mb = 10GB yarn.nodemanager.localizer.cache.cleanup.interval-ms = 10 Minutes Spce Used: du -shc /data/yarn/nm/usercache/MyApplication/* 981M /data/yarn/nm/usercache/ MyApplication /appcache 9.3G /data/yarn/nm/usercache/ MyApplication /filecache
... View more
10-31-2019
10:16 AM
Hi I am running out of space on one of our DN, and found that Yarn container logs are consuming around 50GB, though there are 3 Spark jobs/applications running which can be seen under Yarn > Applications tab, but I see a lot of other application ids also listed, would like to know if these could be deleted to reclaim space (the ones which are not running) Secondly, any precautions to be taken into consideration while deleting these container logs. Any guidance would be of great help. Regards Wert
... View more
- Tags:
- disk-usage
- MySQL
- YARN
Labels:
09-10-2019
10:05 PM
@EricL When I click on File Browser from HDFS Status page, ideally it should take me to File Browser page, but here it stays at HDFS Status page. I have not come across any error in Reports Manager, have attached a snippet from RM logs. Regards Wert 2019-09-11 16:40:53,660 INFO org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer: The ratio of extra space for indexing is set with value 0.2 2019-09-11 16:40:53,660 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting index build 2019-09-11 16:43:08,426 INFO com.cloudera.cmf.BasicScmProxy: Failed request to SCM: 302 2019-09-11 16:43:09,426 INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required. 2019-09-11 16:43:09,597 INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM 2019-09-11 16:43:09,631 INFO com.cloudera.cmf.BasicScmProxy: Authenticated to SCM. 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.LuceneImageVisitor: Wrote '/var/lib/cloudera-scm-headlamp/hdfs/aggregates.tmp' (25243 bytes) 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting publishing HBase metrics 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.LuceneImageVisitor: Finished publishing HBase metrics 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.HeadlampIndex: Temporarily shutting down search service 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.HeadlampServiceImpl: Saving existing quotas for later merge 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.HeadlampServiceImpl: Getting all files with quotas of type ALL 2019-09-11 16:46:49,092 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) sortBy: null sortRerverse: false limit: 10000 2019-09-11 16:46:49,093 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) 2019-09-11 16:46:49,093 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for '+(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *})' took 1ms 2019-09-11 16:46:49,094 INFO com.cloudera.headlamp.HeadlampIndex: Swapping index and FS image files 2019-09-11 16:46:49,095 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old FS Image (/var/lib/cloudera-scm-headlamp/hdfs/fsimage) 2019-09-11 16:46:49,205 INFO com.cloudera.headlamp.HeadlampIndex: Moving new FS Image in place. 2019-09-11 16:46:49,205 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Index (/var/lib/cloudera-scm-headlamp/hdfs/index) 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Index in place. 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Aggregate Summary (/var/lib/cloudera-scm-headlamp/hdfs/aggregates) 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Aggregate Summary in place. 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Starting up search service 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/ 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 0ms 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating DSQUOTA in quota cache for '/' from -1 to -1 2019-09-11 16:46:53,165 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000 2019-09-11 16:46:53,167 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/ 2019-09-11 16:46:53,167 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 2ms 2019-09-11 16:46:53,167 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating NSQUOTA in quota cache for '/' from 9223372036854775807 to 9223372036854775807 2019-09-11 16:46:53,383 INFO com.cloudera.headlamp.HeadlampIndexManager: Finished indexing HDFS services
... View more
09-09-2019
10:46 PM
Hello,
We recently upgraded from CM 5.14.x to 5.16. I have noticed that post upgraded HDFS file browser options is not working, When I go to HDFS → File Browser option there is no change , nor there is any error, I have also tried it in IE and see the same behavior.
Any help / guidance in fixing this issue.
CM & CDH Ver - 5.16.2
Regards
Wert
... View more
Labels:
08-20-2019
11:07 PM
Hi Ben, I managed to resolve this by dropping the existing oozie database and removing oozie folder from /var/lib/, re-create oozie database and re-install oozie. Thanks wert
... View more
08-19-2019
09:57 PM
Hi Bgooley, Thanks for your reply, I get validation error on Oozie service subtab in Hue when I try to enter anything and save it. CM Version = 5.16.1 CDH Version = 5.16.2 Is there any other way this could be fixed Thanks
... View more
08-18-2019
08:49 PM
Hi, I am unable to start node manager on a node and get the attached error, any help is resolving this is much appreciated. Error starting NodeManager
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1 missing files; e.g.: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/000042.sst
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:181)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:245)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1 missing files; e.g.: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/000042.sst
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:950)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:937)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:210)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 5 more
... View more
Labels:
08-16-2019
12:10 AM
Hello All,
Due to memory issues we had to remove some components from our test cluster and one of them was oozie. Currently I am getting an error in Hue "Missing required value: Oozie Service", would appreciate any help/guide in getting this resolved.
... View more
Labels:
07-26-2019
04:25 AM
Hello All, I am trying to deploy a 6-node cluster and one of the nodes I am getting attached error while activating parcels. I had tried to copy the mentioned contents from another host where the parcels are activated but I run out of space on the host where issue persists, surprisingly both the host have the same disk space of 5GB in /opt. Requesting your assistance / guidance in resolving this issue.
... View more
Labels:
07-09-2019
11:54 PM
Hello,
We have a 3 node cluster, on one our node we see Impala Daemon in concerning state with error as (Impala Daemon Ready Check - The readiness of the Impala Daemon to process queries is not known.), while the other daemon works fine. Restarting this daemon fixes the issue momentarily, but the issue soon comes back. Attached is part of the error log.
E0710 15:36:17.214027 28015 impala-server.cc:1454] There was an error processing the impalad catalog update. Requesting a full topic update to recover: NoClassDefFoundError: org/antlr/runtime/CharStream
CAUSED BY: ClassNotFoundException: org.antlr.runtime.CharStream
I0710 15:36:18.856057 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33567
I0710 15:36:20.856436 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33568
I0710 15:36:22.856806 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33569
I0710 15:36:24.857154 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33570
I0710 15:36:25.126052 28015 impala-server.cc:1380] Received large catalog update(>100mb): TABLE:XXXXX is 140.06 MB
I0710 15:36:25.791110 28015 impala-server.cc:1380] Received large catalog update(>100mb): TABLE:XXXXX is 110.30 MB
I0710 15:36:26.857511 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33571
I0710 15:36:28.857841 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33572
I0710 15:36:30.858582 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33573
I0710 15:36:31.781077 28015 jni-util.cc:211] java.lang.NoClassDefFoundError: org/antlr/runtime/CharStream
at org.apache.impala.analysis.SlotRef.<init>(SlotRef.java:47)
at org.apache.impala.analysis.CUP$SqlParser$actions.case588(SqlParser.java:14267)
at org.apache.impala.analysis.CUP$SqlParser$actions.CUP$SqlParser$do_action(SqlParser.java:3909)
at org.apache.impala.analysis.SqlParser.do_action(SqlParser.java:1309)
at java_cup.runtime.lr_parser.parse(lr_parser.java:587)
at org.apache.impala.catalog.View.init(View.java:150)
at org.apache.impala.catalog.View.loadFromThrift(View.java:131)
at org.apache.impala.catalog.Table.fromThrift(Table.java:271)
at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:398)
at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:282)
at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:169)
at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:227)
at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:191)
Caused by: java.lang.ClassNotFoundException: org.antlr.runtime.CharStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
I0710 15:36:31.816437 28015 status.cc:125] NoClassDefFoundError: org/antlr/runtime/CharStream
CAUSED BY: ClassNotFoundException: org.antlr.runtime.CharStream
@ 0x95b1e9 impala::Status::Status()
@ 0xca2552 impala::JniUtil::GetJniExceptionMsg()
@ 0xba2b28 impala::Frontend::UpdateCatalogCache()
@ 0xbc0469 impala::ImpalaServer::CatalogUpdateCallback()
@ 0xc613e3 impala::StatestoreSubscriber::UpdateState()
@ 0xc670d3 impala::StatestoreSubscriberThriftIf::UpdateState()
@ 0x10f2ea8 impala::StatestoreSubscriberProcessor::process_UpdateState()
@ 0x10f20e4 impala::StatestoreSubscriberProcessor::dispatchCall()
@ 0x92b8bc apache::thrift::TDispatchProcessor::process()
@ 0xafbc4f apache::thrift::server::TAcceptQueueServer::Task::run()
@ 0xaf6545 impala::ThriftThread::RunRunnable()
@ 0xaf7322 boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0xd15193 impala::Thread::SuperviseThread()
@ 0xd158d4 boost::detail::thread_data<>::run()
@ 0x12919aa (unknown)
@ 0x7f44d68a9dd5 start_thread
@ 0x7f44d65d2ead __clone
Any help / guidance is much appericated
Regards
Wert
... View more
Labels:
06-25-2019
09:41 PM
I have a small POC cluster for our Dev team wherein disk space is precious. I see that a lot of data and metadata located in: data/kudu/ts/data hence was hoping if I could delete them and recover space. Since I am unsure whether deleteing this would affect application performace or render it useless, wanted some information / advise on this.
... View more
06-21-2019
03:35 AM
Hello All, Would there be any implications on Kudu applications performance etc, in case we delete Kudu TS data. Requesting any help / guidance on the same. Regards Wert
... View more
06-17-2019
08:01 PM
Hell All, My understanding of Canary files was that data is written to a file and once the test is completed, this file is removed automatically. I recently ran a Directory Usage report and I came across many of these files (/tmp/.cloudera_health_monitoring_canary_files/.canary_file_2018_12_11-23_46_17). The size shows as zero KB, and the File and Directory Count =1 do these also gets stored in the Namenode Metadata? Secondly, would these get auto deleted from hdfs? If not, would deleting them manually have any repercussions on Hadoop Cluster? Are there any best practice / guide to effectively manage these files, Kindly advice. Regards Wert
... View more
- Tags:
- cloudera manager
- HDFS
Labels:
06-02-2019
09:06 PM
1 Kudo
Hi Li, Thanks for your help Regards Wert
... View more
05-23-2019
06:23 PM
Hello, We have been facing issues with our CM Service Monitor, for the past couple of days it fails and the starts to work again, we also get the below alerts repeatedly : Concerning : The last metrics aggregation run duration is 25.4 second(s). Warning threshold: 10 second(s). Concerning : Average time spent paused was 27.9 second(s) (46.47%) per minute over the previous 5 minute(s). Warning threshold: 30.00%. There are no configuration changes done on this node, attached is the SM log file. 5:48:14.372 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2543ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2718ms
5:48:14.373 AM ERROR SafeAvroResponderServlet
Error procesing Avro request
org.mortbay.jetty.EofException
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:623)
at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeLength(SafeAvroHttpTransceiver.java:128)
at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeBuffers(SafeAvroHttpTransceiver.java:120)
at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:57)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368)
at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129)
at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161)
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714)
... 21 more
5:48:14.375 AM WARN log Committed before 500 Error processing POST request. Check the system logs for more information.
5:48:14.375 AM ERROR log
/
java.lang.IllegalStateException: Committed
at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
at org.mortbay.jetty.Response.sendError(Response.java:240)
at com.cloudera.enterprise.SafeAvroResponderServlet.logAndSuppressException(SafeAvroResponderServlet.java:69)
at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:59)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
5:48:32.030 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3051ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3431ms
5:48:36.128 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2597ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2962ms
5:48:39.909 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2781ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2863ms
5:48:42.965 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2049ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2480ms
5:48:46.620 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3155ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3204ms
5:48:49.569 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2448ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2534ms
5:48:53.360 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2790ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2807ms
Any help / guidance is appreciated
... View more
Labels:
05-22-2019
10:17 PM
Our Service Monitor has been acting weird for the last 3-4 days it keeps failing and throws random error- Pause duration error - Average time spent paused was 22.4 second(s) (37.38%) per minute over the previous 5 minute(s). Warning threshold: 30.00%. Following is the error log of service monitor : 4:56:04.002 PM ERROR SafeAvroResponderServlet Error procesing Avro request org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:623) at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeLength(SafeAvroHttpTransceiver.java:128) at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeBuffers(SafeAvroHttpTransceiver.java:120) at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:57) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161) at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714) ... 21 more 4:56:04.004 PM WARN log Committed before 500 Error processing POST request. Check the system logs for more information. 4:56:04.004 PM ERROR log / java.lang.IllegalStateException: Committed at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) at org.mortbay.jetty.Response.sendError(Response.java:240) at com.cloudera.enterprise.SafeAvroResponderServlet.logAndSuppressException(SafeAvroResponderServlet.java:69) at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:59) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 4:56:09.003 PM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3512ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3509ms I tried to increase Java Heap Size to 1.5Gb but it failed with out of memory error. Kindly assist with any recommendations / guidance to fix this issue
... View more
Labels:
05-08-2019
08:12 AM
We have been getting swap alerts on one of our worker nodes; have checked swappiness setting on this particular worker node and, vm.swappiness is set to 1 as recommended. We see these alerts come up on Yarn, Impala and Kudu. CM – 5.14 Swap Space - 8GB Physical Memory – 65 GB OS – Redhat 7.6 Any help /guidance is appreciated.
... View more
Labels:
04-17-2019
10:44 PM
I need to change my Community Nickname but the option is disabled, is there a way I can change my nickname ??
... View more
04-17-2019
10:32 PM
Hello All, I have been asked to create some Health Check report of our Hadoop Cluster for Senior Management, I see all the reports in Cloudera Manger are Chart based and as per my understanding one needs access to cloudera manager console to access these charts, is there a way that these dashboard charts can be access without the need to login to cloudera manager console or has anyone created some reports which can be emailed daily in an excel, PowerPoint format etc. Any help / pointers on this is much appreciated.
... View more
Labels:
02-20-2019
10:34 PM
Hello All, When I login to our Name Node, terminal window and run klist I get the below details, [my_id@MYHOSTNAME ~]$ klist Ticket cache: KEYRING:persistent:309057:krb_ccache_0yNoeTe Default principal: MY_ID@MY DOMAIN.COM Valid starting Expires Service principal 02/21/2019 18:31:50 02/22/2019 04:31:50 krbtgt/MY DOMAIN.COM renew until 02/28/2019 18:31:50 When I sudo -su hdfs I see that I am on hdfs as I get : "hdfs@Hostname My_ID" as soon as I try to run anyhdfs commands e.g. - hdfs dfsadmin -report, I get a long error, below is a part of it: Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 41 more ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSExcep tion: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "MY_HOSTNAME.MY DOMAIN.com/IP ADDRESS"; destination host is: " MY_HOSTNAME.MY DOMAIN.com":8021; Kindly suggest how do I fix this, kinda newbie with kerberos. Regards Wert
... View more
Labels:
02-13-2019
10:07 PM
Hi Tony, Thanks for your reply, appericate all the hep provided by you and Gzigldrum Regards Wert
... View more
02-13-2019
07:39 PM
Hi Gzigldrum, Following is the details requested, unfortunately I am unablle to run (du -hs /var/*) seems I do not have required permissions. Host Monitor Error Host Monitor Storage Directory Free Space: This role's Host Monitor Storage Directory is on a filesystem with less than 5.0 GiB of its space free. /var/lib/cloudera-host-monitor (free: 3.2 GiB (16.09%), capacity: 20.0 GiB) *********************************** Event Server Error Log Directory Free Space : This role's Log Directory is on a filesystem with less than 5.0 GiB of its space free. /var/log/cloudera-scm-eventserver (free: 4.9 GiB (35.25%), capacity: 14.0 GiB) *********************************** Alert Publisher Error Log Directory Free Space : This role's Log Directory is on a filesystem with less than 10.0 GiB of its space free. /var/log/cloudera-scm-alertpublisher (free: 5.0 GiB (35.81%), capacity: 14.0 GiB
... View more
