Reports Manager Pause Duration Issue

[#1]

Hello All,

Off late we have been getting concerning alerts for Reports Manager (The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.) occasionally reports manager service also goes to bad state.

I believe, this has to do with Reports Manager Heap Memory, but am in a doubt hence wanted to get a confirmation before playing around with it. I have attached a snippet of the error, which I get.

Also would like to know if there is any reference guide / document by which, we can calculate and periodically adjust such items.

HDFS Used – 10TB / Total 90TB
FSImage – 980M
CM - 6.16.2 / CDH - 5.16.2

The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.

[root@MY Server cloudera-scm-headlamp]# tail -n 300 mgmt-cmf-mgmt-REPORTSMANAGER-MY Server.com.log.out
2019-11-20 19:03:43,443 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1363ms: GC pool 'ParNew' had collection(s): count=1 time=65ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1537ms
2019-11-20 19:03:48,933 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1965ms: GC pool 'ParNew' had collection(s): count=1 time=834ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1571ms
2019-11-20 19:04:10,282 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1228ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1280ms
2019-11-20 19:04:15,629 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1345ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1366ms
2019-11-20 19:04:26,295 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1321ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1343ms
2019-11-20 19:04:31,499 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1114ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms
2019-11-20 19:04:36,891 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1390ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1399ms
2019-11-20 19:04:52,016 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1138ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1158ms
2019-11-20 19:05:12,572 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1014ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1226ms
2019-11-20 19:05:28,066 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1051ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1311ms
2019-11-20 19:05:52,991 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1325ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1332ms
2019-11-20 19:06:35,411 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1104ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1395ms
2019-11-20 19:06:46,384 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1083ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1421ms
2019-11-20 19:06:51,484 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1098ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms
2019-11-20 19:07:07,179 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1249ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1273ms
2019-11-20 19:07:27,892 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1103ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1258ms
2019-11-20 19:07:33,073 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1179ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1294ms
2019-11-20 19:07:38,111 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1036ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1096ms
2019-11-20 19:07:44,198 INFO com.cloudera.cmf.BasicScmProxy: Failed request to SCM: 302
2019-11-20 19:07:45,318 INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required.
2019-11-20 19:07:45,394 INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM
2019-11-20 19:07:45,403 INFO com.cloudera.cmf.BasicScmProxy: Authenticated to SCM.
2019-11-20 19:08:07,837 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1157ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1218ms
2019-11-20 19:08:28,276 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1133ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1248ms
2019-11-20 19:08:33,105 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1227ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1251ms
2019-11-20 19:08:38,015 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1179ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1336ms
2019-11-20 19:08:40,247 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1175ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1312ms
2019-11-20 19:08:44,598 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1156ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1284ms
2019-11-20 19:08:46,743 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1094ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1312ms
2019-11-20 19:08:53,086 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1162ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1340ms
2019-11-20 19:08:55,207 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1120ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1306ms
2019-11-20 19:08:57,373 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1109ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1241ms
2019-11-20 19:08:59,443 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1069ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1203ms
2019-11-20 19:09:03,683 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1171ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1307ms
2019-11-20 19:09:05,929 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1191ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1355ms
2019-11-20 19:09:12,724 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1046ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1235ms
2019-11-20 19:09:14,848 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1123ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1292ms
2019-11-20 19:09:17,095 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1247ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1373ms
2019-11-20 19:09:20,294 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1142ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1283ms
2019-11-20 19:09:27,791 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1084ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1524ms
2019-11-20 19:09:30,897 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1050ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1358ms
2019-11-20 19:09:48,006 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1283ms
2019-11-20 19:09:50,625 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1118ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1155ms
2019-11-20 19:10:05,535 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1174ms
2019-11-20 19:10:12,452 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1164ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1268ms
2019-11-20 19:10:24,685 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1234ms
2019-11-20 19:10:34,427 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1163ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1312ms
2019-11-20 19:10:39,574 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1083ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1419ms
2019-11-20 19:10:49,650 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1198ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1413ms
2019-11-20 19:10:54,658 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1006ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1184ms
2019-11-20 19:10:59,817 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1081ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1249ms
2019-11-20 19:11:10,429 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1215ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1294ms
2019-11-20 19:11:20,762 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1194ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1237ms
2019-11-20 19:11:25,839 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1075ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1331ms
2019-11-20 19:11:31,441 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1057ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1378ms
2019-11-20 19:11:36,533 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1065ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1200ms
2019-11-20 19:11:51,468 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1064ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1352ms
2019-11-20 19:11:57,174 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1189ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1529ms
2019-11-20 19:12:02,353 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1162ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1340ms
2019-11-20 19:12:07,629 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1273ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1436ms
2019-11-20 19:12:18,074 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1127ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Wrote '/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/aggregates.tmp' (25361 bytes)
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting publishing HBase metrics
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Finished publishing HBase metrics
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampIndex: Temporarily shutting down search service
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Saving existing quotas for later merge
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Getting all files with quotas of type ALL
2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) sortBy: null sortRerverse: false limit: 10000
2019-11-20 19:12:27,861 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *})
2019-11-20 19:12:27,861 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for '+(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *})' took 1ms
2019-11-20 19:12:27,862 INFO com.cloudera.headlamp.HeadlampIndex: Swapping index and FS image files
2019-11-20 19:12:27,862 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old FS Image (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/fsimage)
2019-11-20 19:12:28,121 INFO com.cloudera.headlamp.HeadlampIndex: Moving new FS Image in place.
2019-11-20 19:12:28,121 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Index (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index)
2019-11-20 19:12:30,733 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Index in place.
2019-11-20 19:12:30,733 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Aggregate Summary (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/aggregates)
2019-11-20 19:12:30,734 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Aggregate Summary in place.
2019-11-20 19:12:30,734 INFO com.cloudera.headlamp.HeadlampIndex: Starting up search service
2019-11-20 19:12:30,803 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000
2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/
2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 1ms
2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating DSQUOTA in quota cache for '/' from -1 to -1
2019-11-20 19:12:30,980 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000
2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/
2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 1ms
2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating NSQUOTA in quota cache for '/' from 9223372036854775807 to 9223372036854775807
2019-11-20 19:12:32,272 INFO com.cloudera.headlamp.HeadlampIndexManager: Finished indexing HDFS services
2019-11-20 19:38:45,654 INFO com.cloudera.cmf.BasicScmProxy: Failed request to SCM: 302
2019-11-20 19:38:46,655 INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required.
2019-11-20 19:38:46,755 INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM
2019-11-20 19:38:46,874 INFO com.cloudera.cmf.BasicScmProxy: Authenticated to SCM.

Appreciate  any help in this regard

Regards

Wert

[#2]

Hi @wert_1311 ,

It has been a long time since I looked at sizing, so I forget the specifics.  The following rule has helped in the past, though:

(4 x fsimage_size) + 3GB

So, if your fsimage is 980, 7 or 8GB should be appropriate as a starting point.

The Reports Manager downloads the fsimage from the NameNode and then parses/indexes it, so this it needs enough heap to fit the fsimage in memory plus overhead for the indexing process.

Based on what you have collected, increasing the Reports Manager heap sounds like the right call.

Just make sure you have enough free memory on the host before increasing the heap.

Ben

[#3]

@wert_1311,

Yes, @bgooley is right, it is 3-4 times of fsimage, documentation here:
https://docs.cloudera.com/documentation/enterprise/release-notes/topics/hardware_requirements_guide....

What's the current heap setting for RM?

Cheers
Eric

[#4]

@bgooley 

Thanks for your reply and assistance.

@EricL 

Current heap from RM is 1GB

Regards

Wert

[#5 · Accepted solution]

@wert_1311

So based on 1GB fsimage file, you need to have at least 4GB+ for RM to function properly. Please try to increase and observe if there is improvement.

Cheers
Eric

[#6]

@EricL 

Thanks for your assistance.

Regards

Wert