Reports Manager Pause Duration Issue

wert_1311 — Wed, 20 Nov 2019 07:40:21 GMT

Hello All,

Off late we have been getting concerning alerts for Reports Manager (The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.) occasionally reports manager service also goes to bad state.

I believe, this has to do with Reports Manager Heap Memory, but am in a doubt hence wanted to get a confirmation before playing around with it. I have attached a snippet of the error, which I get.

Also would like to know if there is any reference guide / document by which, we can calculate and periodically adjust such items.

HDFS Used – 10TB / Total 90TB
FSImage – 980M
CM - 6.16.2 / CDH - 5.16.2

The health test result for REPORTS_MANAGER_PAUSE_DURATION has become concerning: Average time spent paused was 35.6 second(s) (59.41%) per minute over the previous 5 minute(s). Warning threshold: 30.00%. [root@MY Server cloudera-scm-headlamp]# tail -n 300 mgmt-cmf-mgmt-REPORTSMANAGER-MY Server.com.log.out 2019-11-20 19:03:43,443 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1363ms: GC pool 'ParNew' had collection(s): count=1 time=65ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1537ms 2019-11-20 19:03:48,933 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1965ms: GC pool 'ParNew' had collection(s): count=1 time=834ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1571ms 2019-11-20 19:04:10,282 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1228ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1280ms 2019-11-20 19:04:15,629 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1345ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1366ms 2019-11-20 19:04:26,295 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1321ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1343ms 2019-11-20 19:04:31,499 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1114ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms 2019-11-20 19:04:36,891 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1390ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1399ms 2019-11-20 19:04:52,016 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1138ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1158ms 2019-11-20 19:05:12,572 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1014ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1226ms 2019-11-20 19:05:28,066 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1051ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1311ms 2019-11-20 19:05:52,991 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1325ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1332ms 2019-11-20 19:06:35,411 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1104ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1395ms 2019-11-20 19:06:46,384 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1083ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1421ms 2019-11-20 19:06:51,484 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1098ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms 2019-11-20 19:07:07,179 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1249ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1273ms 2019-11-20 19:07:27,892 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1103ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1258ms 2019-11-20 19:07:33,073 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1179ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1294ms 2019-11-20 19:07:38,111 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1036ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1096ms 2019-11-20 19:07:44,198 INFO com.cloudera.cmf.BasicScmProxy: Failed request to SCM: 302 2019-11-20 19:07:45,318 INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required. 2019-11-20 19:07:45,394 INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM 2019-11-20 19:07:45,403 INFO com.cloudera.cmf.BasicScmProxy: Authenticated to SCM. 2019-11-20 19:08:07,837 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1157ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1218ms 2019-11-20 19:08:28,276 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1133ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1248ms 2019-11-20 19:08:33,105 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1227ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1251ms 2019-11-20 19:08:38,015 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1179ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1336ms 2019-11-20 19:08:40,247 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1175ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1312ms 2019-11-20 19:08:44,598 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1156ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1284ms 2019-11-20 19:08:46,743 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1094ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1312ms 2019-11-20 19:08:53,086 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1162ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1340ms 2019-11-20 19:08:55,207 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1120ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1306ms 2019-11-20 19:08:57,373 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1109ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1241ms 2019-11-20 19:08:59,443 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1069ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1203ms 2019-11-20 19:09:03,683 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1171ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1307ms 2019-11-20 19:09:05,929 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1191ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1355ms 2019-11-20 19:09:12,724 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1046ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1235ms 2019-11-20 19:09:14,848 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1123ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1292ms 2019-11-20 19:09:17,095 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1247ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1373ms 2019-11-20 19:09:20,294 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1142ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1283ms 2019-11-20 19:09:27,791 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1084ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1524ms 2019-11-20 19:09:30,897 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1050ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1358ms 2019-11-20 19:09:48,006 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1283ms 2019-11-20 19:09:50,625 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1118ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1155ms 2019-11-20 19:10:05,535 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1174ms 2019-11-20 19:10:12,452 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1164ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1268ms 2019-11-20 19:10:24,685 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1234ms 2019-11-20 19:10:34,427 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1163ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1312ms 2019-11-20 19:10:39,574 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1083ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1419ms 2019-11-20 19:10:49,650 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1198ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1413ms 2019-11-20 19:10:54,658 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1006ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1184ms 2019-11-20 19:10:59,817 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1081ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1249ms 2019-11-20 19:11:10,429 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1215ms: GC pool 'ParNew' had collection(s): count=1 time=1ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1294ms 2019-11-20 19:11:20,762 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1194ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1237ms 2019-11-20 19:11:25,839 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1075ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1331ms 2019-11-20 19:11:31,441 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1057ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1378ms 2019-11-20 19:11:36,533 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1065ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1200ms 2019-11-20 19:11:51,468 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1064ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1352ms 2019-11-20 19:11:57,174 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1189ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1529ms 2019-11-20 19:12:02,353 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1162ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=1340ms 2019-11-20 19:12:07,629 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1273ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1436ms 2019-11-20 19:12:18,074 INFO com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1127ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1252ms 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Wrote '/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/aggregates.tmp' (25361 bytes) 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting publishing HBase metrics 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.LuceneImageVisitor: Finished publishing HBase metrics 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampIndex: Temporarily shutting down search service 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Saving existing quotas for later merge 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Getting all files with quotas of type ALL 2019-11-20 19:12:27,859 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) sortBy: null sortRerverse: false limit: 10000 2019-11-20 19:12:27,861 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) 2019-11-20 19:12:27,861 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for '+(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *})' took 1ms 2019-11-20 19:12:27,862 INFO com.cloudera.headlamp.HeadlampIndex: Swapping index and FS image files 2019-11-20 19:12:27,862 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old FS Image (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/fsimage) 2019-11-20 19:12:28,121 INFO com.cloudera.headlamp.HeadlampIndex: Moving new FS Image in place. 2019-11-20 19:12:28,121 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Index (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index) 2019-11-20 19:12:30,733 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Index in place. 2019-11-20 19:12:30,733 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Aggregate Summary (/var/lib/cloudera-scm-headlamp/hdfs/nameservice1/aggregates) 2019-11-20 19:12:30,734 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Aggregate Summary in place. 2019-11-20 19:12:30,734 INFO com.cloudera.headlamp.HeadlampIndex: Starting up search service 2019-11-20 19:12:30,803 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000 2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/ 2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 1ms 2019-11-20 19:12:30,804 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating DSQUOTA in quota cache for '/' from -1 to -1 2019-11-20 19:12:30,980 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000 2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/ 2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 1ms 2019-11-20 19:12:30,981 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating NSQUOTA in quota cache for '/' from 9223372036854775807 to 9223372036854775807 2019-11-20 19:12:32,272 INFO com.cloudera.headlamp.HeadlampIndexManager: Finished indexing HDFS services 2019-11-20 19:38:45,654 INFO com.cloudera.cmf.BasicScmProxy: Failed request to SCM: 302 2019-11-20 19:38:46,655 INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required. 2019-11-20 19:38:46,755 INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM 2019-11-20 19:38:46,874 INFO com.cloudera.cmf.BasicScmProxy: Authenticated to SCM.

Appreciate any help in this regard

Regards

Wert

Re: Reports Manager Pause Duration Issue

bgooley — Wed, 20 Nov 2019 16:41:16 GMT

Hi @wert_1311 ,

It has been a long time since I looked at sizing, so I forget the specifics. The following rule has helped in the past, though:

(4 x fsimage_size) + 3GB

So, if your fsimage is 980, 7 or 8GB should be appropriate as a starting point.

The Reports Manager downloads the fsimage from the NameNode and then parses/indexes it, so this it needs enough heap to fit the fsimage in memory plus overhead for the indexing process.

Based on what you have collected, increasing the Reports Manager heap sounds like the right call.

Just make sure you have enough free memory on the host before increasing the heap.

Ben

question Reports Manager Pause Duration Issue in Support Questions

Reports Manager Pause Duration Issue

Re: Reports Manager Pause Duration Issue

Re: Reports Manager Pause Duration Issue

Re: Reports Manager Pause Duration Issue

Re: Reports Manager Pause Duration Issue

Re: Reports Manager Pause Duration Issue