Hi Team,   Facing authentication error while downloading parcels from archives.cloudera.com site and tried to generate credentials but facing issues. Can someone please suggest me what way i can solve this issue.   Best Regards, Vinod ... View more

Hello @rblough @Shelton    I have added 5 slaves to cloudera manager and deployed slave services and started them. But for NodeManager's we are seeing unknown state even after restarting multiple times.   I could see below logs, 2020-10-02 08:52:43,149 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15: SIGTERM 2020-10-02 08:56:16,234 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config.   Could you please help me if i am missing anything ?   Regards, Vinod ... View more

Hello Team,   I am facing issue with HBase snapshots restoring for big table in one of my environment but the same snapshot is working fine in other environment.   Created snapshot in X environment and exported to other Y environment, Once exported we are restoring the table in Y environment. So, here the problem is we are seeing counts mismatch and seeing more rows in Y when comparing with X.   NOTE1: Here there is no r/w operations when we create a snapshots in X environment. NOTE2: There is no r/w operations in Y when we are restoring the table.   But the same snapshot when we truncate the table and restore, it is working fine in Y, I dont see any issues in HBase logs aswell. Can someone please help me to solve this issue.   Thanks, Vinod ... View more

Hi Team,   We have a job it is running from long and keep rolling logs with below WARNINGS,     2020-09-06 00:00:36,391 WARN [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Unexpected: procfs stat file is not in the expected format for process with pid 45669 2020-09-06 00:00:39,472 WARN [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Unexpected: procfs stat file is not in the expected format for process with pid 45669 2020-09-06 00:00:42,558 WARN [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Unexpected: procfs stat file is not in the expected format for process with pid 45669 2020-09-06 00:00:45,638 WARN [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Unexpected: procfs stat file is not in the expected format for process with pid 45669 2020-09-06 00:00:48,706 WARN [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Unexpected: procfs stat file is not in the expected format for process with pid 45669 2020-09-06 00:00:51,792 WARN [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Unexpected: procfs stat file is not in the expected format for process with pid 45669     May i know what is this issue ? Any suggestions would be appriciated.   Thanks, Vinod ... View more

Hello @jsensharma @Shelton ,   Can you please help me to launch docker container for my YARN job ? I have installed CDH 6.0.1 & Docker 1.13.1 versions, Any support or help would be appreciated.   Thanks, Vinod     ... View more

Hello @Shelton ,   Can you please tell me whether we can integrate cloudera alerts through opsgenie API ? Like using below command,   curl -X POST https://api.opsgenie.com/alerts -H "Content-Type: application/json" -H "Authorization: GenieKey *****************************************" -d '{ "message": "Test Message from API" }'   Actually I have created DL in opsgenie and given that DL in cloudera recipients list, So it is sending mail notification to opsgenie and we are getting alerts through opsgenie. But here sometimes its taking more time to trigger the alerts, To avoid this issue, I was trying to integrate opsgenie API like above command to call through cloudera. Please do the needful.   Best Regards, Vinod ... View more

Hi Team,   Can someone please look into the below strange issue,   Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hbase/.hbase-snapshot/.tmp/Table_Name     The above error is throwing when i am running HBase snapshot Import and the job is failing. Not re gularly but failing some of the times. Please do the needful and help me with this issue.   Thanks & Regards,   ... View more

Hello Team,   We usually get alert whenever the region server exit but from last couple of days we are failing to get notification alerts through Cloudera Manager.   Can someone please have a look and let me know please.   Best Regards, Vinod ... View more

Hello Team, Can some one please help us to understand whats causing the issue.   Our Region Server Memory configurations: ================================   HBASE_REGIONSERVER_OPTS: ======================== -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ParallelRefProcEnabled -XX:ParallelGCThreads=48 -XX:-ResizePLAB   HBase Master OPTS: ================   -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ParallelRefProcEnabled -XX:ParallelGCThreads=24 -XX:-ResizePLAB   block cache size = 0.4 MemStore size = 0.25 Heap size of Region Server = 32 GB our GC threshold limits are warning 30.0 and critical 60.0      And sometimes we are seeing MemStore full errors in region servers and which is causing the region server connectivity issues with cluster.   Please help me to resolve the issue and let me know do we needs to increase or add any of the configurations in GC settings.   Thanks & Regards, Vinod   ... View more

Hello Team,   We are facing frequent region server aborting in hbase and below is the error logs, ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog Failed close of WAL writer hdfs://nameservice1/hbase/WALs/regionserver74.enterprisenet.org,60020,1576226974029/regionserver74.enterprisenet.org%2C60020%2C1576226974029.null0.1585728806234, unflushedEntries=1 org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1869) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:954) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:728) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2037) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1926) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... 1 more Caused by: java.lang.ArrayIndexOutOfBoundsException: -3 at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2033) ... 5 more FATAL org.apache.hadoop.hbase.regionserver.HRegionServer ABORTING region server regionserver74.enterprisenet.org,60020,1576226974029: Failed log close in log roller org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: hdfs://nameservice1/hbase/WALs/regionserver74.enterprisenet.org,60020,1576226974029/regionserver74.enterprisenet.org%2C60020%2C1576226974029.null0.1585728806234, unflushedEntries=1 at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:1004) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:728) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1869) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:954) ... 3 more Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2037) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1926) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... 1 more Caused by: java.lang.ArrayIndexOutOfBoundsException: -3 at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2033) ... 5 more Apr 1, 4:41:04.707 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 1890784 Apr 1, 4:41:04.712 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 383952 Apr 1, 4:41:04.804 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 12854792 Apr 1, 4:41:04.828 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 192 Apr 1, 4:41:04.860 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 1001512 Apr 1, 4:41:04.884 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 187456 Apr 1, 4:41:04.920 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 17354880 Apr 1, 4:41:04.964 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 108344 Apr 1, 4:41:04.983 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 928648 Apr 1, 4:41:05.031 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 10125336 Apr 1, 4:41:05.151 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 7477760 Apr 1, 4:41:05.199 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 82640 Apr 1, 4:41:05.249 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 1609448 Apr 1, 4:41:05.269 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 13459968 Apr 1, 4:41:05.316 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion Memstore size is 6679936   Can some one please help me to fix this issue?   Best Regards, Vinod ... View more