Member since
11-08-2018
96
Posts
3
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2581 | 09-12-2019 10:04 AM | |
3730 | 02-12-2019 06:56 AM |
02-12-2021
06:41 AM
Hi Team, Facing authentication error while downloading parcels from archives.cloudera.com site and tried to generate credentials but facing issues. Can someone please suggest me what way i can solve this issue. Best Regards, Vinod
... View more
Labels:
12-08-2020
06:36 AM
Thank you so much and it is resolved. But why it is not replicating ? When it is keeping under /oldWALs directory it should replicate right ? Can you please give me clarification. Best Regards, Vinod
... View more
12-08-2020
05:29 AM
Hello @smdas Thank you so much for your response. Followed above steps and find below details, ls /hbase/replication [peers, rs] ls /hbase/replication/peers [] Now i have deleted replication in ZNODE and added hbase.replication as false in hbase-site.xml and restarted HBase. Find below details after restart, ls /hbase/replication [peers, rs] ls /hbase/replication/peers [] And now i can see it is cleared and empty under /hbase/oldWALs directory in hdfs. But if you can observe attached screen shot, That is not enabled right? Any differences? Regards, Vinod
... View more
12-07-2020
10:59 PM
Can someone please help me? That would be great and thanks in advance...!!
... View more
12-07-2020
03:34 AM
Hello Team, I am facing issue in my cloudera cluster and HDFS used space is growing and "/hbase/oldWALs" is occupied more than 50%. I can confirm that, HBase replication is disabled and TTL is set to 1 Min. hbase master logcleaner ttl = 1 Min hbase replication = false HBase logs i can see below warn's, WARN org apache hadoop hbase master cleaner CleanerChore: A file cleanerhostname 60000.oldLogCleaner is stopped, won't delete any more files in /nameservice1/hbase/oldWALs And checked list of peers in hbase, hbase(main):001:0> list_peers PEER_ID CLUSTER_KEY STATE TABLE_CFS 0 row(s) in 0.2360 seconds I dont see anything, Please help me with your comments. Thanks & Regards, Vinod
... View more
Labels:
- Labels:
-
Apache HBase
11-20-2020
04:33 AM
Hello Team, Can anyone please help me with your comments. Thanks, Vinod
... View more
10-29-2020
10:09 PM
Hello @Shelton I have added above property in yarn-site.xml for nodemanager and restarted, Still i see same issue and same logs in nodemanager and it is in unknown state, WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config Please give me your valuable response. Thanks, Vinod
... View more
10-07-2020
09:50 PM
Hello @Shelton Any suggestions ? My only doubt is, As other nodemanagers are running fine in the cluster why these newly added nodemanagers were going to unknown state? Regards, Vinod
... View more
10-05-2020
07:41 AM
Hello @Shelton Yes we are using YARN and other Nodemanagers are up and running with same configurations but for newly added nodes we are facing above logs and they are going to unknown state. Yes i verified the above parameter but i dont see the above parameter but in yarn configurations page, I can see " Enable Shuffle Auxiliary Service " was enabled. Please suggest me your comments. Thanks, Vinod
... View more
10-02-2020
07:18 AM
Hello @rblough @Shelton I have added 5 slaves to cloudera manager and deployed slave services and started them. But for NodeManager's we are seeing unknown state even after restarting multiple times. I could see below logs, 2020-10-02 08:52:43,149 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15: SIGTERM 2020-10-02 08:56:16,234 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config. Could you please help me if i am missing anything ? Regards, Vinod
... View more
Labels:
- Labels:
-
Apache YARN
10-02-2020
06:15 AM
Hello @rblough First of all thank you so much for your response..!! 1. The X and Y environments are having different HBase versions. 2. HBase X version is 1.3.1 and HBase Y Version is 1.0.0 3. We are not clearing the table but following below steps, echo "disable '$1'" | hbase shell echo "restore_snapshot '$1_SNAPSHOT_$DATE'" | hbase shell echo "enable '$1'" | hbase shell Daily we are doing the same but sometimes we are seeing the counts mismatch. So, we are doing truncating and restoring the same snapshots and working fine in Y environment. Please help us your inputs. Thanks & Regards, Vinod
... View more
09-18-2020
06:36 AM
Hello Team, I am facing issue with HBase snapshots restoring for big table in one of my environment but the same snapshot is working fine in other environment. Created snapshot in X environment and exported to other Y environment, Once exported we are restoring the table in Y environment. So, here the problem is we are seeing counts mismatch and seeing more rows in Y when comparing with X. NOTE1: Here there is no r/w operations when we create a snapshots in X environment. NOTE2: There is no r/w operations in Y when we are restoring the table. But the same snapshot when we truncate the table and restore, it is working fine in Y, I dont see any issues in HBase logs aswell. Can someone please help me to solve this issue. Thanks, Vinod
... View more
Labels:
- Labels:
-
Apache HBase
08-13-2020
05:52 AM
Hello Team, Can someone please help me with your inputs. Best Regards, Vinod
... View more
08-13-2020
05:44 AM
1 Kudo
Hi @Shelton Could you please help me. Thanks, Vinod
... View more
08-12-2020
12:42 AM
Hello Team, Any help would be appreciated. Best Regards, Vinod
... View more
08-12-2020
12:12 AM
1 Kudo
Hello @Shelton Thanks for your response, I have just installed CDH 6.0.1 with 6 nodes and in another server i have installed docker in edge node. I dont have any reference to integrate docker with yarn, If you can help me with any steps to integrate docker that would be great. Thanks, Vinod
... View more
08-11-2020
08:40 AM
1 Kudo
Hello @jsensharma @Shelton , Can you please help me to launch docker container for my YARN job ? I have installed CDH 6.0.1 & Docker 1.13.1 versions, Any support or help would be appreciated. Thanks, Vinod
... View more
Labels:
- Labels:
-
Apache YARN
-
Docker
08-11-2020
08:30 AM
Hello @Shelton , Can you please tell me whether we can integrate cloudera alerts through opsgenie API ? Like using below command, curl -X POST https://api.opsgenie.com/alerts -H "Content-Type: application/json" -H "Authorization: GenieKey *****************************************" -d '{ "message": "Test Message from API" }' Actually I have created DL in opsgenie and given that DL in cloudera recipients list, So it is sending mail notification to opsgenie and we are getting alerts through opsgenie. But here sometimes its taking more time to trigger the alerts, To avoid this issue, I was trying to integrate opsgenie API like above command to call through cloudera. Please do the needful. Best Regards, Vinod
... View more
Labels:
- Labels:
-
Cloudera Manager
07-28-2020
03:50 AM
Hello Team, We just want to know like, From what version of CDH is supporting by cloudera if we go for licenced version of CDH ? Can someone please let me know. Thanks, Vinod
... View more
Labels:
- Labels:
-
Cloudera Manager
05-31-2020
10:29 PM
Hello Team, Can someone please give me your response and help me with this issue. Regards, Vinod
... View more
05-31-2020
10:28 PM
Hi @Shelton , Thanks for your response and yes i have tried re-genrating key-tab files, But no luck. And the above two servers are master nodes (Zookeeper, journal nodes and other master services are running). Please let me know if you need any other details. Thanks, Vinod
... View more
05-29-2020
06:26 AM
Hello @Shelton @venkatsambath Hope you all doing good. None of the configuration files are bundled with our jars and they use our latest updated config files in the cluster only. What i am missing here ? what could be the reason for failing after enabling Kerberos ? Please do the needful. Best Regards, Vinod
... View more
05-29-2020
06:22 AM
Hey @satz There is no other process is running at that time and only one job is running and frequently it is failing with below error. Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hbase/.hbase-snapshot/.tmp/ And the import job is failing. But when i re-ran it is completing with out any bugs. Please help me with your thoughts. Thanks, Vinod
... View more
05-21-2020
06:45 AM
Hi Team,
Can someone please look into the below strange issue,
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hbase/.hbase-snapshot/.tmp/Table_Name
The above error is throwing when i am running HBase snapshot Import and the job is failing.
Not re gularly but failing some of the times.
Please do the needful and help me with this issue.
Thanks & Regards,
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache HBase
-
HDFS
03-27-2020
10:13 AM
Hello @Shelton, There is no logs generated on that time interval. But i will check for yourquestions with my team and get back to you. Thanks for responding. Best Regards, Vinod
... View more
03-27-2020
09:13 AM
Hello @venkatsambath @Shelton Can some one please revert back and that would be great for me. Best Regards, Vinod
... View more
03-27-2020
05:41 AM
Hi @venkatsambath , Find below one of our application logs , INFO 2020Mar27 01:13:07,422 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: Sleeping for >> 300000 ms ERROR 2020Mar27 01:18:16,485 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: Exception occurred while checking for isReadyToRun flag >> Can't get the location org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268) at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140) at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135) at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888) at com.class.engineering.portfolio.creditinghadoop.config.ConfigTbl.readValue(ConfigTbl.java:115) at com.class.engineering.portfolio.creditinghadoop.config.ConfigDao.read(ConfigDao.java:63) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.isReadyToRun(MrFinalResolver.java:344) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.main(MrFinalResolver.java:116) Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to hostname003.enterprisenet.org/1.1.1.1:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hostname003.enterprisenet.org/1.1.1.1:60020 is closing. Call id=4045, waitTime=2 at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1243) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1214) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:31865) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1580) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1294) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299) ... 12 more Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hostname003.enterprisenet.org/1.1.1.1:60020 is closing. Call id=4045, waitTime=2 at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1033) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:840) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568) ERROR 2020Mar27 01:18:25,554 main com.class.engineering.portfolio.resolution.util.HLogUtil: Exception occured while eriting log message >> Failed 1 action: IOException: 1 time, org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1 time, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207) at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1658) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:208) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1496) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1107) at com.class.engineering.portfolio.creditinghadoop.log.HLogTbl.write(HLogTbl.java:102) at com.class.engineering.portfolio.creditinghadoop.log.HLogDao.write(HLogDao.java:50) at com.class.engineering.portfolio.creditinghadoop.log.HLog.writeMsg(HLog.java:20) at com.class.engineering.portfolio.resolution.util.HLogUtil.writeMsg(HLogUtil.java:18) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.isReadyToRun(MrFinalResolver.java:358) at com.class.engineering.portfolio.finalresolution.main.MrFinalResolver.main(MrFinalResolver.java:116) INFO 2020Mar27 01:18:25,555 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: isReady false. Elapsed time: 18130 ms. INFO 2020Mar27 01:18:25,555 main com.class.engineering.portfolio.finalresolution.main.MrFinalResolver: Sleeping for >> 300000 ms Please revert back. Best Regards, Vinod
... View more
03-26-2020
11:56 PM
Hi @venkatsambath One problem is resolved, I mean to say i am able to run sample yarn job. Found that one of server having some permission issue and deleted /disk(1,2,3,4,5)/yarn/nm directory and restarted the nodemanager and ran yarn job. Its worked. And i tried our application related job and this time getting some different error's, This job i am running in hostname001 and getting below error and seeing some connection failure to hostname003 server. ERROR 2020Mar27 02:35:46,850 main com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain: org.apache.hadoop.hbase.client.RetriesExhaustedException thrown: Can't get the location org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135) ~[DMXSLoader-0.0.31.jar:0.0.31] at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888) ~[DMXSLoader-0.0.31.jar:0.0.31] at com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain.hasStagingData(DMXSLoaderMain.java:304) [DMXSLoader-0.0.31.jar:0.0.31] at com.class.engineering.portfolio.dmxsloader.main.DMXSLoaderMain.main(DMXSLoaderMain.java:375) [DMXSLoader-0.0.31.jar:0.0.31] Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to hostname003.enterprisenet.org/10.7.54.13:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hostname003.enterprisenet.org/10.7.54.13:60020 is closing. Call id=2, waitTime=12 Regards, Vinod
... View more
03-26-2020
10:54 PM
Hi @venkatsambath Thanks for your response..! Yes, i have deleted upto usercache and restarted nodemanagers, We installed 10 nodemanagers and find below stats what you asked, hostname1.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname002.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname003.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname55.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname001.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname003.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname028.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname029.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache hostname054.enterprisenet.org sudo namei -l /disk1/yarn/nm/usercache f: /disk1/yarn/nm/usercache dr-xr-xr-x root root / drwxr-xr-x root root disk1 drwxr-xr-x root root yarn drwxr-xr-x yarn hadoop nm drwxr-xr-x yarn yarn usercache a. Yes, the hbase gate way is available in the same server. b. Actually its not worked with hbase user. And i have granted the permissions to mcaf through hbase shell like below, hbase(main):004:0> grant 'mcaf', 'RWXC' , 'TABLE1' 0 row(s) in 0.6570 seconds hbase(main):005:0> user_permission 'TABLE1' User Namespace,Table,Family,Qualifier:Permission mcaf default,TABLE1,,: [Permission: actions=READ,WRITE,EXEC,CREATE] 1 row(s) in 0.3960 seconds hbase(main):006:0> grant 'mcaf', 'RWXC' , 'TABLE2' 0 row(s) in 0.5780 seconds hbase(main):007:0> user_permission 'TABLE2' User Namespace,Table,Family,Qualifier:Permission mcaf default,TABLE2,,: [Permission: actions=READ,WRITE,EXEC,CREATE] 1 row(s) in 0.4060 seconds c. I am not find any error or warn messages in HMaster logs while i am querying. After granting the permissions i have tested my application job and seeing same error messages like above. Please revert back. Best Regards, Vinod
... View more