sagarshimpi
Expert Contributor

Hbase Phoenix Jobs failing with "WrongRegionExcep...

by Expert Contributor in Support Questions
‎08-27-2022 03:01 PM
‎08-27-2022 03:01 PM
Hi Team,   We are using Jmeter to submit job (1300/hr) to hbase/phoenix. HDP3.1.4 and Phoenix 5.0 Job starts failing with below error - 2022-08-25 16:21:44,785 INFO org.apache.phoenix.iterate.BaseResultIterators: Failed to execute task during cancel java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.exceptions.ScannerResetException: Scanner is closed on the server-side at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3468) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range for Get on HRegion OBST:DOCUMENT_METADATA,\x0C\x00\x00\x00,1659594973530.146ed04497483dae508d10d1e2676a12., startKey='\x0C\x00\x00\x00', getEndKey()='\x0CADELMWSQRP\x004bcdbe31987c05d9e88cba377df31f3bbaae274d7df670ed26690fb021c90f5b\x00PERSISTENT', row='\x0CADELSRD\x009bb7104f2f156cec8ecb0e53f95b72affa43969125732ab898c96282356999f7\x00PERSISTENT' at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5713) at org.apache.hadoop.hbase.regionserver.HRegion.prepareGet(HRegion.java:7297) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7290) at org.apache.phoenix.util.IndexUtil.wrapResultUsingOffset(IndexUtil.java:514) at org.apache.phoenix.iterate.RegionScannerFactory$1.nextRaw(RegionScannerFactory.java:197) at org.apache.phoenix.coprocessor.DelegateRegionScanner.nextRaw(DelegateRegionScanner.java:77) at org.apache.phoenix.coprocessor.DelegateRegionScanner.nextRaw(DelegateRegionScanner.java:77) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:274) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3136) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3385) ... 5 more at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1439) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1352) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1239) at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72) at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93) at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.iterate.LimitingResultIterator.next(LimitingResultIterator.java:47) at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:805) at org.apache.calcite.avatica.jdbc.JdbcResultSet.frame(JdbcResultSet.java:148) at org.apache.calcite.avatica.jdbc.JdbcResultSet.create(JdbcResultSet.java:101) at org.apache.calcite.avatica.jdbc.JdbcMeta.execute(JdbcMeta.java:887) at org.apache.calcite.avatica.remote.LocalService.apply(LocalService.java:254) at org.apache.calcite.avatica.remote.Service$ExecuteRequest.accept(Service.java:1032) at org.apache.calcite.avatica.remote.Service$ExecuteRequest.accept(Service.java:1002) at org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:94) at org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46) at org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:127) at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:539) at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.apache.phoenix.shaded.org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.apache.phoenix.shaded.org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.exceptions.ScannerResetException: Scanner is closed on the server-side at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3468) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range for Get on HRegion OBST:DOCUMENT_METADATA,\x0C\x00\x00\x00,1659594973530.146ed04497483dae508d10d1e2676a12., startKey='\x0C\x00\x00\x00', getEndKey()='\x0CADELMWSQRP\x004bcdbe31987c05d9e88cba377df31f3bbaae274d7df670ed26690fb021c90f5b\x00PERSISTENT', row='\x0CADELSRD\x009bb7104f2f156cec8ecb0e53f95b72affa43969125732ab898c96282356999f7\x00PERSISTENT'       Same time we tried to check "select count(*)" with and without index but it gives difference as shown below -   NOTE: Below output is from test cluster where we were able to repro issue. View name might differ in below screenshot -     We suspected below apache bug for  "WrongRegionException: Requested row out of range for Get on HRegion" -  https://issues.apache.org/jira/browse/PHOENIX-3828   For "select count(*)" mismatch - we suspected we are hitting -  [PHOENIX-6090] Local indexes get out of sync after changes for global consistent indexes - ASF JIRA (apache.org)     Can someone help on debugging steps?   ... View more
Labels:

Re: Using yarn logs command

by Expert Contributor in Support Questions
‎12-18-2019 10:37 PM
‎12-18-2019 10:37 PM
Hi @Daggers    Please feel free to select best answer if your questions are answered to close the thread.   Thanks ... View more

Re: Using yarn logs command

by Expert Contributor in Support Questions
‎12-18-2019 07:44 PM
‎12-18-2019 07:44 PM
Hi @Daggers    You can write simple script using yarn rest api to fetch only completed applications [month/daywise] and copy only those applications from hdfs to local. Please check below link -   https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html   ... View more

Re: Worker node '/' directory is full Alerts

by Expert Contributor in Support Questions
‎12-18-2019 01:14 AM
‎12-18-2019 01:14 AM
Hi @Ba  You can probably fetch this information from hdfs cli  rather than going to individual datanode disk.   you can run - hdfs fsck / -files -blocks -locations -->This will give you entire files blocks information with locations on which datanodes the blocks are stored. Just to segregatate wrt each datanodes you can apply some "grep" or "awk" filter   Hope this is what you were looking for. ... View more

Re: Worker node '/' directory is full Alerts

by Expert Contributor in Support Questions
‎12-18-2019 12:57 AM
‎12-18-2019 12:57 AM
@Ba  i didn't get you. can you please elaborate more. ... View more

Re: how can we verify hdfs loaded data replicated...

by Expert Contributor in Support Questions
‎12-16-2019 12:43 AM
‎12-16-2019 12:43 AM
Hi @HadoopHelp    For all questions then answer is -   If you are loading any file to hdfs then using below command you can check if the file is replicated across datanodes or not. The default replication factor is 3 so you can see 3 copied in below command command -   hdfs fsck /myfile.txt -files -blocks -location Eg. hosts is filename in my case hdfs fsck /tmp/hosts -files -blocks -locations /tmp/hosts 1157 bytes, 1 block(s): OK 0. BP-762887186-10.147.167.59-1521037753807:blk_1073748028_7830 len=1157 repl=4 [DatanodeInfoWithStorage[10.1.6.40:1019,DS-6cf46ebf-57fa-4d26-a0f8-f7b99f28424a,DISK], DatanodeInfoWithStorage[10.1.6.44:1019,DS-838d4d62-2069-4b73-b142-76ae1025ae6c,DISK], DatanodeInfoWithStorage[10.1.6.50:1019,DS-da75b9c5-5520-43f4-8e90-60d5982c714d,DISK], DatanodeInfoWithStorage[10.1.6.46:1019,DS-954af47c-1ba7-4057-aacd-1eae700d58cf,DISK]]   In above example you can see it shows blocks present on 3 datanodes ... View more

Re: Using yarn logs command

by Expert Contributor in Support Questions
‎12-15-2019 10:52 PM
‎12-15-2019 10:52 PM
@Daggers    You can also check for HDFS NFS gateway which will allow hdfs filesystem to mount on local OS exposed via NFS.   https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html ... View more

Re: create a multi-cluster system

by Expert Contributor in Support Questions
‎12-15-2019 10:51 PM
‎12-15-2019 10:51 PM
Hi @Hakim    To what i know you cannot use entire hdfs across 2 clusters. If you want you can use HDFS NFS gateway which will allow you to mount hdfs filesystem as mount  point on remote OS.   ... View more

Re: create a multi-cluster system

by Expert Contributor in Support Questions
‎12-15-2019 07:44 PM
‎12-15-2019 07:44 PM
Hi @Hakim  For bigdata technology you can adopt Hadoop platform for build distributed cluster. You can follow link below which will guide you to setup distributed cluster irrespective of number of nodes -   https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/installation.html   Note: You can test for trail subscription which has limited no of service features.   Also Cloudera Manager is Tool which allows you to manage this cluster. ... View more

Re: Using yarn logs command

by Expert Contributor in Support Questions
‎12-15-2019 07:40 PM
1 Kudo
‎12-15-2019 07:40 PM
1 Kudo
Hi @Daggers    I think you can try this -   1. Below properties decides the path for storing yarn logs in hdfs - Belos is sample example from my cluster - yarn.nodemanager.remote-app-log-dir = /app-logs yarn.nodemanager.remote-app-log-dir-suffix = logs-ifile 2. You can do "hadoop dfs -copyToLocal" for above path which will copy all applications to local and then you can pass to splunk ?   Do you think that can work for you? Let me know if you have more questions on above. ... View more

Re: Cloudera on Docker Issue on number of containe...

by Expert Contributor in Support Questions
‎12-08-2019 11:56 PM
‎12-08-2019 11:56 PM
Hi @waelk    To create a case with cloudera you should opt for subscription/license. Probably you need to reach out to sales team at cloudera to explore more. Also pls do check below links -   https://www.cloudera.com/products/pricing.html https://www.cloudera.com/products/pricing/product-features.html   If you are currently running Hadoop for your own learning purpose then you can always post your queries on community and team will be helping you to fix the issues.   For now can you check the log messages for any errors displayed and can u post here. ... View more

Re: Worker node '/' directory is full Alerts

by Expert Contributor in Support Questions
‎12-06-2019 01:11 AM
‎12-06-2019 01:11 AM
Hi @Ba  Did you test ? Do you have any update?     ... View more

Re: Different Log directory free space warning on ...

by Expert Contributor in Support Questions
‎12-06-2019 01:11 AM
‎12-06-2019 01:11 AM
Hi @pdev  Login to the host and execute below command -   for a in /*; do mountpoint -q -- "$a" || du -s -h -x "$a"; done   command which will ignore all mounts and only give size details of filesystem/dirs which resides under "/" You can check and delete data accordingly. ... View more

Re: Worker node '/' directory is full Alerts

by Expert Contributor in Support Questions
‎12-04-2019 01:58 AM
‎12-04-2019 01:58 AM
Hi @Ba As mentioned earlier,   If you are performing copying and remounting of Filesystem then during this activity namenode must be in safe mode and not serving any kind of operation.   If you add new mount to "dfs.name.data.dir", there there is no need to perform any steps only restart of hdfs will be required. No activity for datanodes. ... View more

Re: hive server 2 error

by Expert Contributor in Support Questions
‎12-04-2019 01:37 AM
‎12-04-2019 01:37 AM
Hi @Manoj690  You can clean the database and ambari will take care of this. You do not need to do it manually. ... View more

Re: Worker node '/' directory is full Alerts

by Expert Contributor in Support Questions
‎12-04-2019 01:34 AM
‎12-04-2019 01:34 AM
Hi @BaluD   If this is namenode mount point then make sure the data is backed properly before you move/delete the filesystem.   There will be no impact unless the mount point name is same. Just make sure while you do this activity the namenode was not functional to client. It should be stopped.   I see there is other option -   You can better add the new mount point in HDFS configs ->dfs.name.data.dir [as comma seperated value] Once you change the config and restart Namenode will start writing data to new mount. Once you see that all data is written to new mount comparing size with existing mount point then you can remove old mount from dfs.name.data.dir and restart hdfs   Make sure you do below first - 1. hdfs dfsadmin -safemode on 2. hdfs dfsadmin -saveNamespace ... View more

Re: Worker node '/' directory is full Alerts

by Expert Contributor in Support Questions
‎12-04-2019 01:20 AM
1 Kudo
‎12-04-2019 01:20 AM
1 Kudo
Hi @BaluD  Below will be the steps   1. Lets assume your exsiting data resides in /test [which is wrong mounted on "/"] 2. Create new mount point. eg /data01 [which is /dev/vdd1] 3. Mount the disk [which is wrong mount as off now] on the new mount created in steps1 ie. /data01 4. cp data from existing disk /test to /data01 5. Once copied, test data exist and then remove /test 6. unmount /data01 7. Create mount with /test 8. mount disk to new mount  - mount /dev/vdd1 on /test   Hope steps are clear now.   ... View more

Re: hive server 2 error

by Expert Contributor in Support Questions
‎12-04-2019 01:15 AM
‎12-04-2019 01:15 AM
Hi @Manoj690    You Database does not seems to be clean. It has already tables and schema existing.   Probably you might have already ran below command manually - /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql   Pls confirm.   If YES then you need to clean database and start/install via ambari again. ... View more

Re: Worker node '/' directory is full Alerts

by Expert Contributor in Support Questions
‎12-04-2019 01:09 AM
‎12-04-2019 01:09 AM
Hi @Ba The only way is to copy/move the data to new mount point and then rename. There is no short steps to faster the operation.   Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.   ... View more

Re: hive server 2 error

by Expert Contributor in Support Questions
‎12-03-2019 11:37 PM
‎12-03-2019 11:37 PM
Hi @Manoj690 Is this resolved ?   If so please accept the best answer. If you are still facing issue please let us know the error. We can check this out.   ... View more

Re: Worker node '/' directory is full Alerts

by Expert Contributor in Support Questions
‎12-03-2019 11:35 PM
1 Kudo
‎12-03-2019 11:35 PM
1 Kudo
Hi @BaluD    This is more of Unix question rathen than hadoop 🙂   Please try below command which will ignore all mounts and only give size details of filesystem/dirs which resides under "/"   for a in /*; do mountpoint -q -- "$a" || du -s -h -x "$a"; done  Let me know if that helps. ... View more

Re: Data replication in datanode new

by Expert Contributor in Support Questions
‎12-03-2019 06:29 AM
‎12-03-2019 06:29 AM
Hi @Peruvian81  there is no such option in ambari UI   You can either check from Namenode UI --> datanode tab and see if the block counts are increasing. ... View more

Re: Data replication in datanode new

by Expert Contributor in Support Questions
‎12-02-2019 06:47 AM
‎12-02-2019 06:47 AM
Hi @Peruvian81    Once you add new datanode to cluster and if the replication starts you can see messages somehow like below in datanode logs - which signifies that new node is finalizing blocks written as well as receiving blocks from source node within replication.   DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /<IPADDRESS>:45858, dest: /<IPADDRESS>:1019, bytes: 7526, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-646394656_1, offset: 0, srvID: 973c1ebc-7c88-4163-aea3-8c2e0f4f4975, blockid: BP-826310834-<IPADDRESS>-1480602348927:blk_1237811292_164146312, duration: 9130002 datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-826310834-<IPADDRESS>-1480602348927:blk_1237811295_164146315 src: /<IPADDRESS>:36930 dest: /<IPADDRESS>:1019   ... View more

Re: Mounting Edge Node Directories to External App...

by Expert Contributor in Support Questions
‎11-29-2019 01:38 AM
‎11-29-2019 01:38 AM
Hi @laplacesdemon    Than you for the response and appreciation. I will be happy to contribute and share my experiences gong further. Thank you for accepting the answer. ... View more

Re: Mysql JDBC connection error for ambari install...

by Expert Contributor in Support Questions
‎11-28-2019 10:51 PM
‎11-28-2019 10:51 PM
@Manoj690    Can you remove password from previous comment. Just to avoid escalation of security standards. Can you share the commands you executed previously? ... View more

Re: How to check disks health in Hadoop clusters a...

by Expert Contributor in Support Questions
‎11-28-2019 08:11 PM
‎11-28-2019 08:11 PM
Hi @mike_bronson7    There is no generic/specific tool to monitor HDFS/KAFKA disk.Most client generally use the tool which is incorporated with OS vendor or either opt for thrid party tools. However there are multiple external tools available through which you can achieve this task -   Nagios / OpsView /HPOMI are popular options I've seen. In our environment KAFKA is extensively used and we have HPOMI and Prometheus installed for monitoring.   ... View more

Re: How to Remove a large list of LDAP USERS/GROUP...

by Expert Contributor in Support Questions
‎11-28-2019 07:57 PM
‎11-28-2019 07:57 PM
Hi @Koffi    You can test this out. I tried this in past and it worked for me.   https://community.cloudera.com/t5/Support-Questions/How-to-Remove-all-External-Users-from-the-Ranger-Ranger/td-p/94987   Also do check https://issues.apache.org/jira/browse/RANGER-205   Hope that helps. ... View more

Re: Mysql JDBC connection error for ambari install...

by Expert Contributor in Support Questions
‎11-28-2019 03:28 AM
‎11-28-2019 03:28 AM
@Manoj690  login to mysql and follow step 2 from below link   https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/administering-ambari/content/amb_using_ambari_with_mysql_or_mariadb.html   make sure you give. permission as per your ambari hostname(FQDN) also. ... View more

Re: Mysql JDBC connection error for ambari install...

by Expert Contributor in Support Questions
‎11-28-2019 02:43 AM
‎11-28-2019 02:43 AM
From Ambari node can you try - mysql -u <ambari_DB_username> <Ambari_DB_name> -h <DB_hostname> -P <Port>   Check if you are able to successfully login to the DB from ambari node.     ... View more

Re: Mysql JDBC connection error for ambari install...

by Expert Contributor in Support Questions
‎11-28-2019 02:19 AM
‎11-28-2019 02:19 AM
Hi @Manoj690    Now the error is different -   "jdbc:mysql://xxxxx:3306/ambari" "ambari" "Password" com.mysql.jdbc.Driver ERROR: Unable to connect to the DB. Please check DB connection properties. com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   Probably check connection to your DB server from ambari node. Login to ambari server and telnet - $telnet <DB_HOST>  3306 Make sure IPtables/selinux is disabled.   If you think initial issue is resolved please close this topic my accepting the correct reply on thread.   Will be happy to help you for above issue. Please keep posted. ... View more
Contact Me
Online
Offline
Last Visited
‎09-14-2022 02:31 AM
Public Statistics
Date Registered ‎02-18-2016 01:33 AM
Last Visited ‎09-14-2022 02:31 AM
Total Messages Posted 136
Total Kudos Received 19