Member since
02-18-2016
136
Posts
19
Kudos Received
18
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2190 | 12-18-2019 07:44 PM | |
2220 | 12-15-2019 07:40 PM | |
902 | 12-03-2019 06:29 AM | |
921 | 12-02-2019 06:47 AM | |
2381 | 11-28-2019 02:06 AM |
08-27-2022
03:01 PM
Hi Team, We are using Jmeter to submit job (1300/hr) to hbase/phoenix. HDP3.1.4 and Phoenix 5.0 Job starts failing with below error - 2022-08-25 16:21:44,785 INFO org.apache.phoenix.iterate.BaseResultIterators: Failed to execute task during cancel java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.exceptions.ScannerResetException: Scanner is closed on the server-side at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3468) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range for Get on HRegion OBST:DOCUMENT_METADATA,\x0C\x00\x00\x00,1659594973530.146ed04497483dae508d10d1e2676a12., startKey='\x0C\x00\x00\x00', getEndKey()='\x0CADELMWSQRP\x004bcdbe31987c05d9e88cba377df31f3bbaae274d7df670ed26690fb021c90f5b\x00PERSISTENT', row='\x0CADELSRD\x009bb7104f2f156cec8ecb0e53f95b72affa43969125732ab898c96282356999f7\x00PERSISTENT' at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5713) at org.apache.hadoop.hbase.regionserver.HRegion.prepareGet(HRegion.java:7297) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:7290) at org.apache.phoenix.util.IndexUtil.wrapResultUsingOffset(IndexUtil.java:514) at org.apache.phoenix.iterate.RegionScannerFactory$1.nextRaw(RegionScannerFactory.java:197) at org.apache.phoenix.coprocessor.DelegateRegionScanner.nextRaw(DelegateRegionScanner.java:77) at org.apache.phoenix.coprocessor.DelegateRegionScanner.nextRaw(DelegateRegionScanner.java:77) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:274) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3136) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3385) ... 5 more at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1439) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1352) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1239) at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72) at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93) at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58) at org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44) at org.apache.phoenix.iterate.LimitingResultIterator.next(LimitingResultIterator.java:47) at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:805) at org.apache.calcite.avatica.jdbc.JdbcResultSet.frame(JdbcResultSet.java:148) at org.apache.calcite.avatica.jdbc.JdbcResultSet.create(JdbcResultSet.java:101) at org.apache.calcite.avatica.jdbc.JdbcMeta.execute(JdbcMeta.java:887) at org.apache.calcite.avatica.remote.LocalService.apply(LocalService.java:254) at org.apache.calcite.avatica.remote.Service$ExecuteRequest.accept(Service.java:1032) at org.apache.calcite.avatica.remote.Service$ExecuteRequest.accept(Service.java:1002) at org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:94) at org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46) at org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:127) at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:539) at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.apache.phoenix.shaded.org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.apache.phoenix.shaded.org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.exceptions.ScannerResetException: Scanner is closed on the server-side at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3468) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range for Get on HRegion OBST:DOCUMENT_METADATA,\x0C\x00\x00\x00,1659594973530.146ed04497483dae508d10d1e2676a12., startKey='\x0C\x00\x00\x00', getEndKey()='\x0CADELMWSQRP\x004bcdbe31987c05d9e88cba377df31f3bbaae274d7df670ed26690fb021c90f5b\x00PERSISTENT', row='\x0CADELSRD\x009bb7104f2f156cec8ecb0e53f95b72affa43969125732ab898c96282356999f7\x00PERSISTENT' Same time we tried to check "select count(*)" with and without index but it gives difference as shown below - NOTE: Below output is from test cluster where we were able to repro issue. View name might differ in below screenshot - We suspected below apache bug for "WrongRegionException: Requested row out of range for Get on HRegion" - https://issues.apache.org/jira/browse/PHOENIX-3828 For "select count(*)" mismatch - we suspected we are hitting - [PHOENIX-6090] Local indexes get out of sync after changes for global consistent indexes - ASF JIRA (apache.org) Can someone help on debugging steps?
... View more
Labels:
- Labels:
-
Apache Phoenix
12-18-2019
10:37 PM
Hi @Daggers Please feel free to select best answer if your questions are answered to close the thread. Thanks
... View more
12-18-2019
07:44 PM
Hi @Daggers You can write simple script using yarn rest api to fetch only completed applications [month/daywise] and copy only those applications from hdfs to local. Please check below link - https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
... View more
12-18-2019
01:14 AM
Hi @Ba You can probably fetch this information from hdfs cli rather than going to individual datanode disk. you can run - hdfs fsck / -files -blocks -locations
-->This will give you entire files blocks information with locations on which datanodes the blocks are stored.
Just to segregatate wrt each datanodes you can apply some "grep" or "awk" filter Hope this is what you were looking for.
... View more
12-18-2019
12:57 AM
@Ba i didn't get you. can you please elaborate more.
... View more
12-16-2019
12:43 AM
Hi @HadoopHelp For all questions then answer is - If you are loading any file to hdfs then using below command you can check if the file is replicated across datanodes or not. The default replication factor is 3 so you can see 3 copied in below command command - hdfs fsck /myfile.txt -files -blocks -location
Eg. hosts is filename in my case
hdfs fsck /tmp/hosts -files -blocks -locations
/tmp/hosts 1157 bytes, 1 block(s): OK
0. BP-762887186-10.147.167.59-1521037753807:blk_1073748028_7830 len=1157 repl=4 [DatanodeInfoWithStorage[10.1.6.40:1019,DS-6cf46ebf-57fa-4d26-a0f8-f7b99f28424a,DISK], DatanodeInfoWithStorage[10.1.6.44:1019,DS-838d4d62-2069-4b73-b142-76ae1025ae6c,DISK], DatanodeInfoWithStorage[10.1.6.50:1019,DS-da75b9c5-5520-43f4-8e90-60d5982c714d,DISK], DatanodeInfoWithStorage[10.1.6.46:1019,DS-954af47c-1ba7-4057-aacd-1eae700d58cf,DISK]] In above example you can see it shows blocks present on 3 datanodes
... View more
12-15-2019
10:52 PM
@Daggers You can also check for HDFS NFS gateway which will allow hdfs filesystem to mount on local OS exposed via NFS. https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html
... View more
12-15-2019
10:51 PM
Hi @Hakim To what i know you cannot use entire hdfs across 2 clusters. If you want you can use HDFS NFS gateway which will allow you to mount hdfs filesystem as mount point on remote OS.
... View more
12-15-2019
07:44 PM
Hi @Hakim For bigdata technology you can adopt Hadoop platform for build distributed cluster. You can follow link below which will guide you to setup distributed cluster irrespective of number of nodes - https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/installation.html Note: You can test for trail subscription which has limited no of service features. Also Cloudera Manager is Tool which allows you to manage this cluster.
... View more
12-15-2019
07:40 PM
1 Kudo
Hi @Daggers I think you can try this - 1. Below properties decides the path for storing yarn logs in hdfs - Belos is sample example from my cluster -
yarn.nodemanager.remote-app-log-dir = /app-logs
yarn.nodemanager.remote-app-log-dir-suffix = logs-ifile 2. You can do "hadoop dfs -copyToLocal" for above path which will copy all applications to local and then you can pass to splunk ? Do you think that can work for you? Let me know if you have more questions on above.
... View more
12-08-2019
11:56 PM
Hi @waelk To create a case with cloudera you should opt for subscription/license. Probably you need to reach out to sales team at cloudera to explore more. Also pls do check below links - https://www.cloudera.com/products/pricing.html https://www.cloudera.com/products/pricing/product-features.html If you are currently running Hadoop for your own learning purpose then you can always post your queries on community and team will be helping you to fix the issues. For now can you check the log messages for any errors displayed and can u post here.
... View more
12-06-2019
01:11 AM
Hi @Ba Did you test ? Do you have any update?
... View more
12-06-2019
01:11 AM
Hi @pdev Login to the host and execute below command - for a in /*; do mountpoint -q -- "$a" || du -s -h -x "$a"; done command which will ignore all mounts and only give size details of filesystem/dirs which resides under "/" You can check and delete data accordingly.
... View more
12-04-2019
01:58 AM
Hi @Ba As mentioned earlier, If you are performing copying and remounting of Filesystem then during this activity namenode must be in safe mode and not serving any kind of operation. If you add new mount to "dfs.name.data.dir", there there is no need to perform any steps only restart of hdfs will be required. No activity for datanodes.
... View more
12-04-2019
01:37 AM
Hi @Manoj690 You can clean the database and ambari will take care of this. You do not need to do it manually.
... View more
12-04-2019
01:34 AM
Hi @BaluD If this is namenode mount point then make sure the data is backed properly before you move/delete the filesystem. There will be no impact unless the mount point name is same. Just make sure while you do this activity the namenode was not functional to client. It should be stopped. I see there is other option - You can better add the new mount point in HDFS configs ->dfs.name.data.dir [as comma seperated value] Once you change the config and restart Namenode will start writing data to new mount. Once you see that all data is written to new mount comparing size with existing mount point then you can remove old mount from dfs.name.data.dir and restart hdfs Make sure you do below first - 1. hdfs dfsadmin -safemode on 2. hdfs dfsadmin -saveNamespace
... View more
12-04-2019
01:20 AM
1 Kudo
Hi @BaluD Below will be the steps 1. Lets assume your exsiting data resides in /test [which is wrong mounted on "/"] 2. Create new mount point. eg /data01 [which is /dev/vdd1] 3. Mount the disk [which is wrong mount as off now] on the new mount created in steps1 ie. /data01 4. cp data from existing disk /test to /data01 5. Once copied, test data exist and then remove /test 6. unmount /data01 7. Create mount with /test 8. mount disk to new mount - mount /dev/vdd1 on /test Hope steps are clear now.
... View more
12-04-2019
01:15 AM
Hi @Manoj690 You Database does not seems to be clean. It has already tables and schema existing. Probably you might have already ran below command manually - /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Pls confirm. If YES then you need to clean database and start/install via ambari again.
... View more
12-04-2019
01:09 AM
Hi @Ba The only way is to copy/move the data to new mount point and then rename. There is no short steps to faster the operation. Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
... View more
12-03-2019
11:37 PM
Hi @Manoj690 Is this resolved ? If so please accept the best answer. If you are still facing issue please let us know the error. We can check this out.
... View more
12-03-2019
11:35 PM
1 Kudo
Hi @BaluD This is more of Unix question rathen than hadoop 🙂 Please try below command which will ignore all mounts and only give size details of filesystem/dirs which resides under "/" for a in /*; do mountpoint -q -- "$a" || du -s -h -x "$a"; done Let me know if that helps.
... View more
12-03-2019
06:29 AM
Hi @Peruvian81 there is no such option in ambari UI You can either check from Namenode UI --> datanode tab and see if the block counts are increasing.
... View more
12-02-2019
06:47 AM
Hi @Peruvian81 Once you add new datanode to cluster and if the replication starts you can see messages somehow like below in datanode logs - which signifies that new node is finalizing blocks written as well as receiving blocks from source node within replication. DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1490)) - src: /<IPADDRESS>:45858, dest: /<IPADDRESS>:1019, bytes: 7526, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-646394656_1, offset: 0, srvID: 973c1ebc-7c88-4163-aea3-8c2e0f4f4975, blockid: BP-826310834-<IPADDRESS>-1480602348927:blk_1237811292_164146312, duration: 9130002
datanode.DataNode (DataXceiver.java:writeBlock(669)) - Receiving BP-826310834-<IPADDRESS>-1480602348927:blk_1237811295_164146315 src: /<IPADDRESS>:36930 dest: /<IPADDRESS>:1019
... View more
11-29-2019
01:38 AM
Hi @laplacesdemon Than you for the response and appreciation. I will be happy to contribute and share my experiences gong further. Thank you for accepting the answer.
... View more
11-28-2019
10:51 PM
@Manoj690 Can you remove password from previous comment. Just to avoid escalation of security standards. Can you share the commands you executed previously?
... View more
11-28-2019
08:11 PM
Hi @mike_bronson7 There is no generic/specific tool to monitor HDFS/KAFKA disk.Most client generally use the tool which is incorporated with OS vendor or either opt for thrid party tools. However there are multiple external tools available through which you can achieve this task - Nagios / OpsView /HPOMI are popular options I've seen. In our environment KAFKA is extensively used and we have HPOMI and Prometheus installed for monitoring.
... View more
11-28-2019
07:57 PM
Hi @Koffi You can test this out. I tried this in past and it worked for me. https://community.cloudera.com/t5/Support-Questions/How-to-Remove-all-External-Users-from-the-Ranger-Ranger/td-p/94987 Also do check https://issues.apache.org/jira/browse/RANGER-205 Hope that helps.
... View more
11-28-2019
03:28 AM
@Manoj690 login to mysql and follow step 2 from below link https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/administering-ambari/content/amb_using_ambari_with_mysql_or_mariadb.html make sure you give. permission as per your ambari hostname(FQDN) also.
... View more
11-28-2019
02:43 AM
From Ambari node can you try - mysql -u <ambari_DB_username> <Ambari_DB_name> -h <DB_hostname> -P <Port> Check if you are able to successfully login to the DB from ambari node.
... View more
11-28-2019
02:19 AM
Hi @Manoj690 Now the error is different - "jdbc:mysql://xxxxx:3306/ambari" "ambari" "Password" com.mysql.jdbc.Driver
ERROR: Unable to connect to the DB. Please check DB connection properties.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure Probably check connection to your DB server from ambari node. Login to ambari server and telnet - $telnet <DB_HOST> 3306 Make sure IPtables/selinux is disabled. If you think initial issue is resolved please close this topic my accepting the correct reply on thread. Will be happy to help you for above issue. Please keep posted.
... View more