Member since
09-12-2017
9
Posts
0
Kudos Received
0
Solutions
03-20-2018
09:45 AM
I am getting the below error while i am trying to connect the hbase through Oozie spark action. I already tried increasing the RPC time in hbase-site.xml file but it's still failing Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Failed after attempts=36, exceptions:
Tue Mar 20 09:21:17 UTC 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68255: row 'source_sandbox,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=sandbox.hortonworks.com,16020,1521536133690, seqNum=0
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Tue Mar 20 09:21:17 UTC 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68255: row 'source_sandbox,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=sandbox.hortonworks.com,16020,1521536133690, seqNum=0
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:324)
at org.apache.hadoop.hbase.client.HRegionLocator.getAllRegionLocations(HRegionLocator.java:89)
at org.apache.hadoop.hbase.util.RegionSizeCalculator.init(RegionSizeCalculator.java:94)
at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:81)
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:256)
at org.apache.hadoop.hbase.mapreduce.TableInputFormat.getSplits(TableInputFormat.java:237)
at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:124)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:242)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:240)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:240)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1928)
at org.apache.spark.rdd.RDD.count(RDD.scala:1164)
at org.apache.spark.api.java.JavaRDDLike$class.count(JavaRDDLike.scala:440)
at org.apache.spark.api.java.AbstractJavaRDDLike.count(JavaRDDLike.scala:46)
at SparkJob.main(SparkJob.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:745)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:311)
at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:232)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:58)
at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:239)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68255: row 'source_sandbox,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=sandbox.hortonworks.com,16020,1521536133690, seqNum=0
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to sandbox.hortonworks.com/172.17.0.2:16020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to sandbox.hortonworks.com/172.17.0.2:16020 is closing. Call id=9, waitTime=12
at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1258)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:379)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:201)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:63)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 4 more
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to sandbox.hortonworks.com/172.17.0.2:16020 is closing. Call id=9, waitTime=12
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1047)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:846)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:574)
The spark job works fine when i run it as a spark submit. It is only failing when i run it on oozie. how do i fix this?
... View more
Labels:
- Labels:
-
Apache Oozie
-
Apache Spark
03-16-2018
11:58 AM
Clearing the queues actually solved the issue for me. Thanks
... View more
03-16-2018
08:38 AM
The stdout which i posted above was observed in the yarn logs. i also get this warning though 08:17:57,832 [main] WARN org.apache.hadoop.security.token.Token - Cannot find class for token kind HBASE_AUTH_TOKEN
2018-03-16 08:17:57,832 [main] WARN org.apache.hadoop.security.token.Token - Cannot find class for token kind HBASE_AUTH_TOKEN Could this be the problem?
... View more
03-15-2018
01:02 PM
If a yarn job is launched, you can view the logs further using below command or by directly viewing through RM UI yarn logs -applicationId <application_ID>.
... View more
03-15-2018
12:48 PM
I am using a spark-action which runs my spark job to get the data from the HBase. I have made all the configurations (https://community.hortonworks.com/content/supportkb/49407/how-to-set-up-oozie-to-connect-to-secured-hbase-cl-1.html). When i run the oozie job it just keeps running indefinitely. The stdout just keeps repeating this below lines 2018-03-15 17:07:56,235 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1521130356618_0004 (state: ACCEPTED)
2018-03-15 17:07:57,237 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1521130356618_0004 (state: ACCEPTED)
2018-03-15 17:07:58,242 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1521130356618_0004 (state: ACCEPTED)
2018-03-15 17:07:59,247 [main] INFO org.apache.spark.deploy.yarn.Client - Application report for application_1521130356618_0004 (state: ACCEPTED)
My workflow.xml <workflow-app xmlns="uri:oozie:workflow:0.5" name="demo-workflow">
<credentials>
<credential name="hbaseauth" type="hbase">
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hbase.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hbase.master.kerberos.principal</name>
<value>hbase/_HOST@HORTONWORKS.COM</value>
</property>
<property>
<name>hbase.regionserver.kerberos.principal</name>
<value>hbase/_HOST@HORTONWORKS.COM</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>sandbox.hortonworks.com</value>
</property>
<property>
<name>hadoop.rpc.protection</name>
<value>authentication</value>
</property>
<property>
<name>hbase.rpc.protection</name>
<value>authentication</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>zookeeper.znode.parent</name>
<value>/hbase-secure</value>
</property>
</credential>
</credentials>
<start to="sparkjob"/>
<action name="sparkjob" cred="hbaseauth">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>sandbox.hortonworks.com:8032</job-tracker>
<name-node>hdfs://sandbox.hortonworks.com:8020</name-node>
<configuration>
<property>
<name>oozie.launcher.mapred.job.queue.name</name>
<value>default</value>
</property>
<property>
<name>oozie.launcher.mapreduce.map.memory.mb</name>
<value>4096</value>
</property>
<property>
<name>oozie.launcher.yarn.app.mapreduce.am.resource.mb</name>
<value>1024</value>
</property>
<property>
<name>mapreduce.job.queuename</name>
<value>default</value>
</property>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
</configuration>
<master>yarn-client</master>
<mode>client</mode>
<name>oozie-sparkjob</name>
<class>SparkJob</class>
<jar>
hdfs://sandbox.hortonworks.com/user/oozie/lib/ooziesparkjobhbase-1.0.0-1.0-SNAPSHOT.jar
</jar>
<spark-opts>--executor-memory 2G --num-executors 5 --queue default --conf spark.ui.port=44040 --files
/usr/hdp/current/spark-client/conf/hive-site.xml --jars
/usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar
--conf spark.yarn.historyServer.address=sandbox.hortonworks.com:18080 --conf spark.eventLog.dir=hdfs://sandbox.hortonworks.com:8020/user/spark/applicationHistory --conf spark.eventLog.enabled=true
</spark-opts>
</spark>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Workflow failed, error message[]${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
My spark job just counts the number of rows in the table and then must print the value. I am not sure what is wrong. Appreciate any help in this. Thanks
... View more
Labels:
- Labels:
-
Apache Oozie
-
Apache YARN
10-10-2017
09:29 AM
I finally got the replication to work. The ports 16000 and 16020 needed to be opened, it looks like hortonworks opens 16010 and
16030 for HBase but not 16000 or 16020. Once I opened these ports I was
able to perform replication.
... View more
10-04-2017
02:31 PM
Hi guys, I am trying to achieve HBase replication using 2 HDP 2.5 Sandboxes. I have assigned a static ip to both of my dockers. I have also changed the Hostnames of both my HDP Clusters. The container ip is also different. Source cluster ip : 192.168.56.132 Destination cluster ip : 192.168.56.134 This is what the hbase logs also state : 2017-10-09 19:27:43,335 INFO [main-SendThread(sandbox.hortonworks.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/192.168.56.134:2181. Will not attempt to authenticate using SASL (unknown error)
2017-10-09 19:27:43,336 INFO [main-SendThread(sandbox.hortonworks.com:2181)] zookeeper.ClientCnxn: Socket connection established to sandbox.hortonworks.com/192.168.56.134:2181, initiating session
2017-10-09 19:27:43,339 INFO [main-SendThread(sandbox.hortonworks.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/192.168.56.134:2181, sessionid = 0x15f01dfbff5003f, negotiated timeout = 40000
2017-10-09 19:27:43,344 INFO [main-EventThread.replicationSource,2] zookeeper.RecoverableZooKeeper: Process identifier=connection to cluster: 2 connecting to ZooKeeper ensemble=sandbox.hortonworks.com:2181
2017-10-09 19:27:43,344 INFO [main-EventThread.replicationSource,2] zookeeper.ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@74d4170f
2017-10-09 19:27:43,349 INFO [main.replicationSource,2-SendThread(sandbox.hortonworks.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/192.168.56.134:2181. Will not attempt to authenticate using SASL (unknown error)
2017-10-09 19:27:43,350 INFO [main.replicationSource,2-SendThread(sandbox.hortonworks.com:2181)] zookeeper.ClientCnxn: Socket connection established to sandbox.hortonworks.com/192.168.56.134:2181, initiating session
2017-10-09 19:27:43,355 INFO [main.replicationSource,2-SendThread(sandbox.hortonworks.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/192.168.56.134:2181, sessionid = 0x15f01dfbff50040, negotiated timeout = 40000
2017-10-09 19:27:43,357 INFO [main-EventThread.replicationSource,2] regionserver.ReplicationSource: Replicating 27a00408-868e-4274-86e8-acc15d2c6697 -> 2ed86c4c-3fb1-4fc8-a5bb-0e2fef15f0eb If i am not wrong the last line here means the connection is established. However I'm not able enable_table_replication '<table name>' hbase shell -d shows below error cannot connect to sandbox.hortonworks.com/192.168.56.134:16020. How do i open this port? Any help is appreciated. Thanks
... View more
Labels:
- Labels:
-
Apache HBase
09-22-2017
10:13 AM
I am trying to demonstrate a task on HBase replication for which i want to have more than one HDP sandbox in my system. I am currently running a docker on Virtualbox in my Ubuntu system. I want to be able to manage another sandbox. Can someone tell me how i can achieve this? Thanks in advance.
... View more
Labels:
- Labels:
-
Apache HBase
09-12-2017
01:06 PM
I have a HDP cluster & have java client that needs to
download the configuration files (hdfs-site.xml, core-site.xml, hbase-site) at runtime? how do i achieve this? Cloudera Manager provides provide URL way downloading config files, do we have something similar with ambari?
... View more
Labels:
- Labels:
-
Apache Ambari
