Member since
10-01-2016
156
Posts
6
Kudos Received
6
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2183 | 04-04-2019 09:41 PM | |
| 889 | 06-04-2018 08:34 AM | |
| 323 | 05-23-2018 01:03 PM | |
| 926 | 05-21-2018 07:12 AM | |
| 399 | 05-08-2018 10:48 AM |
04-07-2020
04:11 AM
I need to restart the physical Centos 6.9 server that Cloudera Manager runs on and manage a multinode CDH 5.X cluster. I have searched but couldn't find any Cloudera document for this process if there any. It is a production environment and I need some tips and advice. Thanks in advance.
Services on this node:
HDFS JournalNode
Hive Gateway
Hue Load Balancer
Hue Server
Hue Kerberos Ticket Renewer
Kafka Gateway
Cloudera Management Service Activity Monitor
Cloudera Management Service Alert Publisher
Cloudera Management Service Event Server
Cloudera Management Service Host Monitor
Cloudera Management Service Navigator Audit Server
Cloudera Management Service Navigator Metadata Server
Cloudera Management Service Reports Manager
Cloudera Management Service Service Monitor
Oozie Server
Sentry Gateway
Spark 2 Gateway
Spark 2 History Server
Spark Gateway
Sqoop 1 Client Gateway
ZooKeeper Server
... View more
Labels:
03-25-2020
06:28 AM
Thank you very much. This is the one that satisfies me. Documents are expected to make clear and simple things, not complicated.
... View more
03-23-2020
01:39 AM
I know thank you but I still don't understand why we add same property (dfs.datanode.balance.max.concurrent.moves) on a different section DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml although same property (dfs.datanode.balance.max.concurrent.moves) already exists in Cloudera Manager. Is CM supposed to refuse this addition?
... View more
03-22-2020
12:15 PM
Yes, you are right. I have not realized that. But if dfs.datanode.ec.reconstruction.xmits.weight is already in hdfs-site.xml why Cloudera document makes us add the same property for balancer and DataNode again, what is the point?
... View more
03-22-2020
02:49 AM
I am trying to rebalance hdfs with Cloudera Manager 6.3 with HDFS Balancer Document
It says add the same property dfs.datanode.balance.max.concurrent.moves into different section
DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml
Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml
But before adding the property I searched and saw dfs.datanode.balance.max.concurrent.moves was already there. Nevertheless, I did what the document says. After adding properties Cloudera Manager asked me to restart/redeploy stale configurations. Before restart, I saw totally different properties added.
I don't understand although we seem to add the same property why different properties are added to hdfs-site.xml?
... View more
Labels:
03-06-2020
03:36 AM
Thank you when I copy sigar-1.6.4.jar to /opt/cloudera/parcels/CDH/lib/spark/jars/ error disappeared. But deleting the metrics.properties file still a problem.
... View more
03-04-2020
11:51 PM
I am trying to follow https://github.com/ibm-research-ireland/sparkoscope Enabling Spark Optimization through Cross-stack Monitoring and Visualization. I can't find spark.properties configuration on Cloudera Manager or as a file in /etc/spark/conf. So I created a file named metric.properties and put metrics in it as following
executor.source.jvm.class=org.apache.spark.metrics.source.SigarSource
executor.sink.hdfs.class=org.apache.spark.metrics.sink.HDFSSink
executor.sink.hdfs.pollPeriod = 1
executor.sink.hdfs.dir = hdfs://cloudera.impketra.com:8020/custom-metrics
executor.sink.hdfs.unit = seconds
executor.sink.mqtt.class=org.apache.spark.metrics.sink.MQTTSink
executor.sink.mqtt.pollPeriod = 1
executor.sink.mqtt.host = cloudera.impketra.com
executor.sink.mqtt.port = 1883
executor.sink.mqtt.unit = seconds
But after a while ( I think after restarting Spark service) the metrics.properties file is deleted.
Additionally when I submit
spark-submit --class org.apache.spark.examples.SparkPi --master yarn /opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/spark-examples_2.11-2.4.0-cdh6.3.1.jar 10000
I get this ERROR
20/03/05 10:17:05 ERROR metrics.MetricsSystem: Sink class org.apache.spark.metrics.sink.MQTTSink cannot be instantiated
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:70)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:281)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.metrics.sink.MQTTSink
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:242)
at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:198)
at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:194)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:130)
at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:194)
at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:102)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:365)
at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:201)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:221)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:71)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:70)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
... 4 more
... View more
Labels:
02-28-2020
05:06 AM
In CDH 6.X I can't find Advanced spark2-metrics-properties in Spark config. Should I create manually?
... View more
12-12-2019
04:04 AM
I have installed one node CDH 6.3. I have HBase and I can use hbase shell. I want to use Phonenix in order to use SQL on HBase. I have downloaded apache-phoenix-5.0.0-HBase-2.0-bin.tar.gz, tar it and sof linked phoenix-5.0.0-HBase-2.0-queryserver.jar and phoenix-5.0.0-HBase-2.0-server.jar to /opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hbase/lib/ On terminal I run: [root@cloudera ~]# /opt/phoenix/bin/psql.py cloudera:2181 /opt/phoenix/examples/STOCK_SYMBOL.csv /opt/phoenix/examples/STOCK_SYMBOL.sql But got the following error: SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-phoenix-5.0.0-HBase-2.0-bin/phoenix-5.0.0-HBase-2.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.
19/12/12 14:54:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:469)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:222)
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:273)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3134)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3383)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
... 3 more
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:138)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1297)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1239)
at org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)
at org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)
at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:64)
at org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
at org.apache.phoenix.compile.PostDDLCompiler$2.execute(PostDDLCompiler.java:290)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.updateData(ConnectionQueryServicesImpl.java:3706)
at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1132)
at org.apache.phoenix.compile.CreateTableCompiler$1.execute(CreateTableCompiler.java:192)
at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:408)
at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:391)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:390)
at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:378)
at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1825)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.createOtherSystemTables(ConnectionQueryServicesImpl.java:2722)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$800(ConnectionQueryServicesImpl.java:272)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2618)
at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2532)
at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2532)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.phoenix.util.PhoenixRuntime.main(PhoenixRuntime.java:259)
Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:469)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:222)
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:273)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3134)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3383)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
... 3 more
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1292)
... 28 more
Caused by: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:469)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:222)
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:273)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3134)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3383)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
... 3 more
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:138)
at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:172)
at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
at org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
at org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:135)
at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:120)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:469)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:222)
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:273)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3134)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3383)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
... 3 more
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:138)
at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:153)
at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:165)
... 10 more
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:469)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:222)
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:273)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3134)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3383)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
... 3 more
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:358)
at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:346)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:344)
at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
... 3 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:469)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:222)
at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:273)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3134)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3383)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
... 3 more
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:161)
at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:191)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
... 1 more
... View more
10-28-2019
06:45 PM
Here are newly imported HDP 2.6.5 Sandbox spark-shell --master yarn [root@sandbox-hdp ~]# spark-shell --master yarn
SPARK_MAJOR_VERSION is set to 2, using Spark2
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at <a href="http://sandbox-hdp.hortonworks.com:4040" target="_blank">http://sandbox-hdp.hortonworks.com:4040</a>
Spark context available as 'sc' (master = yarn, app id = application_1572283124735_0001).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.0.2.6.5.0-292
/_/
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_171)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
... View more
10-27-2019
11:01 AM
Where did you paste log4j2.xml? I solved the problem by adding the following : -Dlog4j.configurationFile=/usr/hdp/current/flume-server/conf/log4j2.xml My whole command is: /usr/hdp/current/flume-server/bin/flume-ng agent --conf conf --conf-file /home/maria_dev/flume_uygulama_01/flumelogs.conf --name a1 -Dflume.root.logger=INFO,console -Dlog4j.configurationFile=/usr/hdp/current/flume-server/conf/log4j2.xml
... View more
10-27-2019
05:21 AM
Hi @Shelton No specific steps actually. I just open HDP 2.6.5 Sandbox, connect it via ssh then run spark-shell --master yarn Alternatively, I tried to start Spark on Zeppelin and examined the logs. ERROR was the same. Spark-shell has opened in local mode, no problem. But I can't start it yarn mode. But I managed to start a newly imported Sandbox.
... View more
10-26-2019
10:42 PM
Unfortunately, after a while, the same problem occurred again.
... View more
10-19-2019
04:18 AM
I tried to add kafka-clients-1.0.0.jar into /usr/hdp/2.6.5.0-292/kafka/libs folder but it is no use.
... View more
10-19-2019
02:45 AM
I deleted Sandbox and freshly imported it. It worked.
... View more
10-19-2019
01:14 AM
In HDP 2.6.5 I tried
[root@sandbox-hdp ~]# spark-shell --master yarn
But got following errors. It opens scala shell but cannot create spark session.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/10/19 06:04:17 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FAILED!
19/10/19 06:04:17 ERROR TransportClient: Failed to send RPC 5080948039683175202 to /172.18.0.2:52542: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
19/10/19 06:04:17 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful
java.io.IOException: Failed to send RPC 5080948039683175202 to /172.18.0.2:52542: java.nio.channels.ClosedChannelException
....
....
Caused by: java.nio.channels.ClosedChannelException
...
19/10/19 06:04:17 ERROR Utils: Uncaught exception in thread Yarn application state monitor
org.apache.spark.SparkException: Exception thrown in awaitResult:
...
Caused by: java.io.IOException: Failed to send RPC 5080948039683175202 to /172.18.0.2:52542: java.nio.channels.ClosedChannelException
...
19/10/19 06:04:17 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Spark context stopped while waiting for backend
...
console>:14: error: not found: value spark
import spark.implicits._
^
<console>:14: error: not found: value spark
import spark.sql
Then I looked at yarn logs:
19/10/19 06:21:35 INFO RMProxy: Connecting to ResourceManager at sandbox-hdp.hortonworks.com/172.18.0.2:8030
19/10/19 06:21:35 INFO YarnRMClient: Registering the ApplicationMaster
19/10/19 06:21:35 INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)
....
19/10/19 06:21:35 INFO RMProxy: Connecting to ResourceManager at sandbox-hdp.hortonworks.com/172.18.0.2:8030
19/10/19 06:21:35 INFO YarnRMClient: Registering the ApplicationMaster
19/10/19 06:21:35 INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)
... View more
10-19-2019
01:04 AM
I am using HDP 2.6.5. All services seems fine on Ambari. When I try to open spark-shell with master yarn, I get the following error:
[root@sandbox-hdp ~]# spark-shell --master yarn
SPARK_MAJOR_VERSION is set to 2, using Spark2
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/10/19 06:04:17 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FAILED!
19/10/19 06:04:17 ERROR TransportClient: Failed to send RPC 5080948039683175202 to /172.18.0.2:52542: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
19/10/19 06:04:17 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful
java.io.IOException: Failed to send RPC 5080948039683175202 to /172.18.0.2:52542: java.nio.channels.ClosedChannelException
at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
19/10/19 06:04:17 ERROR Utils: Uncaught exception in thread Yarn application state monitor
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:565)
at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:95)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:155)
at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:508)
at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1752)
at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1931)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1930)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:112)
Caused by: java.io.IOException: Failed to send RPC 5080948039683175202 to /172.18.0.2:52542: java.nio.channels.ClosedChannelException
at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
19/10/19 06:04:17 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Spark context stopped while waiting for backend
at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:669)
at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:177)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:559)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103)
at $line3.$read$$iw$$iw.<init>(<console>:15)
at $line3.$read$$iw.<init>(<console>:43)
at $line3.$read.<init>(<console>:45)
at $line3.$read$.<init>(<console>:49)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.$print$lzycompute(<console>:7)
at $line3.$eval$.$print(<console>:6)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:91)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:78)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:77)
at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:110)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
at org.apache.spark.repl.Main$.doMain(Main.scala:76)
at org.apache.spark.repl.Main$.main(Main.scala:56)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:906)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
java.lang.IllegalStateException: Spark context stopped while waiting for backend
at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:669)
at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:177)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:559)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103)
... 55 elided
<console>:14: error: not found: value spark
import spark.implicits._
^
<console>:14: error: not found: value spark
import spark.sql
^
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.0.2.6.5.0-292
/_/
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_171)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
Then I looked yarn logs from http://sandbox-hdp.hortonworks.com:8088/cluster/app/application_1571461878346_0005
and I see following error log:
Application application_1571461878346_0005 failed 2 times due to AM Container for appattempt_1571461878346_0005_000002 exited with exitCode: 1
For more detailed output, check the application tracking page: <a href="http://sandbox-hdp.hortonworks.com:8088/cluster/app/application_1571461878346_0005" target="_blank">http://sandbox-hdp.hortonworks.com:8088/cluster/app/application_1571461878346_0005</a> Then click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1571461878346_0005_02_000001
Exit code: 1
Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
hostname> \
--cores \
1 \
--app-id \
application_1571461878346_0005 \
--user-class-path \
file:$PWD/__app__.jar \
1><LOG_DIR>/stdout \
2><LOG_DIR>/stderr
resources:
__spark_libs__ -> resource { scheme: "hdfs" host: "sandbox-hdp.hortonworks.com" port: 8020 file: "/hdp/apps/2.6.5.0-292/spark2/spark2-hdp-yarn-archive.tar.gz" } size: 210149527 timestamp: 1571458858104 type: ARCHIVE visibility: PUBLIC
__spark_conf__ -> resource { scheme: "hdfs" host: "sandbox-hdp.hortonworks.com" port: 8020 file: "/user/root/.sparkStaging/application_1571461878346_0005/__spark_conf__.zip" } size: 260074 timestamp: 1571464761206 type: ARCHIVE visibility: PRIVATE
===============================================================================
19/10/19 05:59:29 INFO RMProxy: Connecting to ResourceManager at sandbox-hdp.hortonworks.com/172.18.0.2:8030
19/10/19 05:59:29 INFO YarnRMClient: Registering the ApplicationMaster
19/10/19 05:59:29 INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.records.Resource.getMemorySize()J
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl$ResourceReverseMemoryThenCpuComparator.compare(AMRMClientImpl.java:121)
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl$ResourceReverseMemoryThenCpuComparator.compare(AMRMClientImpl.java:118)
at java.util.TreeMap.compare(TreeMap.java:1295)
at java.util.TreeMap.put(TreeMap.java:538)
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.addResourceRequest(AMRMClientImpl.java:666)
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.addContainerRequest(AMRMClientImpl.java:453)
at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$updateResourceRequests$9.apply(YarnAllocator.scala:365)
at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$updateResourceRequests$9.apply(YarnAllocator.scala:364)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.deploy.yarn.YarnAllocator.updateResourceRequests(YarnAllocator.scala:364)
at org.apache.spark.deploy.yarn.YarnAllocator.allocateResources(YarnAllocator.scala:263)
at org.apache.spark.deploy.yarn.ApplicationMaster.registerAM(ApplicationMaster.scala:476)
at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:534)
at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:347)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:839)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:869)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
19/10/19 05:59:29 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Shutdown hook called before final status was reported.)
19/10/19 05:59:29 INFO ApplicationMaster: Deleting staging directory hdfs://sandbox-hdp.hortonworks.com:8020/user/root/.sparkStaging/application_1571461878346_0005
19/10/19 05:59:29 INFO ShutdownHookManager: Shutdown hook called
Failing this attempt. Failing the application.
... View more
10-12-2019
10:49 PM
This (NULL issue) can be caused by data schema mismatch.
... View more
10-12-2019
10:18 PM
I explained how to manually set a parameter at runtime beeline. Before to do this you have to set sth on Ambari Hive settings. Please refer: https://community.cloudera.com/t5/Support-Questions/params-that-are-allowed-to-be-modified-at-runtime-beeline/m-p/280063/highlight/true#M208647 Just use hive.security.authorization.sqlstd.confwhitelist.append=mapreduce.job.reduces
... View more
10-12-2019
10:10 PM
1 Kudo
Go to Ambari -> Hive -> Configs -> Advanced -> Custom hive-site Click Add Property Insert followings into the opening window: hive.security.authorization.sqlstd.confwhitelist.append=mapred.compress.map.output After saving, restart hive services. Then connect to beeline and set your parameter. I experienced a similar problem for mapreduce.job.reduces parameter and it worked.
... View more
10-08-2019
08:19 AM
My solution was a similar: in command line as a root systemctl stop mysqld
systemctl set-environment MYSQLD_OPTS="--skip-grant-tables --skip-networking"
systemctl start mysqld
mysql -u root in mysql shell use mysql;
update user set authentication_string=PASSWORD("hadoop") where User='root';
flush privileges;
quit After this change, I got trouble hive and ranger due to connection error to MySQL. Then I had to change hive and ranger mysql database root user password from Ambari. It is ok now.
... View more
10-04-2019
12:51 PM
Thank you @Shelton . I have just deleted the line specified in error message in /c/Users/user/.ssh/known_hosts It worked. I had used HDP 2.6.4 before this one. I think there was already a key for the same hostname, once delete it i was able to connect via ssh. This time added new key that belongs to HDP 2.6.5 Sandbox. In this case, Should I accept your solution? Because you offer additional solutions that I have not verified.
... View more
10-04-2019
07:13 AM
I have newly imported HDP 2.6.5 Sandbox after removing the 2.6.4, and tried to connect via ssh. My host machine Windows 10. My command:
ssh root@sandbox-hdp.hortonworks.com -p 2222
The error output:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:8F0VOvcJIA2FD3Ld90kv5FiD3270onl/+uDeDBNffQ0.
Please contact your system administrator.
Add correct host key in /c/Users/user/.ssh/known_hosts to get rid of this message.
Offending RSA key in /c/Users/user/.ssh/known_hosts:3
RSA host key for [sandbox-hdp.hortonworks.com]:2222 has changed and you have requested strict checking.
Host key verification failed.
... View more
Labels:
07-31-2019
06:46 AM
Hi @Shu thank you. Adding spark-yarn_x.x.jar to maven repository solved the problem. But I have come across other errors. Anyway in here the problem was parsing the yarn and it is solved.
... View more
07-30-2019
09:34 AM
Hi @Shu I tried yarn-client and spark-yarn.jar But it can't pass the Could not parse Master URL: 'yarn' ERROR
... View more
07-26-2019
05:19 PM
I use IntelliJ IDE installed on my Windows 10 laptop and try to run spark job in yarn mode on my 5 node HDP 3.1.1 cluster. My codes: package p1
import org.apache.spark.sql.{SparkSession, functions => F}
import org.apache.log4j.{Logger, Level}
object SparkDeneme extends App {
Logger.getLogger("org").setLevel(Level.INFO)
val spark = SparkSession.builder()
.appName("SparkDeneme")
.master("yarn")
.config("spark.hadoop.fs.defaultFS","hdfs://node1.impektra.com:8020")
.config("spark.hadoop.yarn.resoursemanager.address","node1.impektra.com:8030")
.getOrCreate()
}
import spark.implicits._
val sc = spark.sparkContext
val dfFromList = sc.parallelize(List(1,2,3,4,5,6)).toDF("rakamlar")
// dfFromList.printSchema()
dfFromList.show() When I run get following error: 19/07/26 20:00:32 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2744)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:492)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:933)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:924)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:924)
at p1.SparkDeneme$.delayedEndpoint$p1$SparkDeneme$1(SparkDeneme.scala:17)
at p1.SparkDeneme$delayedInit$body.apply(SparkDeneme.scala:8)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at p1.SparkDeneme$.main(SparkDeneme.scala:8)
at p1.SparkDeneme.main(SparkDeneme.scala) I tried to get help from this tutorial Anyone who has succeeded to run Spark YARN mode in IntelliJ?
... View more
Labels:
05-31-2019
12:20 AM
Hi, I am using HDP 3.1 fully distributed cluster. Whenever I start/restart hive services I see new tez application in YARN Resource Manager. I don't understad why?
... View more
Labels:
05-27-2019
04:30 PM
I checked my tez directory [root@node2 ~]# ls -l /usr/hdp/3.1.0.0-78/tez/
total 0
drwxr-xr-x 2 tez hadoop 44 May 27 00:57 conf
lrwxrwxrwx 1 root root 13 May 27 01:12 conf;5ceb0f32 -> /etc/tez/conf
lrwxrwxrwx 1 root root 13 May 27 01:34 conf;5ceb1477 -> /etc/tez/conf
lrwxrwxrwx 1 root root 13 May 27 02:09 conf;5ceb1c24 -> /etc/tez/conf
lrwxrwxrwx 1 root root 13 May 27 03:58 conf;5ceb3650 -> /etc/tez/conf
lrwxrwxrwx 1 root root 13 May 27 06:50 conf;5ceb5dd1 -> /etc/tez/conf
lrwxrwxrwx 1 root root 13 May 27 18:55 conf;5cec0874 -> /etc/tez/conf
lrwxrwxrwx 1 root root 13 May 27 19:13 conf;5cec0c93 -> /etc/tez/conf
lrwxrwxrwx 1 root root 13 May 27 19:17 conf;5cec0d9c -> /etc/tez/conf
lrwxrwxrwx 1 root root 13 May 27 19:21 conf;5cec0e81 -> /etc/tez/conf I think every unsuccessful attempt added a new line. I removed all [root@node2 ~]# rm -rf /usr/hdp/3.1.0.0-78/tez/* Using Ambari I reinstalled tez client for this node only. It worked.
... View more
05-25-2019
06:17 PM
Thank you @Stylianos Sideridis. The issue cooled. Next time I'll keep it in mind, thanks again.
... View more
