Member since
04-20-2016
3
Posts
0
Kudos Received
0
Solutions
04-27-2016
05:16 PM
Hi, I follow exactly the same instructions used in this tutorial and when I execute the Spark Scala shell : ./spark-shell --master yarn-client --jars /usr/hdp/2.4.0.0-169/hadoop/hadoop-aws-2.7.1.2.4.0.0-169.jar,/usr/hdp/2.4.0.0-169/hadoop/hadoop-auth.jar,/usr/hdp/2.4.0.0-169/hadoop/aws-java-sdk-1.10.65.jar --driver-memory 512m --executor-memory 512m I get this exception : ERROR SparkContext: Error initializing SparkContext.
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated
....... Caused by: java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentialsProvider
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.newInstance(Class.java:412)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
Not that all jars included in the spark-shell command exist. Any Idea about this error ? Thank you for you help
... View more
04-20-2016
06:15 PM
I lost all heartbeats on some datanodes of my cluster after a restart of machines. The problem is located just after the connection of ambari-agent of the host with ambari-server The last log I received in /var/log/ambari-agent/ambari-agent.log file of the defected DataNode : INFO 2016-04-20 17:59:48,925 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2016-04-20 17:59:48,927 main.py:283 - Connecting to Ambari server at https://hmaster1.xxx.local:8440 (10.10.238.111) NetUtil.py:60 - Connecting to https://hmaster1.xxx.local:8440/ca With the working Datanodes the process continue with this line of log : INFO 2016-04-20 17:51:22,147 threadpool.py:52 - Started thread pool with 3 core threads and 20 maximum threads In the Log of the ambari-server located in /var/log/ambari-server/ambari-server.log file. I see anything between the defected DataNode and Ambari Master. I notice that I use the last version of ambari 2.2.1.1 and centos 7 with the last updates. I disabled all firewall rules and I have the same configuration for the working dataNode and the defected one. Any idea about this strange problem ?
... View more
Labels:
- Labels:
-
Apache Ambari