@kwabstian53  Also as we see that the webhdfs call is actually failing with 404 (resource not found) hence it will be good to find which resource is it ... gthen later we can chekc why it does not exist on HDFS> We can try grepping the "webhdfs' in the Active NameNode HDFS Audit log exactly when we notice  'HDFS Service check failure' We can grep the logs as following: # grep 'proto\=webhdfs' /var/log/hadoop/hdfs/hdfs-audit.log   So when the hdfs service check will fail then exactly at the same time we will know why it failed and for which resource. .   ... View more

@kwabstian53 As we see that the HDFS check is failing while making the webhdfs call because it is not able to find the resource which it is trying to list on HDFS. java.io.IOException: xxxx-master-01:50070: Unexpected HTTP response: code=404 != 200, op=GETFILESTATUS, message=Not Found at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:499) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:135)   So in order to isolate the issue we will need to do the following: 1. Please let us know when exactly the HDFS service check is failing? Is it failing when you are trying to access any specific Ambari view? We may enable DEBUG logging for tta view to get the exact resource path to find out why it is not existing? 2. If you are just running a HDFS Service check from ambari UI as Ambari UI --> HDFS --> Service actions --> 'Run Service Check' then in that case we know that it tries to list files from the HDFS tmp directory as 'hdfs' user so we can try testing it manually to see if the HDFS directory has any issue ? # curl -iLv -X GET "http://xxxx-master-01:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs"   Can you please share the complete output of the above command ?  Please replace the  "xxxx-master-01:50070" with your actual Active NameNode Host & Port. ... View more

@Koffi  It may or may not be related to the JDK. As currently based on the error it looks more related to the issue with the SSL keystore/certificate configured for the NodeManager. As we see the following cause of failure. Caused by: javax.crypto.BadPaddingException: Given final block not properly padded at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:989) at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:845) - Can you please let us know how you actually configured the SSL for NodeManager can you please share the exact details / properties that you changed in YARN config? - Also can you please check if you are able to list the keystore/certificate properly without any issue? - It will be good to know how did you generate the certificates Self Signed or CA Signed? - When was it working earlier? Did you recently configure the keystore/certificate for NodeManager? ... View more

@lvic4594_  Great to know that the issue is resolved after making the recommended changes to use the producer.config argument explicitly  --producer.config /etc/kafka/conf/producer.properties  As the issue is resolved, hence it will be great to mark this thread as Solved. So that other users can quickly find the resolved threads/answers,. ... View more

@kwabstian53  1. Can you please share the NameNode logs?    2. Do you see the NameNode started properly Or are you able to access the NameNode UI properly? 3. Also can you please check if the NameNodes are listening to the ports like 8020 / 50070 ... # netstat -tnlpa | grep `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid`   ... View more

@Koffi  Can you please share more detailed NodeManager log along with complete error trace including Caused By section. The snippet that you posted shows the effect of failure that  "Problem starting http server" but the actual problem will be logged somewhere before this line in the nodemanager log.   Like there can be a Disk space issue on the Nodemanager host where it writes to 'yarn.nodemanager.log-dirs' directory OR it might not be able to bind to the NP port ... etc..    So sharing more detailed NodeManager log will give us better idea. ... View more

@Manoj690  Looks like there is another post from your side on similar issue.  If you are noticing the DB errors in amb ari-server.log then please refer to: https://community.cloudera.com/t5/Support-Questions/ambari-server-status/m-p/268201   It will be really great if you can share some more lines from the ambari-server.log specially the "Caused By" section of the error.  Check if you can attach the log here.   There can be various reasons behind the mentioned database error like ambari might not be able to connect to DB host/port. (firewall issue) The DB credentials might be wrong.  DB might not be running ...    Based on the DB type (MySQL/ Postgres/ Oracle) we can use some standalone client utility OR ambari utility to test the connectivity and DB credentials from ambari server host. Example: If Ambari DB is MySQL   /usr/jdk64/jdk1.8.0_112/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://dbhost.example.com:3306/ambari" "ambari" "bigdata" com.mysql.jdbc.Driver   Example: If Ambari DB is Postgres   /usr/jdk64/jdk1.8.0_112/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/lib/ambari-server/postgresql-9.3-1101-jdbc4.jar org.apache.ambari.server.DBConnectionVerification "jdbc:postgresql://dbhost.example.com:5432/ambari" "ambari" "bigdata" org.postgresql.Driver     You can find your ambari DB details from the following command. In the above command i am using default dummy credentials and URL you will need to find the values for DB url/credentials...etc based on your config, Driver JAR path...etc     # grep 'jdbc' /etc/ambari-server/conf/ambari.properties       .   ... View more

@vinith_alampall  As you are getting this error: sh: 0: Can't open ./start-yarn.sh   Hence you should check couple of things:   1. Is this file existing in the current directory? # cd /PATH/TO # ls -lart /PATH/TO/start-yarn.sh 2. What is the permission on the file? Can the current user read/execute it? # chmod a+x /PATH/TO/start-yarn.sh # which sh # sh -x /PATH/TO/start-yarn.sh   Please replace the '/PATH/TO" with the actual path of the mentioned script. .       ... View more

@Manoj690  Can you please let us know  where exactly do you see that message?  The error message is very generic ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information   - Hence, can you please share the whole ambari-server.log . / amabri-server.out file?   - How frequently do you see this error?     - Do you see any port conflict on ambari server listen port (is that port free?) - Are you running ambari server as non root user?    - Have you checked the permission of ambari log "/var/log/ambari-server" and run directory "/var/run/ambari-server" ? - If your ambari server database running fine and accessible without any issue?   ... View more