Member since
11-14-2018
5
Posts
0
Kudos Received
0
Solutions
11-22-2018
05:59 AM
I am using HDP 2.6.5.0-292. Yes, I cleaned the nodes and did a fresh re-installation receiving the same problem, again. About the zookeeper, I will delete zookeeper from one of the nodes, so it will be installed on three of them. Please notice that the Atlas Metadata Server is up, only the WEB UI server is down and not responding. When I reading the immense and complex log file, there is another message that probably may help: .... Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean name 'get' defined in org.apache.atlas.repository.graph.AtlasGraphProvider: Bean instatiation via factory method failedl nested exception is com.thinkaurelius.titan.core.TitanException: Could not open global configuration.
... View more
11-21-2018
12:01 PM
I am facing a Warning message with the Metadata server. The Error message: http://localhost:21000/api/atlas/admin/status is 0.00s Service Unavailable. And indeed the port does not respond to connection (telnet localhost 21000). The /var/log/atlas/application.log contains the following error: - Context initialization failed (ContextLoader:350) - org.springframework.beans.factory.BeanCreationException: Error creating bean with the name 'org.springframeworkcontext.event.internalEvemtListenerProcessor': BeanPostProcessor before ....... ........ ........ Factory method 'get' threw exception. I have zookeeper working on four nodes,atlas, hbase and ambari-infraSolr installed in two of the nodes (the three softwares are installed in the same two nodes). There are no errors or warnings inside the Solr WEB UI Console and the zookeeper and HBASE seems to work OK. Can you give me a clue what to check in order to find the problem? I tried to delete the three services (HBASE,ambari-infraSolr and Atlas ) and reinstall them, without any influence on the problem that continue to exist. thanks Eduardo
... View more
Labels:
11-15-2018
02:02 PM
Your mentioned configuration is the same than mine, and the jar file is where is it supposed to be (as you wrote above). Please note that I have 4 nodes and the other two are behaving ok, so almost for sure IMHO is not a central configuration problem in Ambari,it is something that is specific to the two specific nodes with problems. I resolved the issue with the following unclean workaround: in the two failed servers I copied the : /usr/hdp/2.3.0.2.6.5.0.292/spark2/aux/spark-2.3.0.2.6.5.-292-yarn-shuffle.jar to /usr/hdp/2.3.0.2.6.5.0.292/hadoop-yarn/lib After copying the jar to the yarn/lib folder, all the namenodes are starting ok, my problem is that I cannot fully understand the source of this error and behavioue before the copy and probably there are additional bad issues in my system (still not in production). Question: Why the two failed nodes write in log : util.ApplicationClassLoader .... - classpath [] but in the good nodes the class is correctly written (in the log: /var/log/hadoop-yarn/yarn) as /usr/hdp/2.6.5.0-292/spark/aux/spark-1.6.3.2.6.5.0-292.jar? Why the : {{stack_root}}/{{spark2_version}}/spark2/aux/* definition is not read as needed in the failed nodes. I also tried to define in ambari without the symbols, i.e: /usr/hdp/2.6.5.0-292/spark/aux/spark-1.6.3.2.6.5.0-292.jar, also, without any success. I am afraid that something is wrong in my installation. Have you any clue what to search next?. how this parameter is read by the namenodes? how the namenodes are started by ambari?. Additional :the file yarn-site.xml in /etc/hadoop/conf is exactly the same in all the nodes (checked by diff) Probably, the best solution is to completely re-install the nameserver or yarn (together with the rpms) on the failed servers. How can I do this? (the delete/add service does not really reinstall the service...) Thank you very much for your previous quick answer!!! Eduardo
... View more
11-15-2018
12:43 PM
Hi Aquilodran Thank you very much for your answer. The advanced yarn-site -> yarn.nodemanager.aux-services.spark2_shuffle.classpath -> {{stack_root}}/{{spark2_version}}/spark2/aux/* is correctly defined in Ambari, also, the jar file correctly installed inside the {{stack_root}}/{{spark2_version}}/spark2/aux folder. I found a dirty workaround by copying the spark*-yarn-shuffle.jar file to : {{stack_root}}/hadoop-yarn/lib. This seems to correct the problem. Inside the log in /var/log/hadoop-yarn/yarn I can find that the classpath for the two failed nodes is empty [] : util.ApplicationClassLoader.java:<init>(105) , but in the two wealthy nodes it is correct. It seems ,at least, that this specific parameter is not read by the nodename processes in the failed nodes I am suspecting that probably something was wrong in the installation, and maybe there are additional hidden problems that I did not encountered, yet. Is any way to completely re-install the yarn package or at least the nodename servers again ?. If not, what is your recommendation to go further with this ? Thanks very much Eduardo
... View more
11-14-2018
07:11 PM
When I start the Nodemanager through ambari, two of the four nodes failed to start. Log mesage: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService. After searching for the class I found that it is defined in the spark-2.3.0.2.6.5.0-292-yarn-shuffle.jar file. That is correctly installed in /usr/hdp/current/spark2-client/aux in all of the nodes. If I start the process manually, adding in the -classpath of the java command, the string: /usr/hdp/current/spark2-client/aux/*, the Nodemanager start, without any error. Question: How can I add the string in the ambari interface? .I added it in the "Advanced yarn-site" -> yarn.application.classpath and it seems that this does not work. Do you have any clue how can successfully correct this problem? Thanks you very much Eduardo
... View more
Labels: