Hi Using Ambari, i have setup hadoop cluster on 3 ec2 m4.2xlarge instances. For few months it has worked fine. But later on i found that node managers are gettin shutdown. I have tried few solutions mentioned in this area like deleting: /var/log/hadoop-yarn/nodemanager/recovery-state/ and triying to restart again. But no luck. On all nodes i found that cpu usage went up to 600%: top - 08:31:21 up 3:35, 1 user, load average: 10.42, 10.60, 10.81 Tasks: 155 total, 2 running, 153 sleeping, 0 stopped, 0 zombie %Cpu(s): 99.9 us, 0.1 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 32780760 total, 14747292 free, 15611484 used, 2421984 buff/cache KiB Swap: 0 total, 0 free, 0 used. 16513908 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20737 yarn 20 0 314724 22880 392 S 780.1 0.1 1401:35 java 1576 root 20 0 8040888 1.226g 13420 S 16.2 3.9 23:19.59 java 5189 hdfs 20 0 2946168 408120 24552 S 0.7 1.2 0:35.23 java [root@ip-172-31-37-158 ~]# ps -ef | grep 20737 yarn 20737 1 99 05:26 ? 23:24:42 /var/tmp/java -c /var/tmp/w.conf Not sure what is the process belonging to "/var/tmp/java -c /var/tmp/w.conf". In the Nodemanager logs i could see below error messages: /var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-ip-172-31-37-158.us-east-2.compute.internal.log.1:2018-06-06 03:12:24,348 ERROR filecontroller.LogAggregationFileController (LogAggregationFileController.java:run(360)) - Failed to setup application log directory for application_1528245572429_0439 /var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-ip-172-31-37-158.us-east-2.compute.internal.log.1:2018-06-07 02:53:24,909 ERROR nodemanager.NodeManager (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM In the ambari UI, i could see below alerts on all machines: Connection failed to http://ip-172-31-37-158.us-east-2.compute.internal:8042 (<urlopen error [Errno 111] Connection refused>) Connection failed to http://ip-172-31-37-158.us-east-2.compute.internal:8042/ws/v1/node/info (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanager_health.py", line 171, in execute url_response = urllib2.urlopen(query, timeout=connection_timeout) File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/usr/lib64/python2.7/urllib2.py", line 449, in _open '_open', req) File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/usr/lib64/python2.7/urllib2.py", line 1244, in http_open return self.do_open(httplib.HTTPConnection, req) File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open raise URLError(err) URLError: <urlopen error [Errno 111] Connection refused> ) ... View more

Hi Team, I am using HDP sandbox vm (version 2.5). I have stared kafka server using below command: $/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-server-start.sh /usr/hdp/2.5.0.0-1245/kafka/config/server.properties Created a new topic: $/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-topics.sh --create --zookeeper sandbox.hortonworks.com:2181 --replication-factor 1 --partitions 1 --topic Hello-Kafka Started console producer: $/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-console-producer.sh --broker-list sandbox.hortonworks.com:9092 --topic Hello-Kafka Started console consumer: $/usr/hdp/2.5.0.0-1245/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181 --topic Hello-Kafka --from-beginning Now i could see that all the messages created at producer are getting consumed properly by consumer. I tried to create a spark kafka streaming application with code mentioned at https://www.tutorialspoint.com/apache_kafka/apache_kafka_integration_spark.htm Below is my project directory structure: [root@sandbox kafkawordcount]# find . . ./build.sbt ./src ./src/main ./src/main/KafkaWordCount.scala Below are build.sbt contents: [root@sandbox kafkawordcount]# cat build.sbt name := "Spark Kafka Project" version := "1.0" scalaVersion := "2.10.5" libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.2" libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.2" libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" % "1.6.2" When i triggered $sbt package, i found spark-kafka-project_2.10-1.0.jar got created. But it is not having any class files in it. Not sure what went wrong here. Below is the output of $sbt package comand. Please help to resolve this problem. [info] Updated file /root/kafkawordcount/project/build.properties: set sbt.version to 1.0.2 [info] Loading project definition from /root/kafkawordcount/project [info] Updating {file:/root/kafkawordcount/project/}kafkawordcount-build... [info] Done updating. [info] Loading settings from build.sbt ... [info] Set current project to Spark Kafka Project (in build file:/root/kafkawordcount/) [info] Updating {file:/root/kafkawordcount/}kafkawordcount... [info] Done updating. [warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible: [warn] * commons-net:commons-net:2.2 is selected over 3.1 [warn] +- org.apache.spark:spark-core_2.10:1.6.2 (depends on 2.2) [warn] +- org.apache.hadoop:hadoop-common:2.2.0 (depends on 3.1) [warn] * com.google.guava:guava:14.0.1 is selected over 11.0.2 [warn] +- org.apache.curator:curator-recipes:2.4.0 (depends on 14.0.1) [warn] +- org.tachyonproject:tachyon-client:0.8.2 (depends on 14.0.1) [warn] +- org.apache.curator:curator-client:2.4.0 (depends on 14.0.1) [warn] +- org.tachyonproject:tachyon-underfs-hdfs:0.8.2 (depends on 14.0.1) [warn] +- org.apache.curator:curator-framework:2.4.0 (depends on 14.0.1) [warn] +- org.tachyonproject:tachyon-underfs-s3:0.8.2 (depends on 14.0.1) [warn] +- org.tachyonproject:tachyon-underfs-local:0.8.2 (depends on 14.0.1) [warn] +- org.apache.hadoop:hadoop-hdfs:2.2.0 (depends on 11.0.2) [warn] +- org.apache.hadoop:hadoop-common:2.2.0 (depends on 11.0.2) [warn] * com.google.code.findbugs:jsr305:1.3.9 is selected over 2.0.1 [warn] +- com.google.guava:guava:11.0.2 (depends on 1.3.9) [warn] +- org.apache.spark:spark-network-common_2.10:1.6.2 (depends on 1.3.9) [warn] +- org.apache.spark:spark-unsafe_2.10:1.6.2 (depends on 1.3.9) [warn] +- org.apache.spark:spark-core_2.10:1.6.2 (depends on 1.3.9) [warn] +- com.fasterxml.jackson.module:jackson-module-scala_2.10:2.4.4 (depends on 2.0.1) [warn] Run 'evicted' to see detailed eviction warnings [info] Packaging /root/kafkawordcount/target/scala-2.10/spark-kafka-project_2.10-1.0.jar ... [info] Done packaging. [success] Total time: 10 s, completed Oct 9, 2017 2:38:12 AM Thanks & Regards, Srinivas. ... View more