I am trying to deploy a my storm topology on cluster. I am using kafka as the input(kafka Spout).This is the error i am getting java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.Log4jLoggerFactory at org.apache.log4j.Logger.getLogger(Logger.java:39) at kafka.utils.Logging$class.logger(Logging.scala:24) at kafka.consumer.SimpleConsumer.logger$lzycompute(SimpleConsumer.scala:35) at kafka.consumer.SimpleConsumer.logger(SimpleConsumer.scala:35) at kafka.utils.Logging$class.info(Logging.scala:75) at kafka.consumer.SimpleConsumer.info(SimpleConsumer.scala:35) at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:94) at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:83) at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:149) at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79) at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:75) at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:65) at org.apache.storm.kafka.PartitionManager.<init>(PartitionManager.java:94) at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) at org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:129) at org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) My pom.xml looks like this:- <dependencies> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-hdfs</artifactId> <version>1.0.1</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10</artifactId> <version>0.9.0.1</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>0.9.0.1</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-kafka</artifactId> <version>1.0.1</version> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-core</artifactId> <scope>provided</scope> <version>1.0.1</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-auth --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-auth</artifactId> <version>2.2.0</version> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-redis</artifactId> <version>1.0.1</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> </dependencies> ... View more

I am scheduling a falcon job, input feed are two hive tables with feed date as partition, output feed is a join table of the two tables with feed date as partition. The job is scheduled, it shows status as running, but fails to update the table with new values. ClusterDefinition:- <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <cluster name="primaryClusterPoc2" description="This is a primary Cluster" colo="primaryColo" xmlns="uri:falcon:cluster:0.1"> <tags>EntityType=Cluster</tags> <interfaces> <interface type="readonly" endpoint="hftp://sandbox.hortonworks.com:50070" version="2.2.0"/> <interface type="write" endpoint="hdfs://sandbox.hortonworks.com:8020" version="2.2.0"/> <interface type="execute" endpoint="sandbox.hortonworks.com:8050" version="2.2.0"/> <interface type="workflow" endpoint="http://sandbox.hortonworks.com:11000/oozie/" version="4.0.0"/> <interface type="messaging" endpoint="tcp://sandbox.hortonworks.com:61616?daemon=true" version="5.1.6"/> <interface type="registry" endpoint="thrift://sandbox.hortonworks.com:9083" version="0.11.0"/> </interfaces> <locations> <location name="staging" path="/apps/falcon/primaryCluster/staging"/> <location name="temp" path="/tmp"/> <location name="working" path="/apps/falcon/primaryCluster/working"/> </locations> <ACL owner="ambari-qa" group="users" permission="0755"/> </cluster> inputFeed1 :- <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <feed name="inputFeed1" description="hello hive table feed" xmlns="uri:falcon:feed:0.1"> <tags>EntityType=Feed</tags> <frequency>minutes(10)</frequency> <timezone>GMT+02:00</timezone> <late-arrival cut-off="minutes(2)"/> <clusters> <cluster name="primaryClusterPoc2" type="source"> <validity start="2016-09-14T18:31Z" end="2016-09-17T18:45Z"/> <retention limit="hours(1)" action="delete"/> <table uri="catalog:d0_stg_pudb:hellohivetable#ds=${YEAR}-${MONTH}-${DAY}"/> </cluster> </clusters> <table uri="catalog:d0_stg_pudb:hellohivetable#ds=${YEAR}-${MONTH}-${DAY}"/> <ACL owner="id847257" group="hdfs" permission="0755"/> <schema location="hcat" provider="hcat"/> <properties/> </feed> inputFeed2 :- <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <feed name="inputFeed2" description="world hive table feed" xmlns="uri:falcon:feed:0.1"> <tags>EntityType=Feed</tags> <frequency>minutes(10)</frequency> <timezone>GMT+02:00</timezone> <late-arrival cut-off="minutes(2)"/> <clusters> <cluster name="primaryClusterPoc2" type="source"> <validity start="2016-09-14T18:31Z" end="2016-09-17T18:45Z"/> <retention limit="hours(1)" action="delete"/> <table uri="catalog:d0_stg_pudb:worldhivetable#ds=${YEAR}-${MONTH}-${DAY}"/> </cluster> </clusters> <table uri="catalog:d0_stg_pudb:worldhivetable#ds=${YEAR}-${MONTH}-${DAY}"/> <ACL owner="id847257" group="hdfs" permission="0755"/> <schema location="hcat" provider="hcat"/> <properties/> </feed> Process :- <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <process name="tryProcess" xmlns="uri:falcon:process:0.1"> <tags>EntityType=Process</tags> <clusters> <cluster name="primaryClusterPoc2"> <validity start="2016-09-14T18:31Z" end="2016-09-17T18:45Z"/> </cluster> </clusters> <parallel>1</parallel> <order>FIFO</order> <frequency>hours(1)</frequency> <timezone>GMT+02:00</timezone> <inputs> <input name="inputHello" feed="inputFeed1" start="now(0,0)" end="now(24,0)"/> <input name="inputWorld" feed="inputFeed2" start="now(0,0)" end="now(24,0)"/> </inputs> <outputs> <output name="outputWorld" feed="outputFeed" instance="now(0,0)"/> </outputs> <workflow name="hiveJoinWorkflow" version="hive-0.13.1" engine="hive" path="/user/id847257/proposal2/helloWorldHive.hql"/> <retry policy="exp-backoff" delay="minutes(3)" attempts="1"/> <ACL owner="id847257" group="hdfs" permission="0755"/> </process> The hql script run_hive_query.hql I am using is below :- use d0_stg_pudb; INSERT OVERWRITE TABLE helloworldhivetable PARTITION (ds) SELECT helloHiveTable.name as a, worldHiveTable.name as b, DATE_ADD(TO_DATE(from_unixtime(UNIX_TIMESTAMP())),5) as ds from helloHiveTable, worldHiveTable where helloHiveTable.id=worldHiveTable.id ... View more