Member since
07-04-2017
7
Posts
0
Kudos Received
0
Solutions
04-18-2018
03:05 PM
Hi @Pierre Villard Is there a way to limit the number of threads per tenant on a multi-tenant NiFi Cluster?
... View more
07-12-2017
11:24 AM
Hi Venkat,
I did the above, but the issue still persists on my cluster. Do you have any idea what else might be wrong or what's worth checking?
Basically I get the same error whether I try to run a service check or see the falcon version ERROR: Unable to initialize Falcon Client object. Cause : Could not authenticate, Authentication failed, URL: <a href="http://<host>:15000/api/options?user.name=falcon&user.name=falcon,">http://:15000/api/options?user.name=falcon&user.name=falcon,>:15000/api/options?user.name=falcon&user.name=falcon,</a>><host>:15000/api/options?user.name=falcon&user.name=falcon, status: 503, message: Service Unavailable
In the falcon.application.log there are the following errors. From what I understood, adding the je-5.0.73.jar file should've solved this problem, but it persists and I run out of ideas. 2017-07-12 09:07:42,568 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:49)
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:50)
at org.apache.falcon.metadata.MetadataMappingService.initializeGraphDB(MetadataMappingService.java:146)
at org.apache.falcon.metadata.MetadataMappingService.init(MetadataMappingService.java:113)
at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47)
at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56)
at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:549)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:58)
at org.apache.falcon.FalconServer.main(FalconServer.java:118)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:45)
... 15 more
Caused by: java.lang.NoClassDefFoundError: com/sleepycat/je/LockMode
at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.<clinit>(BerkeleyJEStoreManager.java:47)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:42)
at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:421)
at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:361)
at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1275)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:93)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:73)
... 20 more
Caused by: java.lang.ClassNotFoundException: com.sleepycat.je.LockMode
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 29 more
2017-07-12 09:07:42,569 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@351f2244{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87)
java.lang.RuntimeException: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:59)
...
... View more
07-10-2017
11:21 AM
Hi, I am scheduling an oozie workflow with falcon. The WF is responsible for executing a shell script that runs spark-submit. Sometimes it is working and the jobs end successfully, but most of the time they get killed. In the oozie error logs, there are a few warnings like: 2017-07-06 14:10:01,907 WARN ParameterVerifier:523 - SERVER[<host>] USER[ambari-qa] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_estimatePi7] JOB[0000040-170706133706258-oozie-oozi-W] ACTION[0000040-170706133706258-oozie-oozi-W@user-action] The application does not define formal parameters in its XML definition
2017-07-06 14:10:01,952 WARN LiteWorkflowAppService:523 - SERVER[<host>] USER[ambari-qa] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_estimatePi7] JOB[0000040-170706133706258-oozie-oozi-W] ACTION[0000040-170706133706258-oozie-oozi-W@user-action] libpath [hdfs://<host>:8020/user/oozie/shell/lib] does not exist
2017-07-06 14:10:02,202 WARN CompletedActionXCommand:523 - SERVER[<host>] USER[-] GROUP[-] TOKEN[] APP[-] JOB[0000040-170706133706258-oozie-oozi-W] ACTION[0000040-170706133706258-oozie-oozi-W@user-action] Received early callback for action still in PREP state; will wait [10,000]ms and requeue up to [5] more times
2017-07-07 07:43:10,658 WARN ShellActionExecutor:523 - SERVER[<host>] USER[ambari-qa] GROUP[-] TOKEN[] APP[ShellAction] JOB[0000007-170707072402346-oozie-oozi-W] ACTION[0000007-170707072402346-oozie-oozi-W@shellAction] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
Since they are warnings, I don't think they're the reason why the jobs are failing. Nevertheless I've tried to get rid of them and failed... For the 'Received early callback for action still in PREP state' warning I've added a 30 second sleep in my shell script, but the warning still occurs occasionally.
For the libpath does not exist warning, I've added the following property to the oozie-site.xml oozie.libpath=${nameNode}/user/oozie/share/lib I've also added this to my job.properties file and falcon process. The warning still states the libs are missing from /user/oozie/shell/lib. Are these even related? It seems like oozie is searching for lib in the directory I specified for my workflow.xml file I don't have any idea what to do about the Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] warning. Can't find anything related to it. Does anybody have any idea what might cause the jobs to fail? Below are the configurations files workflow.xml <workflow-app name="ShellAction" xmlns="uri:oozie:workflow:0.4">
<start to="shellAction"/>
<action name="shellAction">
<shell xmlns="uri:oozie:shell-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<exec>script.sh</exec>
<file>/user/oozie/shell/job.properties#job.properties</file>
<file>/user/oozie/shell/script.sh#script.sh</file>
<file>/user/oozie/shell/PiEstimation.jar#PiEstimation.jar</file>
<capture-output/>
</shell>
<ok to="end"/>
<error to="killAction"/>
</action>
<kill name="killAction">
<message>"Killed job due to error"</message>
</kill>
<end name="end"/>
</workflow-app>
job.properties nameNode=hdfs://[<host>]:8020
jobTracker=[<host>]:8050
queueName=default
oozie.wf.application.path=${nameNode}/user/oozie/shell
oozie.libpath=${nameNode}/user/${user.name}/share/lib
oozie.use.system.libpath=true
script.sh sleep 30
/usr/hdp/current/spark-client/bin/spark-submit --class org.apache.falcon.example.spark.SparkPI --conf spark.ui.port=4050 --driver-memory 2g --executor-memory 1g /apps/spark/PiEstimation.jar 100 >> /apps/spark/PiEstimationOut.log
falcon process <process xmlns='uri:falcon:process:0.1' name='estimatePi7'>
<clusters>
<cluster name='primaryCluster'>
<validity start='2017-07-07T07:10Z' end='2017-07-07T07:45Z'/>
</cluster>
</clusters>
<parallel>1</parallel>
<order>LIFO</order>
<frequency>minutes(5)</frequency>
<timezone>UTC</timezone>
<properties>
<property name="oozie.libpath" value="${nameNode}/user/oozie/share/lib" />
</properties>
<workflow name='ShellAction' engine='oozie' path='/user/oozie/shell/'/>
<retry policy='periodic' delay='minutes(1)' attempts='3'/>
<ACL owner='ambari-qa' group='users' permission='0755'/>
</process>
attaching yarn logs syslog.txtstderr.txtlaunch-containersh.txtdirectoryinfo.txt
... View more
Labels:
- Labels:
-
Apache Falcon
-
Apache Oozie