Member since
09-01-2018
17
Posts
0
Kudos Received
0
Solutions
01-10-2019
08:51 PM
I don't know if this has anything to do with different pyspark version. I have version 2.3.0. Finally, this format worked for me. pyspark2 --driver-class-path "/home/jar/*" --jars /home/jar/noarch-aster-jdbc-driver.jar,/home/jar/sqljdbc42.jar
... View more
01-10-2019
07:16 PM
Hi, I am trying to pull SQL Server data into a Data Frame using pyspark. I'd like to use windows authentication to establish the connection. Here is my connection string: jdbc:sqlserver://servername;databasename=Test;schema=dbo;integratedSecurity=true; However, I am getting an error not able to find sqljdbc_auth.dll "An error occurred while calling o63.jdbc.
: com.microsoft.sqlserver.jdbc.SQLServerException: This driver is not configured for integrated authentication." Is there any workaround so I can connect to SQL server with windows authentication within pyspark ?
... View more
Labels:
01-09-2019
11:17 PM
How can I add multiple jars to --driver-class-path. It works when I add 1 jar file, but does not work when I try to add multiple jars using a comma separated format. Here is the shell command I am using. It does not pick up the second jar file. pyspark2 --driver-class-path /home/jar/noarch-aster-jdbc-driver.jar,/home/jar/sqljdbc42.jar --jars /home/jar/noarch-aster-jdbc-driver.jar,/home/jar/sqljdbc42.jar ?? Thanks.
... View more
Labels:
10-10-2018
05:38 AM
Hello, I am unable to run sqoop with Oozie. I am able to get a successful run using the shell, but keeps getting the same error using Oozie. I tried researching the error but nothing works so far. This is the HDP sandbox and I thought the hive metastore would be pre-configured with Derby ?? Any guidance will be very helpful. Thanks. 2018-10-10 02:10:30,917 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2018-10-10 02:10:30,965 [main] INFO org.apache.hive.hcatalog.common.HiveClientCache - Initializing cache: eviction-timeout=120 initial-capacity=50 maximum-capacity=50
2018-10-10 02:10:31,349 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2018-10-10 02:10:31,350 [main] INFO org.apache.hadoop.hive.metastore.HiveMetaStore - 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
2018-10-10 02:10:31,396 [main] INFO org.apache.hadoop.hive.metastore.ObjectStore - ObjectStore, initialize called
2018-10-10 02:10:31,639 [main] INFO DataNucleus.Persistence - Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
2018-10-10 02:10:31,641 [main] INFO DataNucleus.Persistence - Property datanucleus.cache.level2 unknown - will be ignored
2018-10-10 02:10:33,961 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2018-10-10 02:10:33,963 [main] INFO org.apache.hadoop.hive.metastore.ObjectStore - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Databa
se,Type,FieldSchema,Order"
2018-10-10 02:10:34,138 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2018-10-10 02:10:35,513 [main] WARN DataNucleus.Query - Query for candidates of org.apache.hadoop.hive.metastore.model.MDatabase and subclasses resulted in no possible candidates
Required table missing : "DBS" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCre
ateTables"
org.datanucleus.store.rdbms.exceptions.MissingTableException: Required table missing : "DBS" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData i
s incorrect, or you need to enable "datanucleus.schema.autoCreateTables"
Here is the workflow <action name="sqoop_import">
<sqoop xmlns="uri:oozie:sqoop-action:0.4">
<job-tracker>${resourceManager}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>oozie.action.sharelib.for.sqoop</name>
<value>sqoop,hive,hcatalog</value>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
</property>
</configuration>
<arg>export</arg>
<arg>--connect</arg>
<arg>jdbc:sqlserver://dev01;databasename=Test;schema=dbo;</arg>
<arg>--username</arg>
<arg>***</arg>
<arg>--password</arg>
<arg>***</arg>
<arg>--table</arg>
<arg>test_summary</arg>
<arg>--hcatalog-database</arg>
<arg>Test</arg>
<arg>--hcatalog-table</arg>
<arg>test_summary</arg>
</sqoop>
<ok to="end"/>
<error to="kill"/>
</action>
... View more
Labels:
10-07-2018
09:31 AM
I've been getting this error and unable to identify the root cause. I've tried copying the jar file to the libpath as specified in the error log below "user/root/lib" but no luck. I run the exact same command in the shell and it works fine. It only fails when using Oozie. Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] Any guidance ? Thanks! Error Log: 2018-10-07 05:35:05,066 WARN SqoopActionExecutor:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000441-180930024054767-oozie-oozi-W] ACTION[0000441-180930024054767-oozie-oozi-W@sqoop_truncate] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
2018-10-07 05:52:56,279 WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000441-180930024054767-oozie-oozi-W] ACTION[] The application does not define formal parameters in its XML definition
2018-10-07 05:52:56,291 WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000441-180930024054767-oozie-oozi-W] ACTION[] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/root/lib] does not exist
2018-10-07 05:54:08,671 WARN SqoopActionExecutor:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000441-180930024054767-oozie-oozi-W] ACTION[0000441-180930024054767-oozie-oozi-W@sqoop_truncate] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
2018-10-07 05:59:37,381 WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000441-180930024054767-oozie-oozi-W] ACTION[] The application does not define formal parameters in its XML definition
2018-10-07 06:00:58,772 WARN SqoopActionExecutor:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000441-180930024054767-oozie-oozi-W] ACTION[0000441-180930024054767-oozie-oozi-W@sqoop_truncate] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
... View more
Labels:
10-05-2018
02:40 AM
OK. I got it working. It is because I have "space" with my script path: <script>/user/root/Hive Query Test.hql</script> So I simply removed the space in the script name to HiveQueryTest.hql and now it works. I was hoping the error message would be more clear and easier to debug.
... View more
10-05-2018
01:49 AM
@AnjiReddy Anumolu I tried your workflow but still got the same error. I suspect it is a configuration issue ? but don't know what it is. Here is the job properties: Name Value oozie.use.system.libpath false oozie.wf.application.path hdfs://sandbox-hdp.hortonworks.com:8020/user/root/TestWorkflow.xml user.name admin jobTracker sandbox-hdp.hortonworks.com:8032 oozie.wf.rerun.failnodes false mapreduce.job.user.name admin resourceManager sandbox-hdp.hortonworks.com:8032 nameNode hdfs://sandbox-hdp.hortonworks.com:8020 Here is the log from /var/log/oozie 2018-10-05 01:03:40,167 WARN V2JobServlet:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[PUT http://sandbox-hdp.hortonworks.com:11000/oozie/v2/job/0000138-180930024054767-oozie-oozi-W?action=rerun&user.name=admin] error[E0302], E0302: Invalid parameter [Multiple app paths specified, only one is allowed]
org.apache.oozie.servlet.XServletException: E0302: Invalid parameter [Multiple app paths specified, only one is allowed]
at org.apache.oozie.servlet.ServletUtilities.ValidateAppPath(ServletUtilities.java:78)
at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:257)
at org.apache.oozie.servlet.BaseJobServlet.doPut(BaseJobServlet.java:133)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:730)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:304)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:617)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:576)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.OozieXFrameOptionsFilter.doFilter(OozieXFrameOptionsFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.OozieCSRFFilter.doFilter(OozieCSRFFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:234)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503)
at java.lang.Thread.run(Thread.java:748)
2018-10-05 01:03:59,683 WARN V2JobServlet:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[PUT http://sandbox-hdp.hortonworks.com:11000/oozie/v2/job/0000138-180930024054767-oozie-oozi-W?action=rerun&user.name=admin] error[E0302], E0302: Invalid parameter [Multiple app paths specified, only one is allowed]
org.apache.oozie.servlet.XServletException: E0302: Invalid parameter [Multiple app paths specified, only one is allowed]
at org.apache.oozie.servlet.ServletUtilities.ValidateAppPath(ServletUtilities.java:78)
at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:257)
at org.apache.oozie.servlet.BaseJobServlet.doPut(BaseJobServlet.java:133)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:730)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:304)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:617)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:576)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.OozieXFrameOptionsFilter.doFilter(OozieXFrameOptionsFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.OozieCSRFFilter.doFilter(OozieCSRFFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:234)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503)
at java.lang.Thread.run(Thread.java:748)
2018-10-05 01:05:14,732 WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2018-10-05 01:05:14,742 WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/root/lib] does not exist
2018-10-05 01:05:22,207 WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2018-10-05 01:05:22,217 WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/root/lib] does not exist
2018-10-05 01:05:42,256 WARN ActionStartXCommand:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000140-180930024054767-oozie-oozi-W] ACTION[0000140-180930024054767-oozie-oozi-W@hive_1] Error starting action [hive_1]. ErrorType [ERROR], ErrorCode [NullPointerException], Message [NullPointerException: null]
org.apache.oozie.action.ActionExecutorException: NullPointerException: null
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1258)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1440)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:459)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:82)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:280)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:61)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
2018-10-05 01:05:42,259 WARN ActionStartXCommand:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000140-180930024054767-oozie-oozi-W] ACTION[0000140-180930024054767-oozie-oozi-W@hive_1] Setting Action Status to [DONE]
2018-10-05 01:09:09,691 WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2018-10-05 01:09:09,700 WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/root/lib] does not exist
2018-10-05 01:09:18,269 WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2018-10-05 01:09:18,276 WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/root/lib] does not exist
2018-10-05 01:09:45,389 WARN ActionStartXCommand:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000142-180930024054767-oozie-oozi-W] ACTION[0000142-180930024054767-oozie-oozi-W@hive_1] Error starting action [hive_1]. ErrorType [ERROR], ErrorCode [NullPointerException], Message [NullPointerException: null]
org.apache.oozie.action.ActionExecutorException: NullPointerException: null
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1258)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1440)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:459)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:82)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:280)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:61)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
2018-10-05 01:09:45,391 WARN ActionStartXCommand:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000142-180930024054767-oozie-oozi-W] ACTION[0000142-180930024054767-oozie-oozi-W@hive_1] Setting Action Status to [DONE]
2018-10-05 01:11:11,553 WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2018-10-05 01:11:11,561 WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/root/lib] does not exist
2018-10-05 01:11:21,539 WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2018-10-05 01:11:21,558 WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/root/lib] does not exist
2018-10-05 01:11:40,347 WARN ActionStartXCommand:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000144-180930024054767-oozie-oozi-W] ACTION[0000144-180930024054767-oozie-oozi-W@hive_1] Error starting action [hive_1]. ErrorType [ERROR], ErrorCode [NullPointerException], Message [NullPointerException: null]
org.apache.oozie.action.ActionExecutorException: NullPointerException: null
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1091)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1188)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1440)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:459)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:82)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:280)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:61)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
2018-10-05 01:11:40,352 WARN ActionStartXCommand:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000144-180930024054767-oozie-oozi-W] ACTION[0000144-180930024054767-oozie-oozi-W@hive_1] Setting Action Status to [DONE]
2018-10-05 01:18:41,743 WARN V2JobServlet:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[GET http://sandbox-hdp-security:11000/oozie/v2/job/0000138-180930024054767-oozie-oozi-W?action=rerun&user.name=admin] error[E0302], E0302: Invalid parameter [action]
org.apache.oozie.servlet.XServletException: E0302: Invalid parameter [action]
at org.apache.oozie.servlet.JsonRestServlet.validateRestUrl(JsonRestServlet.java:448)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:294)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:617)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:576)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.OozieXFrameOptionsFilter.doFilter(OozieXFrameOptionsFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.OozieCSRFFilter.doFilter(OozieCSRFFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:234)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503)
at java.lang.Thread.run(Thread.java:748)
2018-10-05 01:18:51,416 WARN V2JobServlet:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[GET http://sandbox-hdp-security:11000/oozie/v2/job/0000138-180930024054767-oozie-oozi-W?action=rerun&user.name=admin] error[E0302], E0302: Invalid parameter [action]
org.apache.oozie.servlet.XServletException: E0302: Invalid parameter [action]
at org.apache.oozie.servlet.JsonRestServlet.validateRestUrl(JsonRestServlet.java:448)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:294)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:617)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:576)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.OozieXFrameOptionsFilter.doFilter(OozieXFrameOptionsFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.OozieCSRFFilter.doFilter(OozieCSRFFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:234)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:610)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503)
at java.lang.Thread.run(Thread.java:748)
2018-10-05 01:21:46,339 WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000144-180930024054767-oozie-oozi-W] ACTION[] The application does not define formal parameters in its XML definition
2018-10-05 01:21:46,346 WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000144-180930024054767-oozie-oozi-W] ACTION[] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/root/lib] does not exist
2018-10-05 01:22:05,542 WARN ActionStartXCommand:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000144-180930024054767-oozie-oozi-W] ACTION[0000144-180930024054767-oozie-oozi-W@hive_1] Error starting action [hive_1]. ErrorType [ERROR], ErrorCode [NullPointerException], Message [NullPointerException: null]
org.apache.oozie.action.ActionExecutorException: NullPointerException: null
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1091)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1188)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1440)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:459)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:82)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:280)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:61)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
2018-10-05 01:22:05,542 WARN ActionStartXCommand:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000144-180930024054767-oozie-oozi-W] ACTION[0000144-180930024054767-oozie-oozi-W@hive_1] Setting Action Status to [DONE]
2018-10-05 01:24:21,892 WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2018-10-05 01:24:21,899 WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/root/lib] does not exist
2018-10-05 01:24:46,623 WARN ActionStartXCommand:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000145-180930024054767-oozie-oozi-W] ACTION[0000145-180930024054767-oozie-oozi-W@hive_1] Error starting action [hive_1]. ErrorType [ERROR], ErrorCode [NullPointerException], Message [NullPointerException: null]
org.apache.oozie.action.ActionExecutorException: NullPointerException: null
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1091)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1188)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1440)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:459)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:82)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:280)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:61)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
2018-10-05 01:24:46,635 WARN ActionStartXCommand:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000145-180930024054767-oozie-oozi-W] ACTION[0000145-180930024054767-oozie-oozi-W@hive_1] Setting Action Status to [DONE]
... View more
10-03-2018
05:28 PM
Hello, I am trying to run a HIVE job with Oozie and keeps getting the same error message. The hql script works fine when running in Hive, so I am sure the problem is not the script, but it's in Oozie. I am using the sandbox for this. Any guidance will be great. I can't find much tutorial for running oozie-hive on the sandbox and the only tutorial I can find doesn't work. WARN ActionStartXCommand:523 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[TestWorkflow] JOB[0000081-180930024054767-oozie-oozi-W] ACTION[0000081-180930024054767-oozie-oozi-W@hive_1] Error starting action [hive_1]. ErrorType [ERROR], ErrorCode [NullPointerException], Message [NullPointerException: null]
org.apache.oozie.action.ActionExecutorException: NullPointerException: null
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1091)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1188)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1440)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:459)
at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:82)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:280)
at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:61)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:640)
at org.apache.oozie.action.hadoop.ScriptLanguageActionExecutor.addScriptToCache(ScriptLanguageActionExecutor.java:97)
at org.apache.oozie.action.hadoop.ScriptLanguageActionExecutor.setupLauncherConf(ScriptLanguageActionExecutor.java:55)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:973) <?xml version="1.0" encoding="UTF-8" standalone="no"?><workflow-app xmlns="uri:oozie:workflow:0.5" name="TestWorkflow">
<start to="hive_1"/>
<action name="hive_1">
<hive xmlns="uri:oozie:hive-action:0.6">
<job-tracker>${resourceManager}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>/user/root/hive-site.xml</job-xml>
<script>/user/root/Hive Query Test.hql</script>
</hive>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>${wf:errorMessage(wf:lastErrorNode())}</message>
</kill>
<end name="end"/>
</workflow-app>
... View more
Labels:
09-27-2018
03:31 AM
Hello, how do you format date like this in HIVE ? from: Tue Sep 26 22:02:11 CDT 2018 to: 2018-09-26 22:02:11 I've tried different format specifier but didn't work. I must have done something wrong. Any help would be helpful. Thanks
... View more
- Tags:
- Data Processing
- Hive
Labels:
09-16-2018
06:54 PM
it ran outside of oozie using spark-submit successfully. just not in oozie.
... View more
09-15-2018
08:53 PM
I have a similar issue. I tried using port 8032 and 8050 for jobtracker. Didn't work. Anyone have this issue resolved ? How did you resolve it ? YARN keeps stating: ACCEPTED: waiting for AM container to be allocated, launched and register with RM. Diagnostics: [Sat Sep 15 02:49:30 +0000 2018] Application is Activated, waiting for resources to be assigned for AM. Details : AM Partition = <DEFAULT_PARTITION> ; Partition Resource = <memory:3000, vCores:8> ; Queue's Absolute capacity = 14.0 % ; Queue's Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 14.0 % ; Job always in "RUNNING" state and never finish.
... View more
09-15-2018
05:04 PM
Still doesn't work. 😞
... View more
09-10-2018
04:23 PM
Hi @Vinicius Higa Murakami. <workflow-app name="spark test"
xmlns="uri:oozie:workflow:0.5">
<start to="spark_1"/>
<action name="spark_1">
<spark
xmlns="uri:oozie:spark-action:0.2">
<job-tracker>${resourceManager}</job-tracker>
<name-node>${nameNode}</name-node>
<master>yarn-cluster</master>
<name>pySpark</name>
<jar>/tmp/pySparkTest.py</jar>
</spark>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>${wf:errorMessage(wf:lastErrorNode())}</message>
</kill>
<end name="end"/>
</workflow-app>
... View more
09-10-2018
04:10 PM
Hi @Vinicius Higa Murakami I guess this is the YARN UI ( http://[**My HDP Sandbox**]:8188/applicationhistory ) ?? If it's not correct please let me know. I am not sure what the message means or how to resolve.
... View more
09-10-2018
12:08 PM
Hello, I have new to HDP. I have set up a very simple oozie job to run spark on the sandbox. Job started but never continue to execute the next step and it just stuck there with a "running" status. I am able to run spark-submit using the terminal so I know the spark script works. Any idea why the job is stuck ? or what other steps I can take to troubleshoot this ? Here is the log: 2018-09-10 02:24:30,295 INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@:start:] Start action [0000014-180908024054802-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-09-10 02:24:30,296 INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@:start:] [***0000014-180908024054802-oozie-oozi-W@:start:***]Action status=DONE
2018-09-10 02:24:30,296 INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@:start:] [***0000014-180908024054802-oozie-oozi-W@:start:***]Action updated in DB!
2018-09-10 02:24:30,328 INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000014-180908024054802-oozie-oozi-W@:start:
2018-09-10 02:24:30,328 INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000014-180908024054802-oozie-oozi-W
2018-09-10 02:24:30,343 INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@spark_1] Start action [0000014-180908024054802-oozie-oozi-W@spark_1] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-09-10 02:24:30,370 INFO SparkActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@spark_1] Added into spark action configuration mapred.child.env=SPARK_HOME=.,HDP_VERSION=2.6.5.0-292
2018-09-10 02:24:31,425 INFO SparkActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@spark_1] Trying to get job [job_1536377145689_0006], attempt [1]
2018-09-10 02:24:31,450 INFO SparkActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@spark_1] checking action, hadoop job ID [job_1536377145689_0006] status [RUNNING]
2018-09-10 02:24:31,453 INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@spark_1] [***0000014-180908024054802-oozie-oozi-W@spark_1***]Action status=RUNNING
2018-09-10 02:24:31,453 INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@spark_1] [***0000014-180908024054802-oozie-oozi-W@spark_1***]Action updated in DB!
2018-09-10 02:24:31,456 INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@spark_1] No Notification URL is defined. Therefore nothing to notify for job 0000014-180908024054802-oozie-oozi-W@spark_1
2018-09-10 02:35:19,084 INFO SparkActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@spark_1] Trying to get job [job_1536377145689_0006], attempt [1]
2018-09-10 02:35:26,289 INFO SparkActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@spark_1] checking action, hadoop job ID [job_1536377145689_0006] status [RUNNING]
2018-09-10 02:46:19,088 INFO SparkActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@spark_1] Trying to get job [job_1536377145689_0006], attempt [1]
2018-09-10 02:46:19,127 INFO SparkActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[spark test] JOB[0000014-180908024054802-oozie-oozi-W] ACTION[0000014-180908024054802-oozie-oozi-W@spark_1] checking action, hadoop job ID [job_1536377145689_0006] status [RUNNING]
... View more
Labels:
09-01-2018
07:22 PM
Thanks. Any suggestion on how to kill query processes in hive ?
... View more
09-01-2018
01:47 PM
Hello, I am using the guided tutorial on the sandbox, specifically "HDP>Develop with Hadoop>Apache Hive" . I am running a small query but it seems stuck and unable to kill the hive processes. I tried to find the hadoop job_id but it doesn't appear any job running. I tried restarting all services but still shows query is running in Tez view. Also, the the dashboard, the "supervisor lines" is showing red now. What does that mean ? Please see screen shots. Any idea how to fix that ? Thanks!
... View more
