Hi all, As mentioned from the title i'm trying to run a shell action that kicks off a spark job but unfortunately i'm consistently getting the following error... 19/05/10 14:03:39 ERROR AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] java.io.IOException: Could not set up IO Streams to <hbaseregionserver> Fri May 10 14:03:39 BST 2019, RpcRetryingCaller{globalStartTime=1557493419339, pause=100, retries=2}, org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: <hbaseregionserver> Been playing around trying to get the script to take in the kerberos ticket but having no luck, as far as I can tell its related to the Oozie job not being able to pass the kerberos ticket any ideas why its not picking it up? I'm at a loss? Related code is below Oozie workflow action  <action name="sparkJ" cred="hive2Cred">         <shell xmlns="uri:oozie:shell-action:0.1">                 <job-tracker>${jobTracker}</job-tracker>                 <name-node>${nameNode}</name-node>                 <configuration>                         <property>                                 <name>mapred.job.queue.name</name>                                 <value>${oozieQueueName}</value>                         </property>                 </configuration>                 <exec>run.sh</exec>                 <file>/thePathToTheScript/run.sh#run.sh</file>                 <file>/thePathToTheProperties/myp.properties#myp.properties</file>                 <capture-output />         </shell>                 <ok to="end" />                 <error to="fail" />         </action> Shell script #!/bin/sh export job_name=SPARK_JOB export configuration=myp.properties export num_executors=10 export executor_memory=1G export queue=YARNQ export max_executors=50 kinit -kt KEYTAB KPRINCIPAL echo "[[[[[[[[[[[[[ Starting Job - name:${job_name}, configuration:${configuration} ]]]]]]]]]]]]]]" /usr/hdp/current/spark2-client/bin/spark-submit \ --name ${job_name} \ --driver-java-options "-Dlog4j.configuration=file:./log4j.properties" \ --num-executors ${num_executors} \ --executor-memory ${executor_memory} \ --master yarn \ --keytab KEYTAB \ --principal KPRINCIPAL \ --supervise \ --deploy-mode cluster \ --queue ${queue} \ --files "./${configuration},./hbase-site.xml,./log4j.properties" \ --conf spark.driver.extraClassPath="/usr/hdp/current/hive-client/lib/datanucleus-*.jar:/usr/hdp/current/tez-client/*.jar" \ --conf spark.executor.extraJavaOptions="-Djava.security.auth.login.config=./jaas.conf -Dlog4j.configuration=file:./log4j.properties"  \ --conf spark.executor.extraClassPath="/usr/hdp/current/hive-client/lib/datanucleus-*.jar:/usr/hdp/current/tez-client/*.jar" \ --conf spark.streaming.stopGracefullyOnShutdown=true \ --conf spark.dynamicAllocation.enabled=true \ --conf spark.shuffle.service.enabled=true \ --conf spark.dynamicAllocation.maxExecutors=${max_executors} \ --conf spark.streaming.concurrentJobs=2 \ --conf spark.streaming.backpressure.enabled=true \ --conf spark.yarn.security.tokens.hive.enabled=true \ --conf spark.yarn.security.tokens.hbase.enabled=true \ --conf spark.streaming.kafka.maxRatePerPartition=5000 \ --conf spark.streaming.backpressure.pid.maxRate=3000 \ --conf spark.streaming.backpressure.pid.minRate=200 \ --conf spark.streaming.backpressure.initialRate=5000 \ --jars /usr/hdp/current/hbase-client/lib/guava-12.0.1.jar,/usr/hdp/current/hbase-client/lib/hbase-common.jar,/usr/hdp/current/hbase-client/lib/hbase-client.jar,/usr/hdp/current/hbase-client/lib/hbase-protocol.jar,/usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar \ --class myclass myjar.jar ./${configuration} Many thanks to any help you can provide. ... View more

Hi all, Witnessing some unexpected behavior i'm hoping someone might be able to clarify for me. I have a spark job which we set the zookeeper root to "/example" and the consumer group to "example_group", the offset here is 2:101 1:104 0:106 Using the same consumer group i switched the zookeeper root to "/example_temp" the idea here being that we can continue testing our job on "/example_temp" while the messages are still sent to the kafka topic and when we switch back to "/example" we'll ingest all the messages since the last offset for "/example", allowing us to test a larger load. Now the weird bit is that the offset in "/example_temp" is lower than that of "/example" 2:65 1:67 0:66 And so when I switched back to "/example", i got this error java.lang.IllegalArgumentException: requirement failed: numRecords must not be negative Because the offset saved on the broker is that of "/example_temp" and this is lower than the value saved in zookeeper for "/example" if you're still with me, my question is how did the offset for "/example_temp" end up lower than that of "/example"? ... View more

Hi all, Related to this question which is a tad outdated , is it now possible to connect from Apache Storm to IBM MQ queues? Hortonworks version 2.6.0 Storm version:1.1.2 Thanks, Dan ... View more

Hi all, I'm looking into the rerun functionality within Oozie coordinators, as per the documentation snippet below I know you can rerun all jobs within an oozie coordinator between a certain time frame or for certain actions. $oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-action 1, 3-4, 7-40] (-action or -date is required to rerun.) [-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z] (if neither -action nor -date is given, the exception will be thrown.) Either -action or -date should be given.If -action and -date both are given, an error will be thrown.Multiple ranges can be used in -action or -date. See the above examples.If one of action in the given list of -action does not exist or not in terminal state, the rerun throws an error.The dates specified in -date must be UTC.Single date specified in -date must be able to find an action with matched nominal time to be effective.If -nocleanup is given, coordinator directories will not be removed; otherwise the 'output-event' will be deleted.If -refresh is set, new dataset is re-evaluated for latest() and future().If -refresh is set, all dependencies will be re-checked; otherwise only missed dependencies will be checked. My question is can you specify that in that time frame you would only like to rerun jobs that have a status of KILLED or FAILED, otherwise it is ignored e.g. SUCCEEDED. Is this a functionality that exists or will I have to write something up in a shell script to retrieve these failed jobs ID's and then rerun them? Any help provided is much appreciated, Thanks, Dan ... View more

Hi all, Bit of background, I have been struggling with an issue in which a table can get locked but the lock gets stuck in a "waiting" state and so does not allow the query to progress, after about 2 hours the job complains that the table is locked and it cannot access it. I'm trying to identify why the table is getting locked which brings me to my question.... If a user runs a query but then disconnects the session can this cause a table to get locked in this state? | lockid | database | table | partition | lock_state | blocked_by | lock_type | transaction_id | last_heartbeat | acquired_at | user | hostname | agent_info | 78011111.2 | db | tbl | NULL | WAITING | 78043210.2 | EXCLUSIVE | NULL | 1523459564452 | NULL | user | host | hive_20180411161225_2b33e811-e44c-59ds-afb3-b4111fcb019a | ... View more

Hi all, Fairly straightforward question, in the Apache Falcon documentation it specifies "The libraries required for the workflow should be available in lib folder in workflow path." I have several workflows which have a dependency on a particularly large jar and so if possible I wanted to avoid duplicating it accross the file system, is there a property we can use in falcon to define this lib location? Thanks, Dan ... View more

Hi all, I was hoping someone could confirm whether something i'm trying to do is possible because i'm currently hitting multiple issues. I would like to add a record into a hive table using and insert statement, within this insert statement I have one column which should add a count value to the table based off of the result of a query. My Hive SQL is below.... use ${database}; set hivevar:deltaCount = select count(*) from ${database}.${hive_table}; DROP TABLE IF EXISTS ${database}.process_status_stg_${hive_table}; create table ${database}.process_status_stg_${hive_table} ( taskName varchar(50) COMMENT 'Name of the task being run to populate data', starttime varchar(50) COMMENT 'time of record addition', status varchar(50) COMMENT 'status of the task', workflowID varchar(50) COMMENT 'workflow ID that is running the task', oozieErrorCode varchar(50) COMMENT 'Error code returned by Oozie', recordsLoadedCount varchar(50) COMMENT 'records pulled in previous load') ; insert into table ${database}.process_status_stg_${hive_table} values ('${hive_table}','${current_time}','${taskStatus}','${workflowID}','${errorCode}', (CASE ${taskStatus} WHEN 'COMPLETED' THEN '${hiveconf:deltaCount}' ELSE 'N/A' end as recordsLoadedCount)); Any help is much appreciated, Thanks ... View more

This question relates to the usage of Apache Falcon any help provided is much appreciated. Hopefully this is a query with a very easy resolution. Our falcon process is trying to define an argument which will be passed to an oozie workflow and then subsequently into an SQL script. This value is a date value which should be passed in the format YYYYMMDDHH. Currently we're trying to do this by defining as the below as you would in a Falcon Feed. <property name ="partitiondate" value="${YEAR}${MONTH}${DAY}${HOUR}"/> But this gives an error, CausedBy: E1004: Expression language evaluation error, Unable to evaluate :${YEAR}${MONTH}${DAY}${HOUR} Is there another way to pass this value? Thanks again for any help. ... View more

Is there any way I can provide multiple JDBC connections within the Sqoop command? For example if JDBC1 is down then Sqoop should pick up JDBC2 and run the import/export. For example: sqoop export --connect "jdbc string 1 / jdbc string 2" It should then pick the correct JDBC URL based on which on is up and running Is this possible? Thanks for any help you can provide, Dan ... View more

Hi all, I'm trying to schedule a job using falcon which will run everytime a new partition is created in Hive, this part appears to be working, however whenever it runs it complains that it is unable to find a variable despite this variable having been declared in the code below variable [inputFeedCCTM] cannot be resolved <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <process name="processLink" xmlns="uri:falcon:process:0.1"> <tags>owner=owner</tags> <pipelines>***</pipelines> <clusters> <cluster name="cluster"> <validity start="2016-07-16T09:00Z" end="2033-01-13T20:00Z"/> </cluster> </clusters> <parallel>1</parallel> <order>FIFO</order> <timeout>hours(1)</timeout> <frequency>days(1)</frequency> <timezone>UTC</timezone> <inputs> <input name="inputFeedLink" feed="feedInLink" start="yesterday(0,0)" end="yesterday(0,0)"/> <input name="inputFeedLink2" feed="feedInLink2" start="yesterday(0,0)" end="yesterday(0,0)"/> <input name="inputFeedLink3" feed="feedInLink3" start="yesterday(0,0)" end="yesterday(0,0)"/> </inputs> <!-- <outputs> <output name="outputFeed" feed="feedOutLink" instance="yesterday(0,0)"/> </outputs>--> <properties> <property name="table" value="linked"/> <property name="appPath" value="/path"/> <property name="workflowAppPath" value="/wfPath"/> </properties> <workflow name="falcon-wf" version="5.0" engine="oozie" path="/path/workflow"/> <retry attempts="2" delay="hours(1)" policy="exp-backoff"/> <ACL owner="Owner" group="hadoop" permission="0x775"/> </process> In the Oozie workflow the variable is being referenced first inside a shell action like so <argument>${inputFeedLink}</argument> Any help is much appreciated Thanks, Dan ... View more

Hi all, As the title suggests i'm trying to work out how to use Hbase within an Oozie workflow. Resources I've found online regarding this subject have been very limited. Do I need to use a Java action within my Oozie workflow in order to use Hbase? I require to create a table which will be used to monitor when my data pulls for a specific table are running/completed/failed, which we are referring to as the process status table. Currently to do this I am inserting a new row to a hive table before the relevant actions start and then once the data pull is completed a new row is inserted depending on whether or not the actions were all successful. I believe that using Hbase instead of Hive would be a much more efficient way to add to this table as only one row is added at a time. Any help provided would be much appreciated ... View more