Member since
01-28-2016
38
Posts
14
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
425 | 03-11-2016 04:16 PM |
05-10-2019
01:21 PM
Hi all, As mentioned from the title i'm trying to run a shell action that kicks off a spark job but unfortunately i'm consistently getting the following error... 19/05/10 14:03:39 ERROR AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] java.io.IOException: Could not set up IO Streams to <hbaseregionserver>
Fri May 10 14:03:39 BST 2019, RpcRetryingCaller{globalStartTime=1557493419339, pause=100, retries=2}, org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: <hbaseregionserver>
Been playing around trying to get the script to take in the kerberos ticket but having no luck, as far as I can tell its related to the Oozie job not being able to pass the kerberos ticket any ideas why its not picking it up? I'm at a loss? Related code is below Oozie workflow action <action name="sparkJ" cred="hive2Cred">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${oozieQueueName}</value>
</property>
</configuration>
<exec>run.sh</exec>
<file>/thePathToTheScript/run.sh#run.sh</file>
<file>/thePathToTheProperties/myp.properties#myp.properties</file>
<capture-output />
</shell>
<ok to="end" />
<error to="fail" />
</action> Shell script
#!/bin/sh
export job_name=SPARK_JOB
export configuration=myp.properties
export num_executors=10
export executor_memory=1G
export queue=YARNQ
export max_executors=50
kinit -kt KEYTAB KPRINCIPAL
echo "[[[[[[[[[[[[[ Starting Job - name:${job_name}, configuration:${configuration} ]]]]]]]]]]]]]]"
/usr/hdp/current/spark2-client/bin/spark-submit \
--name ${job_name} \
--driver-java-options "-Dlog4j.configuration=file:./log4j.properties" \
--num-executors ${num_executors} \
--executor-memory ${executor_memory} \
--master yarn \
--keytab KEYTAB \
--principal KPRINCIPAL \
--supervise \
--deploy-mode cluster \
--queue ${queue} \
--files "./${configuration},./hbase-site.xml,./log4j.properties" \
--conf spark.driver.extraClassPath="/usr/hdp/current/hive-client/lib/datanucleus-*.jar:/usr/hdp/current/tez-client/*.jar" \
--conf spark.executor.extraJavaOptions="-Djava.security.auth.login.config=./jaas.conf -Dlog4j.configuration=file:./log4j.properties" \
--conf spark.executor.extraClassPath="/usr/hdp/current/hive-client/lib/datanucleus-*.jar:/usr/hdp/current/tez-client/*.jar" \
--conf spark.streaming.stopGracefullyOnShutdown=true \
--conf spark.dynamicAllocation.enabled=true \
--conf spark.shuffle.service.enabled=true \
--conf spark.dynamicAllocation.maxExecutors=${max_executors} \
--conf spark.streaming.concurrentJobs=2 \
--conf spark.streaming.backpressure.enabled=true \
--conf spark.yarn.security.tokens.hive.enabled=true \
--conf spark.yarn.security.tokens.hbase.enabled=true \
--conf spark.streaming.kafka.maxRatePerPartition=5000 \
--conf spark.streaming.backpressure.pid.maxRate=3000 \
--conf spark.streaming.backpressure.pid.minRate=200 \
--conf spark.streaming.backpressure.initialRate=5000 \
--jars /usr/hdp/current/hbase-client/lib/guava-12.0.1.jar,/usr/hdp/current/hbase-client/lib/hbase-common.jar,/usr/hdp/current/hbase-client/lib/hbase-client.jar,/usr/hdp/current/hbase-client/lib/hbase-protocol.jar,/usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar \
--class myclass myjar.jar ./${configuration} Many thanks to any help you can provide.
... View more
Labels:
02-20-2019
02:24 PM
Hi all, Witnessing some unexpected behavior i'm hoping someone might be able to clarify for me. I have a spark job which we set the zookeeper root to "/example" and the consumer group to "example_group", the offset here is 2:101 1:104 0:106 Using the same consumer group i switched the zookeeper root to "/example_temp" the idea here being that we can continue testing our job on "/example_temp" while the messages are still sent to the kafka topic and when we switch back to "/example" we'll ingest all the messages since the last offset for "/example", allowing us to test a larger load. Now the weird bit is that the offset in "/example_temp" is lower than that of "/example" 2:65 1:67 0:66 And so when I switched back to "/example", i got this error java.lang.IllegalArgumentException: requirement failed: numRecords must not be negative Because the offset saved on the broker is that of "/example_temp" and this is lower than the value saved in zookeeper for "/example" if you're still with me, my question is how did the offset for "/example_temp" end up lower than that of "/example"?
... View more
Labels:
07-30-2018
03:59 PM
Hi Nikhil, Can you perform any commands on that table? Perhaps try dropping the partition. It seems that the data was removed at some point on HDFS but the hive tables metadata still thinks those partitions exist ALTER TABLE *tableName* drop if exists PARTITION(date_part="2018022313");
... View more
07-30-2018
03:33 PM
Hi all, Related to this question which is a tad outdated , is it now possible to connect from Apache Storm to IBM MQ queues? Hortonworks version 2.6.0 Storm version:1.1.2 Thanks, Dan
... View more
Labels:
06-18-2018
03:07 PM
Hi all, I'm looking into the rerun functionality within Oozie coordinators, as per the documentation snippet below I know you can rerun all jobs within an oozie coordinator between a certain time frame or for certain actions. $oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh]
[-action 1, 3-4, 7-40] (-action or -date is required to rerun.)
[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
(if neither -action nor -date is given, the exception will be thrown.)
Either -action or -date should be given.If -action and -date both are given, an error will be thrown.Multiple ranges can be used in -action or -date. See the above examples.If one of action in the given list of -action does not exist or not in terminal state, the rerun throws an error.The dates specified in -date must be UTC.Single date specified in -date must be able to find an action with matched nominal time to be effective.If -nocleanup is given, coordinator directories will not be removed; otherwise the 'output-event' will be deleted.If -refresh is set, new dataset is re-evaluated for latest() and future().If -refresh is set, all dependencies will be re-checked; otherwise only missed dependencies will be checked. My question is can you specify that in that time frame you would only like to rerun jobs that have a status of KILLED or FAILED, otherwise it is ignored e.g. SUCCEEDED. Is this a functionality that exists or will I have to write something up in a shell script to retrieve these failed jobs ID's and then rerun them? Any help provided is much appreciated, Thanks, Dan
... View more
Labels:
04-11-2018
03:26 PM
Hi all, Bit of background, I have been struggling with an issue in which a table can get locked but the lock gets stuck in a "waiting" state and so does not allow the query to progress, after about 2 hours the job complains that the table is locked and it cannot access it. I'm trying to identify why the table is getting locked which brings me to my question.... If a user runs a query but then disconnects the session can this cause a table to get locked in this state? | lockid | database | table | partition | lock_state | blocked_by | lock_type | transaction_id | last_heartbeat | acquired_at | user | hostname | agent_info | 78011111.2 | db | tbl | NULL | WAITING | 78043210.2 | EXCLUSIVE | NULL | 1523459564452 | NULL | user | host | hive_20180411161225_2b33e811-e44c-59ds-afb3-b4111fcb019a |
... View more
Labels:
11-23-2017
02:54 PM
Hi all, Fairly straightforward question, in the Apache Falcon documentation it specifies "The libraries required for the workflow should be available in lib folder in workflow path." I have several workflows which have a dependency on a particularly large jar and so if possible I wanted to avoid duplicating it accross the file system, is there a property we can use in falcon to define this lib location? Thanks, Dan
... View more
Labels:
10-13-2017
11:13 AM
Thought as much, thanks for confirming
... View more
10-13-2017
10:29 AM
Hi all, Sorry for the basic question i've had little success searching online, i just need clarification whether something is possible. Can i run a beeline command against a file that is in HDFS. I know we use -f in beeline to specify a file when its on the local file system but can this also be done against a file on HDFS. My use case is that i'd like to run a beeline command through a shell action in Oozie. I'm hitting some issues using Hive2 actions so i wanted to try using a shell action instead Any help is much appreciated, Thanks
... View more
Labels:
09-15-2017
01:47 PM
Hi all, I was hoping someone could confirm whether something i'm trying to do is possible because i'm currently hitting multiple issues. I would like to add a record into a hive table using and insert statement, within this insert statement I have one column which should add a count value to the table based off of the result of a query. My Hive SQL is below.... use ${database};
set hivevar:deltaCount = select count(*) from ${database}.${hive_table};
DROP TABLE IF EXISTS ${database}.process_status_stg_${hive_table};
create table ${database}.process_status_stg_${hive_table} (
taskName varchar(50) COMMENT 'Name of the task being run to populate data',
starttime varchar(50) COMMENT 'time of record addition',
status varchar(50) COMMENT 'status of the task',
workflowID varchar(50) COMMENT 'workflow ID that is running the task',
oozieErrorCode varchar(50) COMMENT 'Error code returned by Oozie',
recordsLoadedCount varchar(50) COMMENT 'records pulled in previous load') ;
insert into table ${database}.process_status_stg_${hive_table} values ('${hive_table}','${current_time}','${taskStatus}','${workflowID}','${errorCode}', (CASE ${taskStatus} WHEN 'COMPLETED' THEN '${hiveconf:deltaCount}' ELSE 'N/A' end as recordsLoadedCount));
Any help is much appreciated, Thanks
... View more
Labels:
09-14-2017
01:40 PM
Like it says your permissions haven't been set up quite right. Could be down to some permissions not being applied after your update? Check ranger(or whatever control manager you're using) and confirm that the relevant user/group has the expected write permission on your HDFS location(s).
... View more
08-07-2017
04:28 PM
Does this happen everytime you run the oozie workflow or is this just a one time event? If so i'd just kill it.
... View more
08-07-2017
04:26 PM
This question relates to the usage of Apache Falcon any help provided is much appreciated. Hopefully this is a query with a very easy resolution. Our falcon process is trying to define an argument which will be passed to an oozie workflow and then subsequently into an SQL script. This value is a date value which should be passed in the format YYYYMMDDHH. Currently we're trying to do this by defining as the below as you would in a Falcon Feed. <property name ="partitiondate" value="${YEAR}${MONTH}${DAY}${HOUR}"/> But this gives an error, CausedBy: E1004: Expression language evaluation error, Unable to evaluate :${YEAR}${MONTH}${DAY}${HOUR} Is there another way to pass this value? Thanks again for any help.
... View more
Labels:
08-22-2016
04:13 PM
It is definitely possible. What is the error message you are getting?
... View more
07-28-2016
02:27 PM
Thanks for your quick response @Sunile Manjee. Would you be able to provide a quick example of how this looks?
... View more
07-28-2016
02:00 PM
Is there any way I can provide multiple JDBC connections within the Sqoop command? For example if JDBC1 is down then Sqoop should pick up JDBC2 and run the import/export. For example: sqoop export --connect "jdbc string 1 / jdbc string 2" It should then pick the correct JDBC URL based on which on is up and running Is this possible? Thanks for any help you can provide, Dan
... View more
Labels:
07-19-2016
04:07 PM
Hi all, I'm trying to schedule a job using falcon which will run everytime a new partition is created in Hive, this part appears to be working, however whenever it runs it complains that it is unable to find a variable despite this variable having been declared in the code below variable [inputFeedCCTM] cannot be resolved <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<process name="processLink" xmlns="uri:falcon:process:0.1">
<tags>owner=owner</tags>
<pipelines>***</pipelines>
<clusters>
<cluster name="cluster">
<validity start="2016-07-16T09:00Z" end="2033-01-13T20:00Z"/>
</cluster>
</clusters>
<parallel>1</parallel>
<order>FIFO</order>
<timeout>hours(1)</timeout>
<frequency>days(1)</frequency>
<timezone>UTC</timezone>
<inputs>
<input name="inputFeedLink" feed="feedInLink" start="yesterday(0,0)" end="yesterday(0,0)"/>
<input name="inputFeedLink2" feed="feedInLink2" start="yesterday(0,0)" end="yesterday(0,0)"/>
<input name="inputFeedLink3" feed="feedInLink3" start="yesterday(0,0)" end="yesterday(0,0)"/>
</inputs>
<!-- <outputs>
<output name="outputFeed" feed="feedOutLink" instance="yesterday(0,0)"/>
</outputs>-->
<properties>
<property name="table" value="linked"/>
<property name="appPath" value="/path"/>
<property name="workflowAppPath" value="/wfPath"/>
</properties>
<workflow name="falcon-wf" version="5.0" engine="oozie" path="/path/workflow"/>
<retry attempts="2" delay="hours(1)" policy="exp-backoff"/>
<ACL owner="Owner" group="hadoop" permission="0x775"/>
</process> In the Oozie workflow the variable is being referenced first inside a shell action like so <argument>${inputFeedLink}</argument> Any help is much appreciated Thanks, Dan
... View more
Labels:
07-11-2016
10:00 AM
1 Kudo
Hi Rene, I don't normally use hue so i'm not entirely sure how it creates the job properties. When I store the hive credentials in job.properties I like to do it like this: hivePrincipalJdbcUrl=<yourHiveUrl>;
hivePrincipal=<yourPrincipal> Then at the top of the Oozie workflow.xml declare these credentials <credentials>
<credential name="hive2Cred" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>${hivePrincipalJdbcUrl}</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>${hivePrincipal}</value>
</property>
</credential>
</credentials> Then finally reference these credentials in your Hive Action: <action name="exampleHive" cred="hive2Cred">
<hive2 xmlns="uri:oozie:hive2-action:0.1">
<job-tracker></job-tracker>
<name-node></name-node>
<jdbc-url></jdbc-url>
</hive2>
<ok to="end" />
<error to="fail" />
</action>
... View more
07-06-2016
11:18 AM
2 Kudos
As Ben said there is no functionality from items within the job.properties, without referencing the tags within the workflow they will never be used. You'd want to store these tags in the job.properties file if they change often(such as across different environments) but you still want to use the same workflow, saves you having to go in and edit the workflow each time. Plus it looks a bit nicer 🙂
... View more
06-21-2016
10:28 AM
The reasoning for having it in a table is so that the table can be observed by those who might not have access to the cluster and they can then query the table through Ambari views etc. With that in mind would you suggest the best solution is to use Oozie Java actions?
... View more
06-20-2016
02:35 PM
@Jitendra Yadav, I have updated the question with details of what I would be using Hbase for. Please let me know of any queries.
... View more
06-20-2016
02:09 PM
This helps me regarding authentication when using hbase. But i'm asking from a more generalized "how exactly is Hbase used within Oozie" perspective. So would I need to use a shell or java action, will I need any scripts as I would with Hive etc. I'm fairly new to Hbase so I'd be grateful for any help clarifying this.
... View more
06-20-2016
01:58 PM
Hi all, As the title suggests i'm trying to work out how to use Hbase within an Oozie workflow. Resources I've found online regarding this subject have been very limited. Do I need to use a Java action within my Oozie workflow in order to use Hbase? I require to create a table which will be used to monitor when my data pulls for a specific table are running/completed/failed, which we are referring to as the process status table. Currently to do this I am inserting a new row to a hive table before the relevant actions start and then once the data pull is completed a new row is inserted depending on whether or not the actions were all successful. I believe that using Hbase instead of Hive would be a much more efficient way to add to this table as only one row is added at a time. Any help provided would be much appreciated
... View more
Labels:
06-13-2016
02:15 PM
Thanks for the response Ben. The changes you have suggested have worked to a degree and I know due to the use of partitions that there should be no further degradation to the speed of the query, but this query can still take up to a minute to complete. I shall continue to look into other solutions to this issue and post them if I find any.
... View more
06-08-2016
10:34 AM
Hi all, I am currently pulling the max value of a timestamp column from my tables in Hive and using this to pull data after this date using Sqoop, i am using Oozie in order to perform these steps. This is currently done by running a query against the Hive table to put this value into hdfs and then this is picked up in another Oozie action before being passed to the Sqoop action. This all run perfectly fine, however retrieving the Max timestamp value and putting this into HDFS is currently very slow and I can only see this getting slower as more data is inserted into the table. The Hive SQL I am using to pull this value is as below: INSERT OVERWRITE DIRECTORY '${lastModifiedDateSaveLocation}' select max(${timestamp_column}) from ${hive_table}; Can anyone suggest a more optimized solution to retrieve this max timestamp? Thanks for your help, Dan
... View more
06-03-2016
04:08 PM
I am trying to obtain the date for which the falcon process is running in my oozie workflow. Any idea on how this can be passed to the workflow or obtained directly? Any help is much appreciated.
... View more
Labels:
04-15-2016
03:00 PM
I ended up going with your approach Ben as it suited what I was trying to do a bit better and after much fiddling around I managed to get it working. However, I am getting the value from my query back like this lastModified=+------------------------+--+ | 2016-03-31 21:59:57.0 | +------------------------+--+ Whereas all I really want is the date value not the extra jargon, is this something i can use regex for to get rid of? Thanks
... View more
04-12-2016
10:59 AM
2 Kudos
Hi all, I was hoping someone might be able to detail whether what I am attempting to do is currently possible in Oozie and if so how it could be done. I have seen many sources about getting an output from a shell action and inputting it into a Hive action I have however not seen much on whether this can be done the other way around. So my issue is that I would like to run a hive action which will capture the most recent field in a table based off of the max timestamp. I would then like to pass this timestamp value over to a shell action which will take this value and put it in the where statement for a Sqoop Extract. How would I go about passing this value from the Hive action to the Shell action? Is this possible? Please let me know if you need any additional information, thanks in advance.
... View more
Labels:
03-11-2016
04:16 PM
2 Kudos
I've resolved this issue, turned out the problem came from a driver that was being used --driver
oracle.jdbc.OracleDriver Removed the driver, problem solved!
... View more