Member since
07-26-2016
24
Posts
7
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2177 | 03-07-2018 11:16 PM | |
16754 | 08-20-2017 12:08 AM | |
599 | 06-01-2017 08:26 PM | |
786 | 06-01-2017 05:59 PM | |
1288 | 05-20-2017 12:30 AM |
03-07-2018
11:16 PM
2 Kudos
According to https://issues.apache.org/jira/browse/SPARK-15348, spark now is not support transactional hive table.
... View more
08-20-2017
12:08 AM
The spark.driver.extraclasspath , spark.executor.extraclasspath is easy to understand. If you want all you spark job load a particular depency jars to drivers and executers then you can specify in those property. The --jars is if you want to add dependency jar to a spark job
... View more
07-28-2017
11:30 PM
You need to first make your local machine be able to grant a kerberos ticket from same KDC server. Then configure your browser to work with SPNEGO.
... View more
07-21-2017
08:35 PM
Could you try add below four properties to your workflow corresponding java action ?
<name>mapreduce.job.user.classpath.first</name>
<value>true</value>
</property>
<property>
<name>mapreduce.task.classpath.user.precedence</name>
<value>true</value>
</property>
<property>
<name>oozie.launcher.mapreduce.task.classpath.user.precedence</name>
<value>true</value>
</property>
<property>
<name>oozie.launcher.mapreduce.job.user.classpath.first</name>
<value>true</value>
</property>
... View more
07-17-2017
05:27 PM
One single workflow can contain several actions. But you need to make sure all dependencies are prepared for those actions
... View more
06-20-2017
05:51 AM
Yes, you can use Oozie ssh action to login to the certain host then download data from the HDFS
... View more
06-20-2017
05:49 AM
If the start time is in the past, then the Oozie will compensate the missing workflows. For example, if the coordinator is schedule once per day and start time is two days before, then you should find two more workflows no matter it succeed or failed
... View more
06-19-2017
06:11 PM
Can you try include xmlns="uri:oozie:coordinator:0.4" after the timezone="United_kingdom/London"?
... View more
06-09-2017
07:54 PM
I suggest you develop and test all the job locally first such as Sqoop and Hive script. For the Spark application, please make sure you compile you jar with Hortonworks dependencies to prevent any dependency issue. For troubleshooting Oozie workflow, first thing is to check Oozie log and search by workflow id. Then check the Yarn application log for the Oozie launcher and the child job.
... View more
06-01-2017
08:26 PM
1 Kudo
@dsun yes, you can. First, you need to ensure those service accounts are created in the AD and the cluster hosts all connect to AD and those users are valid. Second, set ignore_groupsusers_create=true in cluster-env.xml then start install HDP
... View more
06-01-2017
05:59 PM
Actually you are not able to force the oozie action running in a certain node unless you are using ssh action. Once the Oozie submit the action to Yarn, Yarn will running the job in one of the NodeManger host.
... View more
06-01-2017
05:56 PM
What error you got from Ambari once you start Oozie?
... View more
05-20-2017
12:30 AM
You can upload your keytab file to workflow lib folder so that the keytab will copy to the container folder no matter the job is running on which nodemanager. Then you can specify the --keytab your-keytab --principal your-principal in your spark-submit command. But you have to upload the update keytab to workflow lib folder every time you change the password.
... View more
05-15-2017
06:14 PM
@Meryem Moumen Can you post the yarn application log for Oozie launcher and the the application launched for the spark job?
... View more
05-11-2017
06:31 PM
You can use fork and join to run the shell script parallel with different hive tables. You need to have ten shell action to run same shell script but pass different parameter. You can reference this link for the fork and join https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a3.1.5_Fork_and_Join_Control_Nodes
... View more
05-10-2017
09:55 PM
Inside your shell script, you may want to check if the one of your hive table fails then script exit with non-zero code. Then the action will failed and goes to email-error.
... View more
05-10-2017
09:31 PM
can you post the oozie launcher application log which will show why the job failed?
... View more
05-10-2017
09:01 PM
I think you can consider using password file. Here is an example command that specifies the password file in the local file system: sqoop import --connect jdbc:mysql://database.example.com/employees \ --username venkatesh --passwordFile ${user.home}/.password
You can store the password file either in local or in hdfs. The file permission should be 400. For you case, I suggest you store it in hdfs then no matter which nodemanger host runs you job, it will be able to access the password file.
... View more
03-31-2017
09:25 PM
2 Kudos
To clear local file cache and user cache for yarn, perform the following:
Find out the cache location by checking the value of the yarn.nodemanager.local-dirs property : <property>
<name>yarn.nodemanager.local-dirs</name>
<value>/hadoop/yarn/local</value>
</property>
Remove filecache and usercache folder located inside the folders that is specified in yarn.nodemanager.local-dirs. [yarn@node2 ~]$ cd /hadoop/yarn/local/
[yarn@node2 local]$ ls filecache nmPrivate spark_shuffle usercache
[yarn@node2 local]$ rm -rf filecache/ usercache/
If there are more than one folder, clean them one by one. Restart YARN service.
... View more
- Find more articles tagged with:
- How-ToTutorial
- Oozie
- Sandbox & Learning
- YARN
Labels:
03-31-2017
05:57 PM
Perform following steps to enable the verbose log for Oozie launcher
Step 1.Add below property to the action configuration section in the workflow file: <configuration>
.....
<property>
<name>oozie.launcher.mapreduce.map.java.opts</name>
<value>-verbose</value>
</property>
</configuration> Step 2.Upload the updated workflow file to workflow folder defined by oozie.wf.application.path in job property file Step 3.Submit the workflow and now you should be able to see verbose log for the Oozie launcher such as class loading information . [Loaded java.lang.ExceptionInInitializerError from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar]
[Loaded org.apache.commons.logging.impl.LogFactoryImpl$2 from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar]
[Loaded org.apache.commons.logging.impl.LogFactoryImpl$1 from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar]
[Loaded org.apache.commons.logging.Log from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar]
[Loaded org.apache.commons.logging.impl.Log4JLogger from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar]
[Loaded org.apache.log4j.spi.AppenderAttachable from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar]
[Loaded org.apache.log4j.Category from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar]
[Loaded org.apache.log4j.Logger from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar]
[Loaded org.apache.log4j.Priority from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar]
[Loaded org.apache.log4j.Level from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar]
[Loaded java.lang.InstantiationError from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar]
[Loaded sun.reflect.UnsafeFieldAccessorFactory from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar]
[Loaded sun.reflect.UnsafeQualifiedStaticFieldAccessorImpl from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar]
[Loaded sun.reflect.UnsafeQualifiedStaticObjectFieldAccessorImpl from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar]
[Loaded java.util.HashMap$EntrySet from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar]
[Loaded java.util.HashMap$HashIterator from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar]
[Loaded java.util.HashMap$EntryIterator from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar]
[Loaded java.util.MissingResourceException from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar]
[Loaded org.apache.log4j.LogManager from file:/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar]
[Loaded java.net.MalformedURLException from /usr/jdk64/jdk1.8.0_60/jre/lib/rt.jar]
... View more
- Find more articles tagged with:
- FAQ
- Governance & Lifecycle
- Oozie
Labels:
03-24-2017
11:09 PM
PROBLEM DESCRIPTION The Oozie service check fails and the following error message is displayed in Ambari: stderr: /var/lib/ambari-agent/data/errors-12523.txt Python script has been killed due to timeout after waiting 300 secs
There is no error in the stdout.
The service check is terminated because the timeout (300 secs by default) is reached. CAUSE This issue occurs when the time taken by Ambari to upload jar and workflow files to hdfs for Oozie service check takes longer than the timeout mentioned in the server settings. Example: Nodes in a cluster are configured on ipv4 proxy, which causes network slowness among nodes. Ambari uploads jar files and workflow files to the hdfs for Oozie service check, depending on network performance time required to upload these files exceeds 300 secs timeout. WORKAROUND To increase the timeout, find and update the timeout set in the metainfo.xml file located in Edit /var/lib/ambari-server/resources/common-services/OOZIE/your_version_number/metainfo.xml, RESOLUTION Improve the network performance, so that the Oozie service check can be finished in 300 secs timeout period.
... View more
- Find more articles tagged with:
- Ambari
- Governance & Lifecycle
- Issue Resolution
- Oozie
Labels:
03-24-2017
10:43 PM
To configure Oozie workflow after enabling the ResourceManager HA, do the following: Find the YARN Resource Manager cluster id. To find the cluster id, Go to Ambari > YARN > configs Search for property yarn.resourcemanager.cluster-id After finding the cluster id, at the Job Property file, provide the YARN Resource Manager cluster id to the job _Tracker like the following: jobTracker= yarn-ha Submit the Oozie job using the modified job property file.
... View more
- Find more articles tagged with:
- FAQ
- Governance & Lifecycle
- Oozie
Labels: