Member since
09-24-2015
76
Posts
32
Kudos Received
10
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
286 | 02-24-2017 09:17 AM | |
1708 | 02-13-2017 04:49 PM | |
636 | 09-26-2016 06:44 AM | |
294 | 09-16-2016 06:24 AM | |
394 | 08-25-2016 02:27 PM |
09-13-2019
01:37 PM
Short Description: This article will help to renew kerberos ticket with Falcon. Article It has been seen that Falcon server fails to perform operations after kerberos credentials expire and following exception occur. Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) To solve this issue, we must set following parameter in Falcon startup.properties through ambari-ui to revalidate kerberos credentials. Value for this properties is in seconds. Stop Falcon server Set parameter : *.falcon.service.authentication.token.validity=<value in seconds> Start Falcon server Note: This article is for version greater than HDP-2.5.* .
... View more
- Find more articles tagged with:
- Falcon
- FAQ
- Kerberos
- Sandbox & Learning
Labels:
03-10-2017
11:34 PM
@n c Logs file you shared from Oozie and Falcon is not helping much to understand actual issue you are facing while running replication. From Falcon logs I found that one WF replication job (0000000-170309153350882-oozie-oozi-W) executed successfully. So if you are seeing that no replication happened can you please look to Oozie UI for this WF job and try to get corresponding Hadoop job to see whether it ran successfully or not.
... View more
03-10-2017
10:36 PM
@mayki wogno Just try to see if by deleting oozie auth token, if it helps you. rm ~/.oozie-auth-token
... View more
03-10-2017
06:54 PM
@n c I have just checked the falconlog.txt and found that replication job executed successfully. WF id : workflowId=0000000-170309153350882-oozie-oozi-W status=SUCCEEDED, Even for this WF job counters get populate as well showing how many bytes and files copied to backup cluster.
... View more
03-09-2017
07:11 PM
@n c Can you please share the falcon application log (/var/log/falcon/falcon.application.log) and oozie logs from /var/log/oozie to diagnose the issue, when you submit the falcon mirroring job.
... View more
02-24-2017
09:17 AM
@Sankar T To resolve this issue, following is the workaround : 1. Remove service org.apache.falcon.metadata.MetadataMappingService from *.application.services in falcon startup.properties available in /etc/falcon/conf/ 2. Restart falcon server Hope this helps
... View more
02-13-2017
04:49 PM
1 Kudo
@Abb Code Just try to see if by defining the required variable through global section in workflow.xml helps you. More details about defining global is available at following URL. https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a19_Global_Configurations
... View more
02-07-2017
08:51 AM
1 Kudo
@rahul gulati Earlier I observed that this similar exception occurred at the time of launching of Oozie workflow. Can you try to set following memory related parameter in Oozie workflow.xml with some higher value like 1024mb so that workflow launches successfully. For e.g: <property>
<name>oozie.launcher.mapred.map.child.java.opts</name>
<value>-Xmx1024m</value>
</property> See if this helps you.
... View more
01-24-2017
05:08 AM
Yes in this case you have to create the _success file manually under the timestamped directory. I will also deploy the Falcon once and check about this.
... View more
01-20-2017
05:12 PM
@Uvaraj Seerangan I think the exception has occurred because in snapshottable directory there no snapshots available. So can you please manually create the snapshots first for snapshottable directory and then submit the extensions. Please see if this works for you.
... View more
01-20-2017
04:58 PM
@Uvaraj Seerangan In this case you have to ensure that _success must be available in input directory. When you ingest data to timestamped input directory, please make sure to add the flag file _success, once ingestion complete.
... View more
01-18-2017
06:11 PM
1 Kudo
@Shikhar Agarwal In Falcon there is properties section in Entities using which you can pass the custom properties. See the following URL : https://falcon.apache.org/EntitySpecification.html#Feed_Specification and "Custom Properties" section
... View more
01-18-2017
04:33 PM
@Uvaraj Seerangan When specifying the Falcon feed entity, have you specified the value for availabilityFlag with _success. If yes, please remove that and try.
... View more
01-13-2017
02:48 PM
@Shihab It looks like that Falcon server did not come up properly due to which Falcon webUI and client is having issues. Can you please share the falcon application log from Falcon logs directory to analyze the Falcon server issue.
... View more
12-12-2016
06:25 PM
@Avijeet Dash If you are noticing that jobs are getting submitted to oozie as "admin" user, then it might be some issue with oozie configuration. Can you please check that. What is the HDP version used.
... View more
11-15-2016
02:08 PM
@Kuldeep Kulkarni Is the launched Hadoop job from action also running, have you checked that this as well. Can you attach the oozie.log to analyze this behaviour.
... View more
10-14-2016
02:46 PM
@narender pasunooti Is "oozie.wf.application.path=<location of workflow path>" specified in job.properties file when submitting the Spark based workflow.
... View more
09-26-2016
11:21 AM
@Ramakrishna Pratapa I have used following job.properties and workflow.xml so that sqoop import works for me in secure environment. In workflow.xml, I have specified the hive-site.xml in <file> that has been copied from /etc/hive/conf/hive-site.xml. job.properties nameNode=hdfs://machine-2-1.openstacklocal:8020
jobTracker=machine-2-1.openstacklocal:8050
queueName=default
oozie.use.system.libpath=true
oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.wf.application.path=${nameNode}/user/ambari-qa/sqoop-import/
oozie.action.sharelib.for.sqoop=sqoop,oozie,hive,hcatalog
mapreduce.job.user.name=ambari-qa
user.name=ambari-qa workflow.xml
... View more
09-26-2016
06:44 AM
2 Kudos
@alina n If there is delay in running of job through Oozie, just please check that ResourceManager must not be in overwhelming state and there must be sufficient capacity in cluster to execute jobs.
... View more
09-23-2016
06:38 PM
Thanks @Ramakrishna Pratapa for providing detailed logs. I think this exception is happening due to mismatch in client server config. So please ensure that hive.metastore.sasl.enabled=true must be set in hive-site.xml for client and server.
... View more
09-23-2016
12:45 PM
@Ramakrishna Pratapa When using following workflow.xml for Sqoop with HCatalog, it works for me : <?xml version="1.0" encoding="UTF-8"?>
<workflow-app xmlns="uri:oozie:workflow:0.4" name="hive-wf">
<credentials>
<credential name='hive_auth' type='hcat'>
<property>
<name>hcat.metastore.uri</name>
<value>thrift://ambari.example.com:9083</value>
</property>
<property>
<name>hcat.metastore.principal</name>
<value>hive/_HOST@EXAMPLE.COM</value>
</property>
</credential>
</credentials>
<start to="import-sqoop"/>
<action name='import-sqoop' cred="hive_auth">
<sqoop xmlns='uri:oozie:sqoop-action:0.4'>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
</configuration>
<arg>import</arg>
<arg>--verbose</arg>
<arg>--connect</arg>
...
...
<file>/user/ambari-qa/oozie_sqoop/hive-site.xml</file>
</sqoop>
<ok to="end"/>
<error to="fail"/>
</action>
Can you please use this workflow.xml. Next time if you see any issue, please provide the HDP version as well.
... View more
09-21-2016
05:06 AM
1 Kudo
@Ramakrishna Pratapa From the exception, it looks like that while trying to run the workflow.xml on secure cluster HCat credential profile has not been set properly. Can you please have a look towards the credential profile and ensure that key-value are correct.
... View more
09-21-2016
04:49 AM
@Gaurab D No it is not necessary to use data condition as a trigger to start Oozie coordinator. Oozie provides time event as well to trigger coordinator at specified time.
... View more
09-19-2016
11:16 AM
1 Kudo
@Gaurab D You are seeing the message ":CoordActionInputCheck:: Missing deps" because input data to start the coordinator might not be available.
... View more
09-16-2016
10:13 AM
1 Kudo
@Gaurab D Are you seeing any other error message in the logs as well along with "Main class" error message. Can you try to submit coordinator again by setting following properties in coordinator.properties file . oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.use.system.libpath=true
... View more
09-16-2016
06:24 AM
2 Kudos
@alina n It looks like that you need to add value “localhost” or "127.0.0.1" for parameter hadoop.proxyuser.oozie.hosts in /etc/hadoop/conf/core-site.xml. Once you add this through ambari-ui, please save and restart the Oozie service.
<property>
<name>hadoop.proxyuser.oozie.hosts</name>
<value>127.0.0.1,localhost,sandbox.hortonworks.com</value>
</property> Hope this helps you.
... View more
09-16-2016
04:43 AM
@Aditi Kumari Have you created the output feed entity as well, can you share that. You have mentioned that scheduled job fails to update the table with new values, has any error occurred. For this you can check the launched workflow on Oozie UI and corresponding Hadoop jobs on RM UI.
... View more
09-09-2016
02:22 PM
@narender pasunooti The above exception occurred because same jar file with different versions might have set in classpath when Spark application launched from Oozie. These issues has been handled in HDP 2.5 and Spark application works successfully with Oozie. But to make the Spark application work with Oozie for HDP 2.4, please perform following prerequiste steps first. https://community.hortonworks.com/articles/48920/how-to-run-spark-action-in-oozie-of-hdp-230.html Hope this helps you.
... View more
09-08-2016
05:18 AM
7 Kudos
In this article, we will see how to
perform the mirroring of Hive data and metadata using Falcon from source
cluster to destination cluster. This article is based on HDP 2.5.
Configure Hive Configure
source and target Hive by clicking “Hive” from the Ambari Services
menu, then click “configs” to add following custom properties on Ambari
UI by scroll down to “Custom hive-site”, click it and then click “Add
Property”. Add following property name with value: hive.metastore.event.listeners = org.apache.hive.hcatalog.listener.DbNotificationListener
hive.metastore.dml.events = true Press OK to save the changes, then click Restart all the impacted services.
Bootstrap Table and DB Before creating Hive DR mirroring job to replicate Hive data/metadata for a DB or Table, it is required to perform initial bootstrap of Table and DB from source to target cluster.
Table Bootstrap For bootstrapping table replication, do an EXPORT of the table in question at source cluster, distcp the export directory to the target cluster, and do an IMPORT at target cluster. Export-Import can be seen here : https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport For example create the table global_sales and insert records: hive > create table global_sales
(customer_id string, item_id string, quantity float, price float, time timestamp)
partitioned by (country string);
hive > insert into table global_sales partition (country = 'us') values ('c1', 'i1', '1', '1', '2001-01-01 01:01:01'); Start Bootstrap ## On source cluster :
hive > export table global_sales to '/user/ambari-qa/export_sql';
$ hadoop distcp hdfs://machine-1-1.openstacklocal:8020/user/ambari-qa/export_sql
hdfs://machine-2-1.openstacklocal:8020/user/ambari-qa/import_sql
## On target cluster :
hive > import table global_sales from '/user/ambari-qa/import_sql'; Above steps will set up the target table in sync with source table so that the events on the source cluster that modify the table will then be replicated over. Database Bootstrap For bootstrapping DB replication, first target DB must be created. This step is expected because DB replication definitions can be set up only on pre-existing DB by users. Second, we need to export all tables in the source D B and import them in the target DB, as described in Table bootstrap.
Set up source and target cluster staging/working directory Source cluster: [root@machine-1-1 ~]# su - falcon hadoop fs -mkdir -p /apps/falcon/primaryCluster/staging hadoop fs -mkdir -p /apps/falcon/primaryCluster/working hadoop fs -chmod 777 /apps/falcon/primaryCluster/staging Target cluster: [root@machine-2-1 ~]# su - falcon hadoop fs -mkdir -p /apps/falcon/backupCluster/staging hadoop fs -mkdir -p /apps/falcon/backupCluster/working hadoop fs -chmod 777 /apps/falcon/backupCluster/staging
Create cluster entity Navigate to Falcon UI from Ambari services menu and create source cluster entity using Falcon UI by clicking “Create” -> “Cluster” Save the source cluster entity by clicking “Next”->”Save” . Create target cluster entity using Falcon UI by clicking “Create” -> “Cluster” Save the cluster entity by clicking “Next”->”Save” .
Insert records in source Hive server for replication. Insert some records in source Hive server to replicate to target Hive server. hive > insert into table global_sales partition (country = 'uk') values ('c2', 'i2', '2', '2', '2001-01-01 01:01:02');
Prepare and submit Hive DR Mirroring To submit the Hive DR mirroring job, click “Create”->”Mirror”->”Hive” and then fill the required values.
Click Next -> Save the Hive DR mirror job.
Submit and Schedule HiveDR
Check output Once scheduled Hive DR process completed (checked from Oozie UI), verify the target Hive server for output. Earlier, we inserted two records at source Hive server and now at target Hive server both records are available. ---
... View more
- Find more articles tagged with:
- disaster-recovery
- Falcon
- Governance & Lifecycle
- Hive
- How-ToTutorial
- Oozie
Labels:
09-07-2016
12:09 PM
@papu ray On further looking the exception, it looks like that this happen when Oozie could not able to get the launched job status from job history server (JHS) . So ensure that JHS must be up and running. Also ensure that job history on HDFS (/mr-history/done) must be owned by mapred:hadoop. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/start_mapreduce_jobhistory_server.html
... View more