This is a problem from the hive move task (which has since been fixed in HIVE-15355) which is called by Sqoop after the import into HDFS. So, disabling move task parallelism is the right solution by adding the configuration parameter hive.mv.files.thread=0. That said, I would suggest using --hcatalog-table option with import which allows for 1. better data fidelity 2. remove one intermediate step of landing on HDFS and then invoking the hive client to do the import ... View more

You should set something like queueName = sqoop in your job properties and refer to it in the workflow action config for the two parameters. Sorry if it was not clear ... View more

when submitting jobs via oozie, there is a laucher job and the launched job (in this case mr job launched by Sqoop). You are probably seeing the launcher job getting submitted to the default queue. To get the launcher job also go to sqoop queue, you need to add the following config property to the workflow.xml or the job propert oozie.launcher.mapreduce.job.queuename = sqoop In general, if you want to pass any config to the launcher job, you need to prefix the config name with oozie.launcher. -- For example, if you are running a hive action and the need to configure a larger map memory for the hive client ... View more

You need to register download and register the bdb jar with ambari server before starting Falcon server. This is required from HDP 2.5 or later now The following steps would help wget -O je-5.0.73.jar http://search.maven.org/remotecontent?filepath=com/sleepycat/je/5.0.73/je-5.0.73.jar cp je-5.0.73.jar /usr/share/ chmod 644 /usr/share/je-5.0.73.jar ambari-server setup --jdbc-db=bdb --jdbc-driver=/usr/share/je-5.0.73.jar ambari-server restart Restart Falcon service ... View more

@Ed Berezitsky >> Small correction: if you use hcatalog, but your table is still textfile format with "|" field delimiter, you'll still have the same issue The output file field delimiters are only needed for HDFS imports. In the case of Hcatalog imports, you tell the text file format properties as part of the storage stanza and the defaults for hive will be used. Essentially, the default storage format should be ok to handle this. BTW, hcatalog import works with most storage formats, not just ORC @Krishna Srinivas You should be able to use a Hive table using Spark SQL also - but may be you have other requirements also. Glad to see that @Ed Berezitsky's solution worked for you ... View more

If HDFS is just an intermediate destination before loading into hive, you can skip the step and directly load into Hive using the hcatalog-table option in sqoop which provides better fidelity of data and removes one step (and supports all Hive data types also) Please see https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_sqoop_hcatalog_integration ... View more

If you are using multiple clusters, you need to make sure that the hadoop configuration that Oozie uses for the target cluster (see oozie.service.HadoopAccessorService.hadoop.configurations property in oozie-site.xml) is correctly configured. By default in a single cluster environment, Oozie will point to the local core-site.xml for this by default ... View more

Why would you restart the RM for this? ... View more

Falcon uses hadoop distributed filesystem abstraction to do the replication -be it s3 or wasb. It essentially uses distcp, so whatever requirements are there for distcp in terms of accessing a filesystem applies to Falcon replication as well ... View more

If your cluster endpoint is ponting to HDFS, then the feed locations will be based on that that unless they are absolute path. Can you provide an example of what you are trying to do and the exceptions that you are getting. Thanks ... View more