When you know your application ID try accessing your yarn logs Usage: yarn logs -applicationId <application ID> [options] COMMAND_OPTIONS Description -applicationId <application ID> Specifies an application id -appOwner <AppOwner> AppOwner (assumed to be current user if not specified) -containerId <ContainerId> ContainerId (must be specified if node address is specified) -help Help -nodeAddress <NodeAddress> NodeAddress in the format nodename:port (must be specified if container id is specified) https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#logs ... View more

if you also want your sqoop command included you can expend the command like this (with set -x and set +x: { echo $(date) set -x beeline -u ${hive2jdbcZooKeeperUrl} -f "file.hql" set +x echo $(date) } 2>&1 | tee /tmp/sqoop.log ... View more

it's not only an issue with split, also with eg concat. escaping a semicolon in hue works fine but beeline errors with the same error as above. Replacing the semicolon with \073 works here as well. ... View more

If you want to use crontab that you already decided to use a time-trigger/interval right? You should really use a coordinator. If you really want to stick with crontab than the command is more or less correct. You have a typo (--oozie should be -oozie, and the port normally is 11000 but I guess you already confirmed the port?) and normally you refer to a job.properties file (stored locally not on hdfs) with -config. So it should look like: oozie job -oozie http://sandbox.hortonworks.com:11000/oozie -config /path/to/job.properties -run In the job.properties file you would have parameters listed like namenode, jobtracker, hcmetastoreuri and of course the one you provide via the -D: oozie.wf.application.path Normally the hdfs://namenode part can be omitted from the apppath url. ... View more

We are currently using a mysql metastore in our test environment so it is possible (we run hpd 2.5). Only thing is that oozie requires the sqoop-site.xml file to be placed somewhere in hdfs to access the metastore. We don't really like the idea that passwords are just stored like that.. ... View more

The answer is outdated. It is possible to use a character attribute as split-by attribute. You only need to add -Dorg.apache.sqoop.splitter.allow_text_splitter=true after your 'sqoop job' statement like this: sqoop job -Dorg.apache.sqoop.splitter.allow_text_splitter=true \\ --create ${JOB_NAME} \\ -- \\ import \\ --connect \"${JDBC}\" \\ --username ${SOURCE_USR} \\ --password-file ${PWD_FILE_PATH} \\ no guarantees though that sqoop splits your records evenly over your mappers though. ... View more

worked! many thanks, you saved my day. ... View more

@Gayathri Reddy G or @Sindhu can you tell me exactly how you added the -Dorg.apache.sqoop.splitter.allow_text_splitter=true to your import statement? I'm trying to create a sqoop job with this import statement but it keeps failing: 16/10/27 17:53:25 ERROR tool.BaseSqoopTool: Error parsing arguments for import: 16/10/27 17:53:25 ERROR tool.BaseSqoopTool: Unrecognized argument: -Dorg.apache.sqoop.splitter.allow_text_splitter=true my shell script to create the sqoop job looks like this: #!/bin/bash sqoop job \ --create <tablename> \ -- \ import "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" \ --connect "jdbc:<URL>" If I remove the -Dorg line all goes file but offcourse the execution of the job failes thats why I need to pass this allow_text_splitter as a parameter. I tried with double quotes, without, but no luck unfortunately ... View more

Hi njayakumar, we have sqoop working with the mysql metastore but oozie gives errors that it can't find the driver to connect to the sqoop metastore. Caused by: java.sql.SQLException: No suitable driver found for 'jdbc:mysql://<server>.com/sqoop' The mysql-connector-java.jar is available in the java folder mentioned by you, also available in: /usr/hdp/current/oozie folders libserver, libtools and lib (as symbolic link) and for sqoop in /usr/hdp/current/scoop/lib folder any thoughts what we are missing here? I already got the same error trying to use the sqoop metastore 'service'. Oozie wasn't able to find that driver either... ... View more