I want to export data from a Hive (Hcatalog) Table to an Oracle database. For that purpose I am using sqoop with the Data Connector for Oracle and Hadoop (--direct Parameter). Since an export is not atomic in Sqoop I dont want to export the data into the target table directly. This is what the --staging-table Parameter in sqoop is for. The Problem is, that you cannot use the --staging-table Parameter when using the Data connector for Oracle and Hadoop. Is there anothery way to do this in --direct mode? ... View more

I want to export data from a Hive (Hcatalog) Table to an Oracle database. For that purpose I am using sqoop with the Data Connector for Oracle and Hadoop (--direct Parameter). Since an export is not atomic in Sqoop I dont want to export the data into the target table directly. This is what the --staging-table Parameter in sqoop is for. The Problem is, that you cannot use the --staging-table Parameter when using the Data connector for Oracle and Hadoop. Is there anothery way to do this in --direct mode?   My sqoop command is: sqoop export --connect jdbc:oracle:thin:@oracle_server:dmine --username USER --password xxx --table TEST_TIMESTAMP --direct --hcatalog-database default --hcatalog-table test_timestamp [--staging-table STA_TEST_TIMESTAMP] The error message with --staging-table Parameter is: The active connection manager (org.apache.sqoop.manager.oracle.OraOopConnManager) does not support staging of data for export. Please retry without specifying the --staging-table option. Is there a workaround for the Data Connector for Hadoop and Oracle? ... View more

I am building a java application that is suppoed to import data via sqoop and do some hdfs operations. This application should run on Cloudera CHD5.12.0 and sqoop 1.4.6   I successfully ran the sqoop import using a java ProcessBuilder. But this sees like a dirty way to do this.   My next approach was to use the sqoop ImortTool. But that way i ran into depreciated issues... I used org.apache.sqoop.tool.ImportTool which accepts a SqoopOptions object in the constructor. But it needs the SqoopOptions from com.cloudera... Package, which is depreciated.   So I tried a differen Solution: import org.apache.sqoop.Sqoop; import org.apache.hadoop.conf.Configuration; Configuration conf = new Configuration(); Sqoop.runTool(sqoopOptions.toArray(new String[sqoopOptions.size()]), conf); This results in folloing Error: 17/10/18 11:01:15 WARN tool.SqoopTool: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 17/10/18 11:01:15 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.12.0 17/10/18 11:01:15 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 17/10/18 11:01:15 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 17/10/18 11:01:15 INFO manager.SqlManager: Using default fetchSize of 1000 17/10/18 11:01:15 ERROR oracle.OracleConnectionFactory: Unable to load the jdbc driver class : oracle.jdbc.OracleDriver 17/10/18 11:01:15 ERROR tool.BaseSqoopTool: Got error creating database manager: java.lang.RuntimeException: Unable to load the jdbc driver class : oracle.jdbc.OracleDriver at org.apache.sqoop.manager.oracle.OracleConnectionFactory.loadJdbcDriver(OracleConnectionFactory.java:75) at org.apache.sqoop.manager.oracle.OracleConnectionFactory.createOracleJdbcConnection(OracleConnectionFactory.java:52) at org.apache.sqoop.manager.oracle.OraOopConnManager.makeConnection(OraOopConnManager.java:97) at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52) at org.apache.sqoop.manager.oracle.OraOopManagerFactory.accept(OraOopManagerFactory.java:114) at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:184) at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:270) at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:95) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:609) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at jobs.util.FullTableImportStrategy.runSqoopImportToHdfsTempDir(FullTableImportStrategy.java:77) at jobs.util.FullTableImportStrategy.execute(FullTableImportStrategy.java:29) at jobs.ImportJob.<init>(ImportJob.java:12) at start.Main.main(Main.java:12) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Oracle JDBC Driver is set up correctly, since everything works from commandline and with the ProcessBuilder.   How can i resolve this error, and is there maybe a better way to run a sqoop command from a java program? Thank you for your help! ... View more

Hi, I want to create a falcon process where the start instance of the input feed is always the start date of the feed. Is there a way to enter a static date instead of functions like now(0,0) or can i reference the start date of the feed? Thank you for your help ... View more

I have a falcon feed on an external hive table. This feed is input to a Falcon Process. When I schedule both, the input feed and the process, the process stays in WAITING status. I understand that hive thinks that the data is not yet there, because no partiton is added for the data. If the process instance would start i could run a MSCK REPAIR TABLE input_table; If i do this manually, the feed is processed as far as data was copied to the right directory. But how can i add the partitions automatically? ... View more

I am on HDP 2.6.1.0 and Ambari shows that Hiveserver2 is up and running. I created tables and ran querys via Hive View and everything was fine. Now I wanted to optimize my table and saw I was using Hiveserver 1.2 all the time (see screenshot). How can I make the Hive View use Hiveserver2? ... View more

I have finally managed to get a falcon export feed to Oracle running. But when i check the job configuration in oozie, i can see, that in every job falcon creates, the --hcatalog-partition-values are the same values. This means the same data is exported every time. What i want is, that for every day, a new partition gets exported. Can you help me please? Here is some code: <feed xmlns='uri:falcon:feed:0.1' name='OracleFeed3'> <availabilityFlag>_success</availabilityFlag> <frequency>days(1)</frequency> <timezone>GMT+02:00</timezone> <late-arrival cut-off='hours(4)'/> <clusters> <cluster name='JoinThrift' type='source'> <validity start='2017-05-17T10:30Z' end='2099-12-31T09:59Z'/> <retention limit='months(999)' action='delete'/> <export> <target name='OracleDatabase' tableName='HADOOP_TEST'> <load type='allowinsert'/> </target> </export> <table uri='catalog:default:adobe_archive#ds_year=${YEAR};ds_month=${MONTH};ds_day=${DAY}'/> </cluster> </clusters> <table uri='catalog:default:adobe_archive#ds_year=${YEAR};ds_month=${MONTH};ds_day=${DAY}'/> <ACL owner='hdfs' group='hadoop' permission='0x755'/> <schema location='/hcat' provider='/hcat'/> <properties> <property name='queueName' value='default'> </property> <property name='jobPriority' value='NORMAL'> </property> </properties> </feed> ... View more

Every day new data is loaded to HDFS. For that data i create an external table in hive. I use this external table as my falcon input feed. It is partitioned by year, month, day. Now i want to clean the data and put it into another hive table that is partitioned by year, month. I want to put it in the second table as soon as it arrives, not only once a month. Is this possible? ... View more

Hi, in our project we have input feed with frequency of one file per day. For archival purposes we want to store all files of one month in the same folder. Is this possible with falcon? I have read that the path pattern must have at least the granularity of the frequency (which is 1 day, since a new file is added each day) but output path pattern should be .../${YEAR}/${MONTH} ... View more

Hi, i am just beginning to learn falcon, but i just cant get my simple example to work. In a cron job i produce my input files with this pattern: files/<year>-<month>-<day>-<hour>-<minute>/file.csv The files are located on HDFS. What i want to do is to create a simple pipeline with all the csv files as input feed as input for a pig script that produces an output feed located at another location on HDFS. in the webUI the feed entity i created doesnt even show some instances. could somebody help me with this? thank you 🙂 ... View more