What does the code below do? val conf = new SparkConf().setMaster("local").setAppName("testApp") val sc= SparkContext.getOrCreate(conf)   Reference: https://www.educba.com/spark-rdd-operations/ ... View more

Hi @aakulov    thanks for the update.   Can you please advise how can I schedule this scoop job so that it will update the hive table with incremental changes(CDC)? for example, suppose the XML fields are updated on Oracle, how can I schedule scoop job to replicate the incremental changes on Hive and Kudu?   Regards,   Roshan ... View more

Thanks for the update.   scala> val myRDD=spark.read.textFile("/devsh_loudacre/frostroad.txt") myRDD: org.apache.spark.sql.Dataset[String] = [value: string] why does myRDD.parallelize not working for above?   scala> val myRDD1=sc.parallelize(myRDD) <console>:26: error: type mismatch; found : org.apache.spark.sql.Dataset[String] required: Seq[?] Error occurred in an application involving default arguments. val myRDD1=sc.parallelize(myRDD)   Does the above mean a dataset has been created? what is the difference between the above and below? val myRDD2=sc.textFile("/devsh_loudacre/frostroad.txt")   can I add the .parallelize function with the above command?   Thanks,   Roshan   ... View more

I managed to fix it but getting error below because of XML types   [root@sandbox-hdp lib]# sqoop job --exec myjob7 Warning: /usr/hdp/3.0.1.0-187/accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.                                                                                         SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 21/06/25 15:32:55 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.3.0.1.0-187 21/06/25 15:32:55 INFO manager.SqlManager: Using default fetchSize of 1000 Enter password: 21/06/25 15:33:09 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled. 21/06/25 15:33:09 INFO manager.SqlManager: Using default fetchSize of 1000 21/06/25 15:33:09 INFO tool.CodeGenTool: Beginning code generation 21/06/25 15:33:10 INFO manager.OracleManager: Time zone has been set to GMT 21/06/25 15:33:10 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM cb_account_master t WHERE 1=0 Exception in thread "main" java.lang.NoClassDefFoundError: oracle/xdb/XMLType at oracle.jdbc.oracore.OracleTypeADT.applyTDSpatches(OracleTypeADT.java:1081) at oracle.jdbc.oracore.OracleTypeADT.parseTDSrec(OracleTypeADT.java:1002) at oracle.jdbc.oracore.OracleTypeADT.parseTDS(OracleTypeADT.java:936) at oracle.jdbc.oracore.OracleTypeADT.init(OracleTypeADT.java:489) at oracle.jdbc.oracore.OracleTypeADT.init(OracleTypeADT.java:470) at oracle.sql.TypeDescriptor.getTypeDescriptor(TypeDescriptor.java:981) at oracle.jdbc.driver.NamedTypeAccessor.otypeFromName(NamedTypeAccessor.java:78) at oracle.jdbc.driver.TypeAccessor.initMetadata(TypeAccessor.java:71) at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:833) at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:897) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1034) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3820) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3867) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1502) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:777) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786) at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289) at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260) at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246) at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:328) at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1879) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1672) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:516) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:656) at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:248) at org.apache.sqoop.tool.JobTool.run(JobTool.java:303) at org.apache.sqoop.Sqoop.run(Sqoop.java:150) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:186) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:240) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:249) at org.apache.sqoop.Sqoop.main(Sqoop.java:258) Caused by: java.lang.ClassNotFoundException: oracle.xdb.XMLType at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 33 more [root@sandbox-hdp lib]#   Kindly advise.   Thanks,   Roshan ... View more

Which methods do you think would be most appropriate to use? I was thinking of using beehive to read the table/run query in real time and save it in parquet. The load from parqet to Kudu? or using Sqoop to read Oracle tables into HDFS. Then from HDFS to Kudu. ... View more