Member since
05-11-2016
29
Posts
1
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
7293 | 09-25-2017 02:06 PM | |
12392 | 08-16-2016 01:41 PM | |
3362 | 05-18-2016 06:30 AM |
09-25-2017
02:06 PM
Sorry, could not focus on this, was busy with production activities. Finally, I could able to run it successfully with below configurations, Jar / py name : ${nameNode}/user/solution.jar Main Class : Module.final_solution Options List : --conf spark.yarn.jar=local:/opt/cloudera/parcels/CDH/lib/spark/lib/spark-assembly.jar Properties: Spark Master : yarn Mode : cluster App name : Final Solution
... View more
09-01-2017
06:33 AM
Yes, other jobs are wokring fine. I am on CDH 5.9, so just tried with jar path in options list which is atleast running the job now. I am encountering below error while running the job, Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, requirement failed
java.lang.IllegalArgumentException: requirement failed
... View more
08-24-2017
08:58 PM
I am trying to run the Spark action in Oozie with spark .jar Provided below values in Hue Oozie Spark action, Jar/py Name: solution.jar Main Class: Module.final_solution Files: /user/hadoop/solution/solution.jar Below are the properties: Spark Master: yarn-master Mode: Cluster App Name: Final Solution Getting below error while running Oozie action with above configurations, Same jar working fine from Edge node with Spark submit. Any help appriciated.
... View more
Labels:
- Labels:
-
Apache Oozie
08-16-2016
01:41 PM
1 Kudo
This issue is resolved now. Took below steps, 1. Removed oozie.libpath line from job.properties 2. Created lib folder in workspace and placed sqljdbc41.jar in lib folder So total 3 things placed in workspace, job.properties, workflow.xml and lib folder.
... View more
08-16-2016
07:08 AM
I have kept only 1 driver sqljdbc41.jar in all directories and Oozie shared lib path. But still I am getting below error, Log Type: stderr
Log Length: 3696
log4j:ERROR Could not find value for key log4j.appender.CLA
log4j:ERROR Could not instantiate appender named "CLA".
log4j:WARN No appenders could be found for logger (org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Aug 16, 2016 9:45:42 AM com.microsoft.sqlserver.jdbc.SQLServerConnection <init>
SEVERE: Java Runtime Environment (JRE) version 1.7 is not supported by this driver. Use the sqljdbc4.jar class library, which provides support for JDBC 4.0.
java.lang.UnsupportedOperationException: Java Runtime Environment (JRE) version 1.7 is not supported by this driver. Use the sqljdbc4.jar class library, which provides support for JDBC 4.0. Even I tried with driver sqljdbc4.jar, but it keep asking other version of driver other than present in directories. After running a workflow, I realized there is workflow.xml generated in workspace but no job.properties. So I manually created job.properties as below and added in workspace, Below is workflow.xml generated , I am not entering any other job properties or parameters in workflow. Not sure if I am doing something wrong here, as I can not see jobTracker and nameNode actual names is workflow.xml file. Please guide in this.
... View more
08-14-2016
07:14 PM
Thank you Harsh. It was my bad to overlook the actul commands inside the Sqoop action. Now I am getting below error, Log Type: stderr Log Length: 3696 log4j:ERROR Could not find value for key log4j.appender.CLA log4j:ERROR Could not instantiate appender named "CLA". log4j:WARN No appenders could be found for logger (org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Aug 14, 2016 9:47:55 PM com.microsoft.sqlserver.jdbc.SQLServerConnection <init> SEVERE: Java Runtime Environment (JRE) version 1.7 is not supported by this driver. Use the sqljdbc4.jar class library, which provides support for JDBC 4.0. java.lang.UnsupportedOperationException: Java Runtime Environment (JRE) version 1.7 is not supported by this driver. Use the sqljdbc4.jar class library, which provides support for JDBC 4.0. I have below drivers in Oozie sahred lib / user/ oozie/ share/ lib/ lib_20150429135817/ sqoop, sqljdbc.jar, sqljdbc4.jar, sqljdbc41.jar, sqljdb42.jar Also have below drivers, Edge node Sqoop2: /var/lib/sqoop2 sqljdbc41.jar Edge node Sqoop: /var/lib/sqoop sqljdbc.jar, sqljdbc4.jar, sqljdbc41.jar, sqljdb42.jar Sqoop2 Server: /var/lib/sqoop2 sqljdbc.jar, sqljdbc4.jar, sqljdbc41.jar, sqljdb42.jar Do I need to add sqljdbc4.jar on Edge node Sqoop2 directory as well remove drivers other than sqljdbc4.jar from all places ? What if in future I want to connect to another SQL Server with different version ? Are these drivers not come with older version compatibiltiy ? Thank you, Vinod
... View more
08-12-2016
08:51 AM
I am running Sqoop action in Oozie, with below script sqoop import --connect jdbc:sqlserver://SQLM03;database=test --username data_user --password pass1 --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --table table123 -m 1 --target-dir /user/data_user/sqoop I have placed SQL Server driver jar in Oozie share lib /user/oozie/share/lib/lib_20150429135817/sqoop I am able to import data using sqoop from command prompt using same script, but getting below error while running from Oozie Log Type: stderr
Log Length: 539
log4j:ERROR Could not find value for key log4j.appender.CLA
log4j:ERROR Could not instantiate appender named "CLA".
log4j:WARN No appenders could be found for logger (org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
No such sqoop tool: sqoop. See 'sqoop help'.
Intercepting System.exit(1)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] Tried to search it in previous posts, but no solution found. Please suggest. Regards, Vinod
... View more
Labels:
07-19-2016
08:26 AM
How can I connect HDFS through .NET ? I want to pass some value via .NET, search the value in HDFS files and return the file name based on matching value. If not possible to connect HDFS directly through .NET, then can I connect HIVE using ODBC connection throught .NET ? Using CDH 5.3.3 Please suggest.
... View more
Labels:
- Labels:
-
HDFS
07-11-2016
06:59 AM
Can I import data in JSON format from Oracle tables using Sqoop ? I am using below version, Sqoop 1.4.5-cdh5.3.3 Thanks, Vinod
... View more
Labels:
- Labels:
-
Apache Sqoop
-
HDFS
06-17-2016
07:41 AM
Using Flume spooldir source to pull files with Flume 1.5.0-cdh5.3.3 version. Everything working fine as expected, but log file is just getting bigger and bigger becuase of below info twice per second 16/06/17 09:19:58 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown. 16/06/17 09:19:58 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown. 16/06/17 09:19:59 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown. 16/06/17 09:19:59 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown. 16/06/17 09:20:00 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown. 16/06/17 09:20:00 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown. Not sure if this is Flume issue or I am missing some configuration here. Please suggest.
... View more
Labels:
- Labels:
-
Apache Flume