Hi All, Can someone please kind to tell me how to import the data from MongoDB to HBase using Spark or Without Using Spark. If not any other Way. Regards, Vijay ... View more

Hi all, I have created a shell action in oozie workflow. It was executing perfectly a while ago. Later day it started showing below error. Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "checkForFileExistance.sh" (in directory "/data1/hadoop/yarn/local/usercache/hadoop/appcache/application_1471524954637_0316/container_e27_1471524954637_0316_01_000002"): error=2, No such file or directory Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "checkForFileExistance.sh" (in directory "/data1/hadoop/yarn/local/usercache/hadoop/appcache/application_1471524954637_0316/container_e27_1471524954637_0316_01_000002"): error=2, No such file or directory java.io.IOException: Cannot run program "checkForFileExistance.sh" (in directory "/data1/hadoop/yarn/local/usercache/hadoop/appcache/application_1471524954637_0316/container_e27_1471524954637_0316_01_000002"): error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:95) at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:57) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:49) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:248) at java.lang.ProcessImpl.start(ProcessImpl.java:134) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ... 17 more Please help me with the error. Please not that the shell script is available in lib folder under workflow directory. ... View more

Hi, I We have installed HDP-2.4.0.0. As per the requirement i need to configure oozie job w.r.t spark action. I have written the code. Workflow.xml: <?xml version="1.0"?> <workflow-app name="${OOZIE_WF_NAME}" xmlns="uri:oozie:workflow:0.5"> <global> <configuration> <property> <name>oozie.launcher.yarn.app.mapreduce.am.env</name> <value>SPARK_HOME=/usr/hdp/2.4.0.0-169/spark/</value> </property> </configuration> </global> <start to="spark-mongo-ETL"/> <action name="spark-mongo-ETL"> <spark xmlns="uri:oozie:spark-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <master>yarn-cluster</master> <mode>cluster</mode> <name>SparkMongoLoading</name> <class>com.SparkSqlExample</class> <jar>${nameNode}${WORKFLOW_HOME}/lib/SparkParquetExample-0.0.1-SNAPSHOT.jar</jar> </spark> <ok to="End"/> <error to="killAction"/> </action> <kill name="killAction"> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="End"/> </workflow-app> Job.properties: nameNode=hdfs://nameNode1:8020 jobTracker=yarnNM:8050 queueName=default user.name=hadoop oozie.libpath=/user/oozie/share/lib/ oozie.use.system.libpath=true WORKFLOW_HOME=/user/hadoop/SparkETL OOZIE_WF_NAME=Spark-Mongo-ETL-wf SPARK_MONGO_JAR=${nameNode}${WORKFLOW_HOME}/lib/SparkParquetExample-0.0.1-SNAPSHOT.jar oozie.wf.application.path=${nameNode}/user/hadoop/SparkETL/ Under lib folder 2 jar are placed SparkParquetExample-0.0.1-SNAPSHOT.jar spark-assembly-1.6.0.2.4.0.0-169-hadoop2.7.1.2.4.0.0-169.jar When I submit the oozie job, the action was killed. Error : Error: java.lang.UnsupportedOperationException: Not implemented by the TFS FileSystem implementation at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:217) at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2624) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:342) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:270) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:432) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164) at org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:256) at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:314) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:146) Also let me know how to pass the jars and files explicitly in the workflow. Command : spark-submit --class com.SparkSqlExample --master yarn-cluster --num-executors 2 --driver-memory 1g --executor-memory 2g --executor-cores 2 --files /usr/hdp/current/spark-client/conf/hive-site.xml --jars /usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar,/usr/hdp/current/spark-client/lib/jackson-core-2.4.4.jar,/usr/hdp/current/spark-client/lib/mongo-hadoop-spark-1.5.2.jar,/usr/share/java/slf4j-simple-1.7.5.jar,/usr/hdp/current/spark-client/lib/spark-core_2.10-1.6.0.jar,/usr/hdp/current/spark-client/lib/spark-hive_2.10-1.6.0.jar,/usr/hdp/current/spark-client/lib/spark-sql_2.10-1.6.0.jar,/usr/hdp/current/spark-client/lib/mongo-hadoop-core-1.5.2.jar,/usr/hdp/current/spark-client/lib/spark-avro_2.10-2.0.1.jar,/usr/hdp/current/spark-client/lib/spark-csv_2.10-1.4.0.jar,/usr/hdp/current/spark-client/lib/spark-mongodb_2.10-0.11.2.jar,/usr/hdp/current/spark-client/lib/spark-streaming_2.10-1.6.0.jar,/usr/hdp/current/spark-client/lib/commons-csv-1.1.jar,/usr/hdp/current/spark-client/lib/mongodb-driver-3.2.2.jar,/usr/hdp/current/spark-client/lib/mongo-hadoop-master-1.5.2.jar,/usr/hdp/current/spark-client/lib/mongo-java-driver-3.2.2.jar,/usr/hdp/current/spark-client/lib/spark-1.6.0.2.4.0.0-169-yarn-shuffle.jar --conf spark.yarn.jar=hdfs:///user/spark/spark-assembly-1.6.0.2.4.0.0-169-hadoop2.7.1.2.4.0.0-169.jar --conf spark.yarn.executor.memoryOverhead=512 /home/hadoop/SparkParquetExample-0.0.1-SNAPSHOT.jar The above command executes successfully Can anyone suggest me the solution. ... View more

<a href="/storage/attachments/5152-img-22062016-154852.png">img-22062016-154852.png</a>Hi Team, We are using HDP 2.4.0.0-169 which is installed on Ubuntu 14.04. There is a spark- mongo application which extracts the data from mongodb into HDFS. This application ins executed by using 3 spark modes which are local[*] , yarn-client and yarn-cluster. All these 3 spark-submit commands works from the command prompt. We have written an oozie workflow to execute the job on hourly basis. While executing the oozie job, job is in RUNNING state but failed due to SLA error. I have followed steps as mentioned below. I have changed the propeties for YARN in ambari yarn.nodemanager.resource.memory-mb = 67436288 and increased upto 200999168 yarn.scheduler.minimum-allocation-mb = 1024 and increased to 2048 yarn.scheduler.maximum-allocation-mb = 8192 and decreased to 6144 2.I have also added below changes in custom spark-defaults : spark.authenticate = false spark.driver.extraLibraryPath = /usr/hdp/2.4.0.0-169/hadoop/lib/native spark.dynamicAllocation.executorIdleTimeout = 60 spark.dynamicAllocation.schedulerBacklogTimeout = 1 spark.executor.extraLibraryPath = /usr/hdp/2.4.0.0-169/hadoop/lib/native spark.serializer = org.apache.spark.serializer.KryoSerializer spark.yarn.am.extraLibraryPath = /usr/hdp/2.4.0.0-169/hadoop/lib/native spark.yarn.config.gatewayPath = /usr/hdp spark.yarn.config.replacementPath = {{HADOOP_COMMON_HOME}}/../../.. spark.yarn.jar = local:/usr/hdp/2.4.0.0-169/spark/lib/spark-assembly-1.6.0.2.4.0.0-169-hadoop2.7.1.2.4.0.0-169.jar 3. Added below property in advanced hadoop-env: export SPARK_HOME=/usr/hdp/2.4.0.0-169/spark 4.I have added spark jars in the oozie lib folder with required permissions. Tried 777 and 755 permissions for all jars. But no luck: spark-assembly-1.6.0.2.4.0.0-169-hadoop2.7.1.2.4.0.0-169.jar spark-1.6.0.2.4.0.0-169-hadoop2.7.1.2.4.0.0-169-yarn-shuffle.jar 5.Also added the above mentioned jars and below mentioned jars in oozie share folder with required permission but no luck /user/oozie/share/lib/lib_20160420150601/oozie Code is defined as follows: workflow.xml: <?xml version="1.0"?> <workflow-app name="sparkmongo" xmlns="uri:oozie:workflow:0.5"> <start to="spark-mongo-ETL"/> <action name="spark-mongo-ETL"> <spark xmlns="uri:oozie:spark-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <master>${master}</master> <mode>client</mode> <name>SparkMongoLoading</name> <class>com.snapfish.spark.etl.MongoETL</class> <jar>/user/hadoop/Sparkmongo/lib/original-etl-0.0.1-SNAPSHOT.jar</jar> <spark-opts> spark.driver.extraClassPath=hdfs://namenode1:8020/user/oozie/share/lib/lib_20150711021244/spark/* spark.yarn.historyServer.address=http://yarnNM:19888/ spark.eventLog.dir=hdfs://namenode1:8020/spark-history spark.eventLog.enabled=true </spark-opts> <arg>orders</arg> </spark> <ok to="End"/> <error to="killAction"/> </action> <kill name="killAction"> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="End"/> </workflow-app> Tried removing below property. but no luck spark.driver.extraClassPath=hdfs://namenode1:8020/user/oozie/share/lib/lib_20150711021244/spark/* spark.yarn.historyServer.address=http://yarnNM:19888/ spark.eventLog.dir=hdfs://namenode1:8020/spark-history spark.eventLog.enabled=true Jars in lib folder : mongo-hadoop-spark-1.5.2.jar mongo-java-driver-3.2.2.jar mongodb-driver-3.2.2.jar original-etl-0.0.1-SNAPSHOT.jar spark-1.6.0.2.4.0.0-169-yarn-shuffle.jar spark-assembly-1.6.0.2.4.0.0-169-hadoop2.7.1.2.4.0.0-169.jar spark-core_2.10-1.6.0.jar spark-mongodb_2.10-0.11.2.jar Tried removing all jars except below two jars but no luck: original-etl-0.0.1-SNAPSHOT.jar spark-assembly-1.6.0.2.4.0.0-169-hadoop2.7.1.2.4.0.0-169.jar job.properties : nameNode=hdfs://namenode1:8020 jobTracker=yarnNM:8050 master=yarn-client queueName=default oozie.use.system.libpath=true oozie.libpath=/user/oozie/share/lib/ user.name=hadoop mapreduce.job.username=yarn oozie.wf.application.path=${nameNode}/user/hadoop/Sparkmongo/ Also changed the below properties but no luck: nameNode=hdfs://namenode1:8020 jobTracker=yarnNM:8050 master=yarn-client queueName=default oozie.use.system.libpath=true user.name=hadoop oozie.wf.application.path=${nameNode}/user/hadoop/Sparkmongo/ nameNode=hdfs://namenode1:8020 jobTracker=yarnNM:8050 master=yarn-client queueName=default oozie.use.system.libpath=true oozie.libpath=/user/oozie/share/lib/ user.name=hadoop oozie.wf.application.path=${nameNode}/user/hadoop/Sparkmongo/ After trying all the combinations i am facing some issue which is attached.capture1.jpgcapture2.jpg ... View more