Cloudera Community
nikil_katturi
user rank
Explorer

Re: spark2 job on oozie

by Explorer in Support Questions
‎08-21-2018 07:29 PM
‎08-21-2018 07:29 PM
@Felix Albani i am facing the issues while i am running the job in oozie . Below is the error 2018-08-21 14:27:16,632 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.IllegalArgumentException: Invalid ContainerId: container_e63_1532612094367_17160_02_000001 at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1343) ... View more

Re: spark2 job on oozie

by Explorer in Support Questions
‎08-21-2018 07:27 PM
‎08-21-2018 07:27 PM
Hi , I am facing this issues . 2018-08-21 14:27:16,632 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.IllegalArgumentException: Invalid ContainerId: container_e63_1532612094367_17160_02_000001 at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1343) ... View more

Re: need help on sending an email using spark scal...

by Explorer in Support Questions
‎08-21-2018 06:55 PM
‎08-21-2018 06:55 PM
Thanks it works ... View more

need help on sending an email using spark scala

by Explorer in Support Questions
‎08-15-2018 05:26 PM
‎08-15-2018 05:26 PM
need help on sending an email using spark scala ... View more
Labels:

Re: How to retrieve last 24-hours data from Spark ...

by Explorer in Support Questions
‎08-03-2018 04:15 PM
‎08-03-2018 04:15 PM
Thanks that worked ... View more

How to retrieve last 24-hours data from Spark Data...

by Explorer in Support Questions
‎08-01-2018 10:13 PM
‎08-01-2018 10:13 PM
I have Timestamp column 2018-08-01 05:02:22.0 , i need to consider or filter the data from the datframe for last 24hrs. ... View more
Labels:

Re: Hive query error on partition table file not ...

by Explorer in Support Questions
‎07-30-2018 05:30 PM
‎07-30-2018 05:30 PM
Yes Daniel we are deleting the process . I dropped the table and recreated , it works now Thanks. ... View more

Hive query error on partition table file not exis...

by Explorer in Support Questions
‎07-30-2018 02:22 PM
‎07-30-2018 02:22 PM
I am seeing the below issues while i am trying to count(*) (doing a full table scan on a partitioned table) java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1532612094367_2667_7_00, diagnostics=[Vertex vertex_1532612094367_2667_7_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: realtimepos_pump_tenders initializer failed, vertex=vertex_1532612094367_2667_7_00 [Map 1], org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://hdp01prod/prod/landing/realtime_pos/pump/tenders/date_part=2018022309 Input path does not exist: hdfs://hdp01prod/prod/landing/realtime_pos/pump/tenders/date_part=2018022310 Input path does not exist: hdfs://hdp01prod/prod/landing/realtime_pos/pump/tenders/date_part=2018022311 Input path does not exist: hdfs://hdp01prod/prod/landing/realtime_pos/pump/tenders/date_part=2018022312 Input path does not exist: hdfs://hdp01prod/prod/landing/realtime_pos/pump/tenders/date_part=2018022313 I am not seeing the above partitioned files on HDFS. I am not able to figure out the reason and how to fix it ... View more
Labels:

Re: Unknown option --job=sample during spark submi...

by Explorer in Support Questions
‎07-27-2018 02:27 PM
‎07-27-2018 02:27 PM
No Felix , still it is not working. Actually i was using the same statement in the past . It did work but now i am facing issues with this. I am trying alternatives but you support would help me in figuring out the solution for this. Let me know if you need the code of mine , i can send it over to you. ... View more

Re: Unknown option --job=sample during spark submi...

by Explorer in Support Questions
‎07-24-2018 08:56 PM
‎07-24-2018 08:56 PM
Thanks for your reply ,i am trying to pass the job name from the command line to my below class to make it parameter --job=$EBDSJOB below is the class that reads the parameter from the spark submit. Please let me know if you have any best solution. val cmdLineParser = new scopt.OptionParser[JobConfiguration]("DataQuality") {<br> head("DataQuality")<br><br> opt[String]("job")<br> .text("Job Name")<br> .action((x, config) => config.copy(job = x))<br><br> opt[String]("appName")<br> .text("Application Name")<br> .action((x, config) => config.copy(appName = x))<br><br> opt[String]("sparkMaster")<br> .text("Spark Master")<br> .action((x, config) => config.copy(sparkMaster = x))<br><br> opt[String]("days")<br> .text("Date Range")<br> .action((x, config) => config.copy(dates = x))<br><br> opt[String]("config")<br> .text("Config Path")<br> .action((x, config) => config.copy(configPath = x))<br><br> help("help").text("prints this help text")<br> checkConfig(cmdLine => if (cmdLine.dates == null) failure("dates must be specified") else success) ... View more
Contact Me
Online
Offline
Last Visited
‎01-31-2019 07:34 PM
Community Statistics
Member Since ‎07-03-2018 06:36 PM
Last Visited ‎01-31-2019 07:34 PM
Posts 20