question Re: Problem in spark jobs in Support Questions

Problem in spark jobs

adityajadia2 — Mon, 28 May 2018 12:06:17 GMT

I am using spark to insert data into hive from avro files.On successful completion i am sending email using spark only.In spark submit command when i am running the jar in the client mode the email is triggering, and when running in cluster mode the email do not trigger.

What is the possible solution for this?

Re: Problem in spark jobs

umair_khan — Tue, 29 May 2018 00:24:11 GMT

In client mode your jar is running on edge node or local machine which will have smtp connectivity. In cluster mode, any of the data nodes could run your jar, so you will need to check connectivity to smtp from all nodes.

Re: Problem in spark jobs

falbani — Tue, 29 May 2018 00:39:00 GMT

@Aditya Jadia

This could probably be related to firewall/communication issue from within cluster worker nodes to email server. To find out exactly what went wrong you should fetch the application logs. yarn logs -applicationId <appId> - Also test email server from within cluster worker nodes with different tool (like telnet) to check connectivity.

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: Problem in spark jobs

adityajadia2 — Fri, 08 Jun 2018 16:10:15 GMT

Thanks.Logs helped me