Support Questions

Find answers, ask questions, and share your expertise

Problem in spark jobs

avatar

I am using spark to insert data into hive from avro files.On successful completion i am sending email using spark only.In spark submit command when i am running the jar in the client mode the email is triggering, and when running in cluster mode the email do not trigger.

What is the possible solution for this?

1 ACCEPTED SOLUTION

avatar
@Aditya Jadia

This could probably be related to firewall/communication issue from within cluster worker nodes to email server. To find out exactly what went wrong you should fetch the application logs. yarn logs -applicationId <appId> - Also test email server from within cluster worker nodes with different tool (like telnet) to check connectivity.

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

View solution in original post

3 REPLIES 3

avatar
Expert Contributor

In client mode your jar is running on edge node or local machine which will have smtp connectivity. In cluster mode, any of the data nodes could run your jar, so you will need to check connectivity to smtp from all nodes.

avatar
@Aditya Jadia

This could probably be related to firewall/communication issue from within cluster worker nodes to email server. To find out exactly what went wrong you should fetch the application logs. yarn logs -applicationId <appId> - Also test email server from within cluster worker nodes with different tool (like telnet) to check connectivity.

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

avatar

Thanks.Logs helped me