Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Problem in spark jobs

Solved Go to solution
Highlighted

Problem in spark jobs

I am using spark to insert data into hive from avro files.On successful completion i am sending email using spark only.In spark submit command when i am running the jar in the client mode the email is triggering, and when running in cluster mode the email do not trigger.

What is the possible solution for this?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Problem in spark jobs

@Aditya Jadia

This could probably be related to firewall/communication issue from within cluster worker nodes to email server. To find out exactly what went wrong you should fetch the application logs. yarn logs -applicationId <appId> - Also test email server from within cluster worker nodes with different tool (like telnet) to check connectivity.

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

View solution in original post

3 REPLIES 3
Highlighted

Re: Problem in spark jobs

Expert Contributor

In client mode your jar is running on edge node or local machine which will have smtp connectivity. In cluster mode, any of the data nodes could run your jar, so you will need to check connectivity to smtp from all nodes.

Highlighted

Re: Problem in spark jobs

@Aditya Jadia

This could probably be related to firewall/communication issue from within cluster worker nodes to email server. To find out exactly what went wrong you should fetch the application logs. yarn logs -applicationId <appId> - Also test email server from within cluster worker nodes with different tool (like telnet) to check connectivity.

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

View solution in original post

Highlighted

Re: Problem in spark jobs

Thanks.Logs helped me

Don't have an account?
Coming from Hortonworks? Activate your account here