Support Questions
Find answers, ask questions, and share your expertise

What are Spark executors that runs from the datanode machines

Solved Go to solution
Highlighted

What are Spark executors that runs from the datanode machines

Executor is a distributed agent that is responsible for executing tasks.

this is very clear

but how to know if there are any issues with the executors that runs from datanode machine?

I asking this question because when I am looking on the datanode machine

I not see any logs that represented the executors , and I not understand how to trace problems about the exectores

the second important quastion:

heartbeat are sent from the executor to the driver

what are the logs that represented this heartbeat ?

how to know if there are any issue with heartbeat sending ?

87584-capture.png

Michael-Bronson
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: What are Spark executors that runs from the datanode machines

@Michael Bronson

In yarn master mode executors will run inside a yarn container.

Spark will launch an Application Master that will be responsible of negotiating the containers with Yarn. Having that said only nodes running Nodemanager are eligible to run executors.

First question: The executor logs you are looking for will be part of the yarn application logs for the container running on the specific node. (yarn logs -applicationId <appId>)

Second question: Executor will notify in case heartbeat fails to reach driver for some network problem/timeout. So this should be in the executor log that is part of the application logs.

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

View solution in original post

1 REPLY 1
Highlighted

Re: What are Spark executors that runs from the datanode machines

@Michael Bronson

In yarn master mode executors will run inside a yarn container.

Spark will launch an Application Master that will be responsible of negotiating the containers with Yarn. Having that said only nodes running Nodemanager are eligible to run executors.

First question: The executor logs you are looking for will be part of the yarn application logs for the container running on the specific node. (yarn logs -applicationId <appId>)

Second question: Executor will notify in case heartbeat fails to reach driver for some network problem/timeout. So this should be in the executor log that is part of the application logs.

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

View solution in original post