Reply
New Contributor
Posts: 4
Registered: ‎04-19-2019

Envelope job not ending for a long time

We are reading parquet files and running some sqls on it. We have a few dependencies to run a big query. All the steps and stages completed in a minute but the job does not end for more than 1 hour. The output file got written at 9:10 below but the job did not end until 9:57. Any idea what could be causing this ? 

 

We observe the problem when

1. Overwrite is specified for the output file is provided

2. when we have more than one output file written in the job.

 

19/04/19 09:09:57 INFO yarn.YarnAllocator: Received 4 containers from YARN, launching executors on 4 of them.
19/04/19 09:10:00 INFO yarn.YarnAllocator: Driver requested a total number of 1 executor(s).
19/04/19 09:10:01 INFO yarn.YarnAllocator: Driver requested a total number of 5 executor(s).
19/04/19 09:10:01 INFO yarn.YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 10168 MB memory (including 3000 MB of overhead)
19/04/19 09:10:01 INFO yarn.YarnAllocator: Submitted 1 unlocalized container requests.
19/04/19 09:10:02 INFO yarn.YarnAllocator: Driver requested a total number of 6 executor(s).
19/04/19 09:10:02 INFO yarn.YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 10168 MB memory (including 3000 MB of overhead)
19/04/19 09:10:02 INFO yarn.YarnAllocator: Submitted 1 unlocalized container requests.
19/04/19 09:10:02 INFO yarn.YarnAllocator: Launching container container_e101_1555242909053_1756_01_000006 on host r0138fn00.xxx.net for executor with ID 5
19/04/19 09:10:02 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
19/04/19 09:10:03 INFO yarn.YarnAllocator: Driver requested a total number of 8 executor(s).
19/04/19 09:10:03 INFO yarn.YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 10168 MB memory (including 3000 MB of overhead)
19/04/19 09:10:03 INFO yarn.YarnAllocator: Submitted 2 unlocalized container requests.
19/04/19 09:10:03 INFO yarn.YarnAllocator: Launching container container_e101_1555242909053_1756_01_000007 on host r0138fn00.xxx.net for executor with ID 6
19/04/19 09:10:03 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
19/04/19 09:10:04 INFO yarn.YarnAllocator: Driver requested a total number of 2 executor(s).
19/04/19 09:10:04 INFO yarn.YarnAllocator: Canceling requests for 2 executor container(s) to have a new desired total 2 executors.
19/04/19 09:10:04 INFO yarn.YarnAllocator: Driver requested a total number of 1 executor(s).
19/04/19 09:10:04 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).
19/04/19 09:10:24 INFO yarn.YarnAllocator: Driver requested a total number of 4 executor(s).
19/04/19 09:10:27 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).
19/04/19 09:10:30 INFO yarn.YarnAllocator: Driver requested a total number of 4 executor(s).
19/04/19 09:10:32 INFO yarn.YarnAllocator: Driver requested a total number of 2 executor(s).
19/04/19 09:10:32 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).
19/04/19 09:57:59 INFO yarn.ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. r01411n0v.xxx.net:12345
19/04/19 09:57:59 INFO yarn.ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. r01411n0v.xx.net:12345
19/04/19 09:57:59 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
19/04/19 09:57:59 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
19/04/19 09:57:59 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.

Cloudera Employee
Posts: 50
Registered: ‎08-26-2015

Re: Envelope job not ending for a long time

There was an issue similar to that a few versions ago, so if you are not on the latest release of Envelope then I would recommend using that.

 

Jeremy

New Contributor
Posts: 4
Registered: ‎04-19-2019

Re: Envelope job not ending for a long time

We are on version 6.1. Is that the right version ?

Cloudera Employee
Posts: 50
Registered: ‎08-26-2015

Re: Envelope job not ending for a long time

Do you mean Envelope 0.6.1? If so, I would recommend upgrading to Envelope 0.7.0, which was released last week.

 

Jeremy

Highlighted
New Contributor
Posts: 4
Registered: ‎04-19-2019

Re: Envelope job not ending for a long time

Thanks Jeremy, Updated to envelope 0.7.0. Let me use that and see.