Cloudera Labs
Provide feedback on Cloudera Labs
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Envelope job not ending for a long time

Envelope job not ending for a long time

New Contributor

We are reading parquet files and running some sqls on it. We have a few dependencies to run a big query. All the steps and stages completed in a minute but the job does not end for more than 1 hour. The output file got written at 9:10 below but the job did not end until 9:57. Any idea what could be causing this ? 

 

We observe the problem when

1. Overwrite is specified for the output file is provided

2. when we have more than one output file written in the job.

 

19/04/19 09:09:57 INFO yarn.YarnAllocator: Received 4 containers from YARN, launching executors on 4 of them.
19/04/19 09:10:00 INFO yarn.YarnAllocator: Driver requested a total number of 1 executor(s).
19/04/19 09:10:01 INFO yarn.YarnAllocator: Driver requested a total number of 5 executor(s).
19/04/19 09:10:01 INFO yarn.YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 10168 MB memory (including 3000 MB of overhead)
19/04/19 09:10:01 INFO yarn.YarnAllocator: Submitted 1 unlocalized container requests.
19/04/19 09:10:02 INFO yarn.YarnAllocator: Driver requested a total number of 6 executor(s).
19/04/19 09:10:02 INFO yarn.YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 10168 MB memory (including 3000 MB of overhead)
19/04/19 09:10:02 INFO yarn.YarnAllocator: Submitted 1 unlocalized container requests.
19/04/19 09:10:02 INFO yarn.YarnAllocator: Launching container container_e101_1555242909053_1756_01_000006 on host r0138fn00.xxx.net for executor with ID 5
19/04/19 09:10:02 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
19/04/19 09:10:03 INFO yarn.YarnAllocator: Driver requested a total number of 8 executor(s).
19/04/19 09:10:03 INFO yarn.YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 10168 MB memory (including 3000 MB of overhead)
19/04/19 09:10:03 INFO yarn.YarnAllocator: Submitted 2 unlocalized container requests.
19/04/19 09:10:03 INFO yarn.YarnAllocator: Launching container container_e101_1555242909053_1756_01_000007 on host r0138fn00.xxx.net for executor with ID 6
19/04/19 09:10:03 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
19/04/19 09:10:04 INFO yarn.YarnAllocator: Driver requested a total number of 2 executor(s).
19/04/19 09:10:04 INFO yarn.YarnAllocator: Canceling requests for 2 executor container(s) to have a new desired total 2 executors.
19/04/19 09:10:04 INFO yarn.YarnAllocator: Driver requested a total number of 1 executor(s).
19/04/19 09:10:04 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).
19/04/19 09:10:24 INFO yarn.YarnAllocator: Driver requested a total number of 4 executor(s).
19/04/19 09:10:27 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).
19/04/19 09:10:30 INFO yarn.YarnAllocator: Driver requested a total number of 4 executor(s).
19/04/19 09:10:32 INFO yarn.YarnAllocator: Driver requested a total number of 2 executor(s).
19/04/19 09:10:32 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).
19/04/19 09:57:59 INFO yarn.ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. r01411n0v.xxx.net:12345
19/04/19 09:57:59 INFO yarn.ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. r01411n0v.xx.net:12345
19/04/19 09:57:59 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
19/04/19 09:57:59 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
19/04/19 09:57:59 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.

4 REPLIES 4
Highlighted

Re: Envelope job not ending for a long time

Rising Star

There was an issue similar to that a few versions ago, so if you are not on the latest release of Envelope then I would recommend using that.

 

Jeremy

Re: Envelope job not ending for a long time

New Contributor

We are on version 6.1. Is that the right version ?

Re: Envelope job not ending for a long time

Rising Star

Do you mean Envelope 0.6.1? If so, I would recommend upgrading to Envelope 0.7.0, which was released last week.

 

Jeremy

Re: Envelope job not ending for a long time

New Contributor

Thanks Jeremy, Updated to envelope 0.7.0. Let me use that and see.