Reply
New Contributor
Posts: 3
Registered: ‎08-07-2016

Spark encounter a strange stop between tasks

[ Edited ]

After upgraded CDH to 5.14.0-1.cdh5.14.0.p0.24 and SPARK2 to 2.2.0.cloudera2-1.cdh5.12.0.p0.232957, spark ETL script will encounter a strange stop between tasks. The spark script was not changed, it's same as before.

 

before.pngbeforeupgraded.pngupgrade

Updated:

 

I found the problem is that the pause encountered when spark reading thousands of files. But the previous version read thousands of files too, and there's no performance issue. Did I miss something here.

New Contributor
Posts: 3
Registered: ‎08-07-2016

Re: Spark encounter a strange stop between tasks

Finally, I rollback the CDH version to 5.13.1-1.cdh5.13.1.p0.2, and everything ok now...

So I guess it's not a spark problem, but other component.

Announcements