Member since
10-03-2018
6
Posts
0
Kudos Received
0
Solutions
11-30-2018
04:20 PM
Sorry, missed this. the issue here is that "S3" isn't a "real" filesystem, there's no file/directory rename, and instead we have to list every file created and copy it over. Which relies on listings being correct, which S3, being eventually consistent, doesn't always hold up. Looks like you've hit an inconsistency on a job commit To get consistent listings (HDP 3) enable S3Guard To avoid the slow rename process and the problems caused by inconistency within a single query, switch to the "S3A Committers" which come with Spark on HDP-3.0. These are specially designed to safely write work into S3 If you can't do either of those, you cannot safely use S3 as a direct destination of work. You should write into HDFS and then, afterwards, copy it to S3.
... View more
10-09-2018
02:20 PM
@Lakshmi Prathyusha, I'm not sure of how to do this in Scala. I guess you may have similar date time functions in Scala as well. You can apply this logic in Scala.
... View more
07-22-2019
05:12 AM
Im facing same issue . did anyone resolved it? please post here how it got fixed?
... View more