question Re: How to split the dataframe of multiple files into multiple smaller dataframes in Spark? in Support Questions

question Re: How to split the dataframe of multiple files into multiple smaller dataframes in Spark? in Support Questions https://community.cloudera.com/t5/Support-Questions/How-to-split-the-dataframe-of-multiple-files-into-multiple/m-p/63161#M22479 You can make a DataFrame over all the files and then filter out the lines you don't want.You can make a DataFrame for just the files you want, then union them together.Both are viable. If you're saying different data types are mixed into sections of each file, that's harder, as you need to use something like mapPartitions to carefully process each file 3 times.   Fri, 29 Dec 2017 13:29:19 GMT srowen 2017-12-29T13:29:19Z