You can make a DataFrame over all the files and then filter out the lines you don't want. You can make a DataFrame for just the files you want, then union them together. Both are viable. If you're saying different data types are mixed into sections of each file, that's harder, as you need to use something like mapPartitions to carefully process each file 3 times.
... View more
