Member since
07-15-2020
1
Post
0
Kudos Received
0
Solutions
07-15-2020
07:54 AM
I have a dozen or so large txt files (15GB) that I have been reading into Spark as csv, but the issue that I am running into is that the headers for these files are delimited by commas(,), whereas all of the data in said columns is delimited by pipes( | ). Does anyone know how I could replace the commas in the headers with pipes, or conversely replace the pipes in the rest of the data with commas? The current layout of the files make it so that all of the data in the files fall into the first column, which messes up the filter I am trying to run on said files. Any help is greatly appreciated.
... View more
Labels:
- Labels:
-
Apache Spark