Parquet as Array[Byte] to DataFrame without writing to disk

JoeR — Fri, 29 Apr 2022 08:47:14 GMT

From an API I receive a parquet as an Array[Byte] (stored here as parquetPayload), and want to convert that into a DataFrame. My current function is below, and includes a write to disk

val tempFilePath = new Path("/tmp/", java.util.UUID.randomUUID().toString + ".tmp") val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration) val stream = fs.create(tempFilePath, true) try { stream.write(parquetPayload, 0, parquetPayload.length) } finally { stream.close() } val df = spark.read.parquet(tempFilePath.toString)

This works, but I would like to avoid the write to disk, and keep this all in memory. Is this possible?

Re: Parquet as Array[Byte] to DataFrame without writing to disk

RangaReddy — Fri, 29 Apr 2022 12:29:57 GMT

Hi @JoeR

Spark will support reading files with multiple file formats like parquet, orc, json, xml, avro,csv etc. I think there is no direct mechanism to read the data from the payload.

If I found a different solution, I will share it with you.

question Parquet as Array[Byte] to DataFrame without writing to disk in Support Questions

Parquet as Array[Byte] to DataFrame without writing to disk

Re: Parquet as Array[Byte] to DataFrame without writing to disk