question Re: Putting SequenceFile key value into a data frame in Support Questions

Putting SequenceFile key value into a data frame

Seaport — Tue, 21 Apr 2026 07:57:23 GMT

I saved thousands of small json files in SequenceFile format to resolve the "small file issue". I use the following pyspark code to parse the json data from saved sequence files.

reader= sc.sequenceFile("/mysequencefile_dir", "org.apache.hadoop.io.Text", "org.apache.hadoop.io.Text") rdd=reader.map(lambda x: x[1]) mydf=spark.read.schema(myschema).json(rdd) mydf.show(truncate=False)

The code worked. However, I do not know how to put the key value from the sequence file, which is actually the original json file name, into the mydf dataframe. Please advise. Thank you.

Regards,

Re: Putting SequenceFile key value into a data frame

Seaport — Thu, 21 Apr 2022 00:46:58 GMT

I did a workaround by injecting the myfilepath element into the json string.

rdd=reader.map(lambda x: str(x[1])[0]+'"myfilepath":"'+x[0]+'",'+str(x[1])[1:])

It does not look like a very clean solution. Is there a better one? Thanks.

Regards

Re: Putting SequenceFile key value into a data frame

araujo — Thu, 21 Apr 2022 05:27:15 GMT

@Seaport ,

Please try the below:

import json def jsonize(k, v): ret = json.loads(v) ret.update({'key': k}) return ret ... rdd = reader.map(lambda x: jsonize(*x)) ...

You need to make sure your schema includes the added key column.

Cheers,

André

Re: Putting SequenceFile key value into a data frame

Seaport — Fri, 22 Apr 2022 00:07:30 GMT

André,

Thanks for the elegant solution.

Regards,