How can I read Mainframe file which is in EBCDIC format?

[#1]

Hi, We have huge number of mainframe files, which are in EBCDIC format. these files are created by mainframe systems. These files are now stored in HDFS as EBCDIC files. I have a need to read these files, (copy books available), split them into multiple files based on record type, and store them as ASCII files in HDFS.

[#2]

@karthick baskaran

You can use following project. It uses JRecord to do the conversion.

https://github.com/tmalaska/CopybookInputFormat

You can use Spark to read your EBCDIC files from hadoop and convert them to ASCII using above library.

[#3]

Hi Mqureshi

I'm very new to this, so i dont know how to do this. But i will try to check some online resources and try this. if i struggle i will come back and ask you for help. if it works, i will let you know about that also. Thanks.

[#4]

Sqoop has a connector to import Mainframe data into HDFS:

https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_literal_sqoop_import_mainframe_literal

[#5]

But this will work only if the file in Mainframe is like a normal test file right. but in my case, the files are in EBCDIC format (has multiple occurrences), some junk values... so can we still do this with the Sqoop connector. i did go over the details in the link and couldnt see anything related to EBCDIC file. but if you think this is going to work, please share more details, and Im interested in knowing about this.

[#6]

The connector is a contribution from Syncsort. Syncsort has decades of experience with building tools for Mainframe data ingestion.

I have used Sqoop extensively; however, never for Mainframe data. Syncsort states: "Each data set will be stored as a separate HDFS file and EBCDIC encoded fixed length data will be stored as ASCII encoded variable length text on HDFS"

http://blog.syncsort.com/2014/06/big-data/big-iron-big-data-mainframe-hadoop-apache-sqoop/

There is also a Spark connector to import Mainframe data:

https://github.com/Syncsort/spark-mainframe-connector

[#7]

Hi Mqureshi

I'm very new to this, so i dont know how to do this. But i will try to check some online resources and try this. if i struggle i will come back and ask you for help. if it works, i will let you know about that also. Thanks.

[#8]

fair enough. see the new answer by @bpreachuk. I was assuming you are loking for free tools but if you can get syncsort or if you already have it, that's the easiest way to do this.

[#9]

@karthick baskaran

I am not sure if you have the ability to use a 3rd Party tool, bit one of our trusted partners is Syncsort. If you've used the mainframe before you'll know who they are. Dealing with EBCDIC conversions, Copybooks, etc. are features that they excel at and provide in their flagship tool. It's called DMX-h and it would do what you need (in fact it can be your Data Integtration tool for all data, not just mainframe). http://www.syncsort.com/en/Products/BigData/DMXh

[#10]

Hi Bpreachuk

Thanks for the answer. No, we do not have the option to buy syncsort.