Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can I read Mainframe file which is in EBCDIC format?

Highlighted

How can I read Mainframe file which is in EBCDIC format?

Explorer

Hi, We have huge number of mainframe files, which are in EBCDIC format. these files are created by mainframe systems. These files are now stored in HDFS as EBCDIC files. I have a need to read these files, (copy books available), split them into multiple files based on record type, and store them as ASCII files in HDFS.

14 REPLIES 14
Highlighted

Re: How can I read Mainframe file which is in EBCDIC format?

Super Guru
@karthick baskaran

You can use following project. It uses JRecord to do the conversion.

https://github.com/tmalaska/CopybookInputFormat

You can use Spark to read your EBCDIC files from hadoop and convert them to ASCII using above library.

Highlighted

Re: How can I read Mainframe file which is in EBCDIC format?

Explorer

Hi Mqureshi

I'm very new to this, so i dont know how to do this. But i will try to check some online resources and try this. if i struggle i will come back and ask you for help. if it works, i will let you know about that also. Thanks.

Highlighted

Re: How can I read Mainframe file which is in EBCDIC format?

Highlighted

Re: How can I read Mainframe file which is in EBCDIC format?

Explorer

But this will work only if the file in Mainframe is like a normal test file right. but in my case, the files are in EBCDIC format (has multiple occurrences), some junk values... so can we still do this with the Sqoop connector. i did go over the details in the link and couldnt see anything related to EBCDIC file. but if you think this is going to work, please share more details, and Im interested in knowing about this.

Highlighted

Re: How can I read Mainframe file which is in EBCDIC format?

The connector is a contribution from Syncsort. Syncsort has decades of experience with building tools for Mainframe data ingestion.

I have used Sqoop extensively; however, never for Mainframe data. Syncsort states: "Each data set will be stored as a separate HDFS file and EBCDIC encoded fixed length data will be stored as ASCII encoded variable length text on HDFS"

http://blog.syncsort.com/2014/06/big-data/big-iron-big-data-mainframe-hadoop-apache-sqoop/

There is also a Spark connector to import Mainframe data:

https://github.com/Syncsort/spark-mainframe-connector

Highlighted

Re: How can I read Mainframe file which is in EBCDIC format?

Explorer

Hi Mqureshi

I'm very new to this, so i dont know how to do this. But i will try to check some online resources and try this. if i struggle i will come back and ask you for help. if it works, i will let you know about that also. Thanks.

Highlighted

Re: How can I read Mainframe file which is in EBCDIC format?

Super Guru

fair enough. see the new answer by @bpreachuk. I was assuming you are loking for free tools but if you can get syncsort or if you already have it, that's the easiest way to do this.

Highlighted

Re: How can I read Mainframe file which is in EBCDIC format?

@karthick baskaran

I am not sure if you have the ability to use a 3rd Party tool, bit one of our trusted partners is Syncsort. If you've used the mainframe before you'll know who they are. Dealing with EBCDIC conversions, Copybooks, etc. are features that they excel at and provide in their flagship tool. It's called DMX-h and it would do what you need (in fact it can be your Data Integtration tool for all data, not just mainframe). http://www.syncsort.com/en/Products/BigData/DMXh

Re: How can I read Mainframe file which is in EBCDIC format?

Explorer

Hi Bpreachuk

Thanks for the answer. No, we do not have the option to buy syncsort.

Don't have an account?
Coming from Hortonworks? Activate your account here