question Re: What is the recommended method to ingest Binary files from Mainframes ? Can sqoop handle this type of transfer in Archives of Support Questions (Read Only)

What is the recommended method to ingest Binary files from Mainframes ? Can sqoop handle this type of transfer

pbalasundaram — Sat, 10 Oct 2015 02:16:06 GMT

Re: What is the recommended method to ingest Binary files from Mainframes ? Can sqoop handle this type of transfer

deepesh1 — Sat, 10 Oct 2015 02:18:47 GMT

Check out http://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_large_objects

Re: What is the recommended method to ingest Binary files from Mainframes ? Can sqoop handle this type of transfer

pbalasundaram — Sat, 10 Oct 2015 03:30:11 GMT

Thank you Deepesh , The reply is appreciated. Large objects may be one way for generic binary object archival.

-- Adding a few more details to this topic.

In sqoop 1.4.6 a mainframe import option is introduced.

http://blog.syncsort.com/2014/06/big-iron-big-data-mainframe-hadoop-apache-sqoop/

The above Syncsort blog suggests that some capabilities like the ability to use cobol copybook metadata, vsam fileformats, Packed-Decimal, and translation of EBCDIC encoded fixed length binary data to ASCII encoded variable length text in HDFS;

It suggests that these capabilities may not be available in Open source Sqoop.

If there are any prior projects where we have experience handling this type of mainframe based dataset please add more details of implementation.

Re: What is the recommended method to ingest Binary files from Mainframes ? Can sqoop handle this type of transfer

bdavay — Thu, 26 Nov 2015 03:10:09 GMT

Hey

We have 2 more clients who wants to ingest data from Mainframe VSAM file format. Is syncsort a partnet with Hortonworks ? How do we proceed to go with Syncsort route ? or any suggestions ?

Re: What is the recommended method to ingest Binary files from Mainframes ? Can sqoop handle this type of transfer

drice1 — Thu, 26 Nov 2015 04:41:12 GMT

@pbalasundaram I've seen a customer use JRecord to build a mapreduce inputformat for HDFS, mapreduce, etc. Look around within github and you should see examples. Obviously this would take more work than just using something off the shelf. Besides syncsort, there are capabilities within attunity also.

Re: What is the recommended method to ingest Binary files from Mainframes ? Can sqoop handle this type of transfer

drice1 — Thu, 26 Nov 2015 04:43:08 GMT

Yes syncsort is a partner look here.

Re: What is the recommended method to ingest Binary files from Mainframes ? Can sqoop handle this type of transfer

LesterMartin — Wed, 02 Dec 2015 03:10:09 GMT

I'm also wondering if anyone has used SyncSort's open source contribution, http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_literal_sqoop_import_mainframe_literal, or their proprietary software, http://www.syncsort.com/en/Solutions/Hadoop-Solutions/Mainframe-Offload-to-Hadoop, in getting data from the mainframe.

Re: What is the recommended method to ingest Binary files from Mainframes ? Can sqoop handle this type of transfer

ashok_padmanabh — Wed, 16 Dec 2015 03:14:44 GMT

Hi - We tried to import this , but are facing issue with the GDG format, the basic open source sqoop 1272 doesn't support many features I guess. Are there any alternatives?