Support Questions
Find answers, ask questions, and share your expertise

What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

Expert Contributor

I know Syncsort is a possible solution here, but wanted to check if we HDF can do the job and if we have any other recommendation other than Syncsort ??

1 ACCEPTED SOLUTION

Accepted Solutions

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

@hduraiswamy - in order of preference

  • SyncSort
  • Use the mainframe’s native JDBC services – often unacceptable as the mainframe must consume additional MIPS to convert into JDBC types before sending over the net
  • Use this open serde which unfortunately skips reading everything except fixed length fields, severely limiting usefulness
  • I've heard about LegStar being used for similar projects, but am not sure how.

View solution in original post

5 REPLIES 5

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

New Contributor

Most ETL vendors have mainframe integration. For near realtime integration, you may want to look into Attunity.

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

Expert Contributor

Thanks @ajaysingh

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

@hduraiswamy - in order of preference

  • SyncSort
  • Use the mainframe’s native JDBC services – often unacceptable as the mainframe must consume additional MIPS to convert into JDBC types before sending over the net
  • Use this open serde which unfortunately skips reading everything except fixed length fields, severely limiting usefulness
  • I've heard about LegStar being used for similar projects, but am not sure how.

View solution in original post

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

Expert Contributor

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

Super Guru

Java Library + Spark => Magic

https://github.com/gmallard/packed-decimal

You could also have that in a dataflow in NiFi

1. get the file via NiFi GetFile

2. ExecuteStreamCommand packed-decimal Java class

2b. or call via Kafka/JMS to Java or Spark program

3. Insert or save as ORC

4. Create a hive table on top