Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

avatar
Super Collaborator

I know Syncsort is a possible solution here, but wanted to check if we HDF can do the job and if we have any other recommendation other than Syncsort ??

1 ACCEPTED SOLUTION

avatar

@hduraiswamy - in order of preference

  • SyncSort
  • Use the mainframe’s native JDBC services – often unacceptable as the mainframe must consume additional MIPS to convert into JDBC types before sending over the net
  • Use this open serde which unfortunately skips reading everything except fixed length fields, severely limiting usefulness
  • I've heard about LegStar being used for similar projects, but am not sure how.

View solution in original post

5 REPLIES 5

avatar
Explorer

Most ETL vendors have mainframe integration. For near realtime integration, you may want to look into Attunity.

avatar
Super Collaborator

Thanks @ajaysingh

avatar

@hduraiswamy - in order of preference

  • SyncSort
  • Use the mainframe’s native JDBC services – often unacceptable as the mainframe must consume additional MIPS to convert into JDBC types before sending over the net
  • Use this open serde which unfortunately skips reading everything except fixed length fields, severely limiting usefulness
  • I've heard about LegStar being used for similar projects, but am not sure how.

avatar
Super Collaborator

avatar
Master Guru

Java Library + Spark => Magic

https://github.com/gmallard/packed-decimal

You could also have that in a dataflow in NiFi

1. get the file via NiFi GetFile

2. ExecuteStreamCommand packed-decimal Java class

2b. or call via Kafka/JMS to Java or Spark program

3. Insert or save as ORC

4. Create a hive table on top