What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

[#1]

I know Syncsort is a possible solution here, but wanted to check if we HDF can do the job and if we have any other recommendation other than Syncsort ??

[#2]

Most ETL vendors have mainframe integration. For near realtime integration, you may want to look into Attunity.

[#3]

Thanks @ajaysingh

[#4 · Accepted solution]

@hduraiswamy - in order of preference

• SyncSort

• Use the mainframe’s native JDBC services – often unacceptable as the mainframe must consume additional MIPS to convert into JDBC types before sending over the net
• Use this open serde which unfortunately skips reading everything except fixed length fields, severely limiting usefulness
• I've heard about LegStar being used for similar projects, but am not sure how.

[#5]

Thanks @Randy Gelhausen

[#6]

Java Library + Spark => Magic

https://github.com/gmallard/packed-decimal

You could also have that in a dataflow in NiFi

1. get the file via NiFi GetFile

2. ExecuteStreamCommand packed-decimal Java class

2b. or call via Kafka/JMS to Java or Spark program

3. Insert or save as ORC

4. Create a hive table on top