Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

SOLVED Go to solution
Highlighted

What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

Expert Contributor

I know Syncsort is a possible solution here, but wanted to check if we HDF can do the job and if we have any other recommendation other than Syncsort ??

1 ACCEPTED SOLUTION

Accepted Solutions

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

@hduraiswamy - in order of preference

  • SyncSort
  • Use the mainframe’s native JDBC services – often unacceptable as the mainframe must consume additional MIPS to convert into JDBC types before sending over the net
  • Use this open serde which unfortunately skips reading everything except fixed length fields, severely limiting usefulness
  • I've heard about LegStar being used for similar projects, but am not sure how.
5 REPLIES 5

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

New Contributor

Most ETL vendors have mainframe integration. For near realtime integration, you may want to look into Attunity.

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

Expert Contributor

Thanks @ajaysingh

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

@hduraiswamy - in order of preference

  • SyncSort
  • Use the mainframe’s native JDBC services – often unacceptable as the mainframe must consume additional MIPS to convert into JDBC types before sending over the net
  • Use this open serde which unfortunately skips reading everything except fixed length fields, severely limiting usefulness
  • I've heard about LegStar being used for similar projects, but am not sure how.

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

Expert Contributor

Re: What is recommended way of moving mainframe data in Packed-Decimal fomat to Hive, either as text or ORC format?

Super Guru

Java Library + Spark => Magic

https://github.com/gmallard/packed-decimal

You could also have that in a dataflow in NiFi

1. get the file via NiFi GetFile

2. ExecuteStreamCommand packed-decimal Java class

2b. or call via Kafka/JMS to Java or Spark program

3. Insert or save as ORC

4. Create a hive table on top