@Raghavendra Gupta, have you tried Cobrix? ... View more

@Karthik Narayanan, you can use Cobrix to parse the EBCDIC files through Spark and stored them on HDFS using whatever format you want. It is open-source. DISCLAIMER: I work for ABSA and I am one of the developers behind this library. Our focus has been: 1) ease of use, 2) performance. ... View more

Very good survey. I would like, however, to introduce one more alternative for your appreciation. At ABSA we have been working on COBOL data source for Spark which we call Cobrix The performance results we have found so far are VERY encouraging and the ease of use is way ahead of Informatica's and similar tools. ... View more

Hi @Dongjoon Hyun. That definitely works, however, the requirements I had could not be addressed with that course of action. I was looking for a solution that works on Parquet files the same way Ranger works with Hive, for instance. I'd like to go to Ranger and set specific permissions directly on Parquet columns without having to first load the files into Hive. After better understanding how Ranger works I could realize that this is not possible, as Ranger works with hooks (plug-ins) to the tools it secures (HDFS, HBase, Hive, etc) and Parquet is simply a file format. A solution I started to investigate is an extension to the HDFS plug-in which could act on Parquet files, filtering access as specified through Ranger. With that solution, Parquet files could be secured at a column-level directly from Ranger as long as it's stored in HDFS. Anyway, thank you very much for the replies and also for checking on the result. ... View more

Hi @Dongjoon Hyun, thanks for the reply. The tutorial is great, very clear, however, how could I apply that to Parquet files? (sorry if a newbie question, but I'm indeed a newbie 🙂 ) ... View more