Hi I have a 2 part question: Source files are in HDFS, normal csv or text files. Part 1: Record counts. I have a huge file, that has more than 50 million records. The total size of the file will be 50GB Average (daily files). Is there an efficient way to do record count (i do not want to do wc -l). Would like to do this in native Hadoop or spark, so i can utilize the Hadoop resources. Part 2: Duplicate Check. For the same file mentioned above, I would like to know if there is way to do duplicate check, based on key fields? I would like to find if the file has duplicate records? if Yes, then i would like to capture them separately (this is optional). Right now, we have some solution that is running for a very long time (more than 30 mins), and i would like to know if there are other options like native map reduce or spark. ... View more

Hi, I have a need to create a batch process, where I have to read (select only a few columns and few records) from a set of mainframe files (stored in a Hadoop cluster), and then convert them into ASCII files and store them in HDFS. There will be more than 100 or 300 files like this. So is there a way to do this using HIVE, or PIG or SPARK? or Java? Also, i want to have an automated way to read these files (scheduled) and pass parameters during runtime (like file name, path, column names and filter condition to filter records). I came to know about the following (https://community.hortonworks.com/questions/80649/how-can-i-read-mainframe-file-which-is-in-ebcdic-f.html) So can you suggest me some ideas please. ... View more

Hi, We have huge number of mainframe files, which are in EBCDIC format. these files are created by mainframe systems. These files are now stored in HDFS as EBCDIC files. I have a need to read these files, (copy books available), split them into multiple files based on record type, and store them as ASCII files in HDFS. ... View more