I have many files. one of which,say, header.csv, serves as a header file,i.e., it contains primary key(in database analogy) which servers as foreign key in the rest of the files. Now, I want to do FOREACH and FILTER as follows: A =LOAD 'header.csv' AS (Id:chararray,f1:chararrat,f2:chararray); B = LOAD 'file1.csv' AS (Id:chararray,t1:chararray); C = LOAD 'file2.csv' AS (Id:chararray) .......... D = foreach A { file1_filtered = FILTER file1 BY Id == A.Id; file2_filtered = FILTER file2 BY Id == A.Id; GENERATE file1_filtered,file2_filtered; }; Finally I need to access the relations file1_filtered and file2_filtered. When I follow this approach I got the following error: "ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse: <line 2651, column 28> expression is not a project expression: (Name: ScalarExpression) Type: null Uid: null)" How can I achieve this in Pig? ... View more

I have independent clusters of HDF and HDP. I wonder if I can have a single KDC Admin server for both of the clusters. If it is possible, how do I achieve that. ... View more

I have two files on my hdfs. One of the file(latest file) contains some updates on the other file(previous file). Now, I want to check if value of specific columns on the latest file also exist on the previous file(or if they have same value), and replace such records of the previous file with records of the latest file.(i.e. delete such records from the previous file and replace with records from latest file). That means, I need to check each record of the previous file against each record of the latest file based on specific columns. If matching is found, delete the whole record from the previous file , then replace with the record from latest file. How can I achieve this with Pig? thanks! ... View more

I have multiple files on my SFTP server with different filenames(filenames have date time appended).Now, I am using ListSftp, RouteOnAttribute, FetchSftp and putHdfs processors. But on the FetchSftp processor I have doubt on how to put all the files on the remote SFTP server to HDFS. Is there any option to provide the list of the file names to "Remote File" property of FetchSftp processor configuration? Thanks! ... View more

I have 7 node Kerberized HDP cluster. I have installed apache Nifi on one of my HDP cluster nodes just for testing purpose. When I try to configure putHdfs processor, the following warning pops up: I tried to set the Kerberos properties as follows: In addition to this, I set nifi.kerberos.krb5.file=/etc/krb5.conf in the nifi.properties file. What is the correct configuration(on the Nifi host or HDFS host) for the putHdfs processor to work properly in this case? Do I need to create Kerberos principal and Kerberos keytab file for the Nifi? Which service's principal or keytab file am I required to provide for "Kerberos Principal" and "Kerberos Keytab" fields in the putHdfs processor configuration?(is it the Nifi's or the Hdfs?) Thanks, ... View more