Hello everyone, I'm trying to configure Log4j for HDFS Namenode and Security Audit logs. In either way Size based rotation or Daily Rotation I should be able to rotate and delete the logs. However It is not working as expected. Can someone send me the best configurations in Log4j for HDFS NN/Audit Logs.Attaching the Log4j configurations . log4j.txt Help me to delete the logs once it reached 10 Days .I would like to Gzip the older files. Please find the below screenshot for the size consumed on the disk We would like to remove Logs based on Size for Audit logs . ... View more

Hello, We are trying to handle Inserts/Updates/Deletes in Nifi . Currently attached the flow which is taking longer time to process the records and inserts into Hbase . I need to use Compositeprimarykey(ServerSerialno,ServerName) .Based on the CompositeKey will do the Inserts/Updates/Deletes.Need to boost the performance by altering the flow/configs.Kindly assist me to use the right flow to achieve this. Attached the flow . ... View more

Hi All, Good day, I'm just checking routeontext processor to route values if matched then route for different action and if not matched route for different action . My Sample flow would be like this : GenerateFlowFile | RouteOnText --- Flow 1 ---> Putfile --- Flow 2 --> Putfile My Text content is simple My car color is Blue My car color is Yellow If Color is Blue -> goes to Flow1 If Color is Yellow -> Goes to Flow 2 Now I have just started the processor I can see queue in Generate flow File is queued upto 10000 and Flow 1 is having 6000 and Flow2 is having somewhat around 1000 with N number of duplicates . Now How can I restrict it to only given content without any repetition .SO expected output is only two text files on respective flows . not n number of values . How can I configure that way . Can you please high level information of why is it happening . Cannot terminate relationship in GenerateFlowFile as the connection is already given to next processor . One important thing is Flow 1 should be executed first and flow 2 should be executed after the flow 1 is completed . ... View more

Hi I need to load the JSON data into HBASE on the below conditions . If the record(composite primary key)(servername+serverid) is matched then update/delete the record , If not matched Insert the record. How Can I do this in Nifi with Hbase for larger dataset. forecasted data size would be 50-100TB approx for the next one year. optional if at possible , I need to discard the invalid entries from JSON on the flow using regular exp . Where I need to discard the entries and those discarded values creates empty space . Kindly let me know how to remove those empty space after removing the blank values. If possible How to avoid JSON breaking into bad values . I couldn't find good tutorial to Insert/Update in HBASE. Attached sample data test.txt ... View more

Hi All, I'm trying to update the JSON data into Hive using this approach https://goo.gl/J7chi3 . While doing this approach I have two issues. 1.My Input record is 20k and output record count is 23.5k approx . some json's are breaking creating duplicates 2.My Input record size is 10k and output record count is 20k . As per the link it should be updating the records if it is already present in the table . Can anyone guide me to do upserts in hive . Apart from the above mentioned methods , Few other failed methods for upserts I have tried to use Merge options in hive refer link : https://community.hortonworks.com/articles/97113/hive-acid-merge-by-example.html --> This merge is not suitable to merge more than 5 GB or more . Taking more hours to complete or not to complete or getting Heap memory error for even 6 GB data. Someone Suggested Merge with Source and Destination as partition. We will be getting error if the destination is partitioned since Merge cannot update the partition key value . Cluster ram size is 250GB Can anyone help me in this Please with definitive steps. . But it should work for Upserts(when record matched then update , if not then Insert) for larger datasets more than 5TB . None of the solutions are working out there in the Internet so far more than a month.Could anyone let me know the valid steps for larger datasets with JSON. ... View more

I'm currently trying to remove the blank line after discarding the lines in Nifi Removing those lines in Nifi {"eventType":"delete","ServerSerial":"0","ServerName":"","deletedat":"2018-08-24 17:56:34.944"} Since the serial number is 0 and empty servername . After removing there will be a blank space created in the final output file I want to replace the blanks space "eventType":"delete","ServerSerial":"1556562030","ServerName":"XYZ_U_O","deletedat":"2018-08-24 17:56:39.974"} {"eventType":"delete","ServerSerial":"0","ServerName":"","deletedat":"2018-08-24 17:56:34.944"} {"eventType":"delete","ServerSerial":"0","ServerName":"","deletedat":"2018-08-24 17:56:34.944"} So I'm currently using ReplaceText to search for the value : {"eventType":"delete","ServerSerial":"0","ServerName":""[^}]+},? and replacing with empty spaces . It failed to remove the blank lines Replaced with ^[ \t]*$\r?\n --> Failed to remove . Replaced with \r\n --> Failed to remove . Replace value is not considered the regex value . What shall I do . How can i replace blank lines after removing the lines based on the Search value. ... View more

We are receiving Hourly JSON data into HDFS. The size of the data would be 7GB per hour . when matched record found on the final table then Update (or) Delete if the record not matched in the final dataset then insert the record. What is best way to do Upserts(Updates and Inserts in Hadoop) for large dataset. Hive or HBase or Nifi . What is flow . Can anyone help us in the flow .updates.txt ... View more

Hiveserver 2 log4j settings for daily rotation and delete more than 10 days logs . Can anyone share the config file to do that. I have already configured that ( attached ) but it is not working as expected . So I would like to know where exactly we need to configure it (Advanced-hive-execlog4j,Advanced hive-log4j,Advanced hive-log4j2,Advanced beeline-log4j2 ) we are usinglog.txt beeline hs2.HA enabled HS2.d ... View more

Hi, I have 3 columns , Userid , Serial no and Timestamp column(YYYY-DD-MM HH:MM:SS) in DB2 . I would like to create a incremental load using Nifi . userid,serialno are random numbers not increased primary numbers . TS is increamental. I have tried to use FLOW 1 QueryFetchTable->ExecuteSQL->PutHIveStreaming --> logattribute (to avoid duplicates)-- Not working , No Error in the logs (refer attached pic for configuration ) nf.pngnf1.pngnf2.png I'm not able to load even single data to hive using this method as I have used max_column is TS . -- Hive table is Partitioned , Bucketed , ORC and transantional FLOW 2 ExecuteSQL->PutHiveStreaming->logattribute --> I'm able to manage the data into hive , but I will not be able to incremental load the data -- Hive table is Partitioned , Bucketed , ORC and transantional Could you please help me to setup a Simple Incremental load using any flow files . Ultimately , I would like a incremental load without duplicates Hive table is Partitioned , Bucketed , ORC and transnational I'm open to any processors in Nifi for an incremental load without duplicates . Can you please create a sample workflow with sample config and sample data @Shu , Any input please . Thanks a lot in advance . Expecting a awesome answer from you 🙂 ... View more

Hi @Shu I have seen your article below and tried everything it was suggested there . You have helped me alot and thanks for that . https://community.hortonworks.com/questions/177112/nifi-puthivestreaming-processor-error.html I'm stuck with the error Refer Using QueryDatabasetable PutHiveStreaming Tables are created with same column names as like Source table , Partitioned , bucketed , and orc. Also could you please help us understand how the flow read the data and how it should be given . we are trying to connect from Db2 to Hive . Do we really need to convert inbetween from db2 to avro or anything of that sort ... View more

Kindly do the step by step to do in HDFS , NiFI and Hive and ZK I need daily log rotation and delete backup more than a week . ... View more