Hello everyone, I'm trying to configure Log4j for HDFS Namenode and Security Audit logs. In either way Size based rotation or Daily Rotation I should be able to rotate and delete the logs. However It is not working as expected. Can someone send me the best configurations in Log4j for HDFS NN/Audit Logs.Attaching the Log4j configurations . log4j.txt Help me to delete the logs once it reached 10 Days .I would like to Gzip the older files. Please find the below screenshot for the size consumed on the disk We would like to remove Logs based on Size for Audit logs . ... View more

Hello, We are trying to handle Inserts/Updates/Deletes in Nifi . Currently attached the flow which is taking longer time to process the records and inserts into Hbase . I need to use Compositeprimarykey(ServerSerialno,ServerName) .Based on the CompositeKey will do the Inserts/Updates/Deletes.Need to boost the performance by altering the flow/configs.Kindly assist me to use the right flow to achieve this. Attached the flow . ... View more

Hi I need to load the JSON data into HBASE on the below conditions . If the record(composite primary key)(servername+serverid) is matched then update/delete the record , If not matched Insert the record. How Can I do this in Nifi with Hbase for larger dataset. forecasted data size would be 50-100TB approx for the next one year. optional if at possible , I need to discard the invalid entries from JSON on the flow using regular exp . Where I need to discard the entries and those discarded values creates empty space . Kindly let me know how to remove those empty space after removing the blank values. If possible How to avoid JSON breaking into bad values . I couldn't find good tutorial to Insert/Update in HBASE. Attached sample data test.txt ... View more

I'm currently trying to remove the blank line after discarding the lines in Nifi Removing those lines in Nifi {"eventType":"delete","ServerSerial":"0","ServerName":"","deletedat":"2018-08-24 17:56:34.944"} Since the serial number is 0 and empty servername . After removing there will be a blank space created in the final output file I want to replace the blanks space "eventType":"delete","ServerSerial":"1556562030","ServerName":"XYZ_U_O","deletedat":"2018-08-24 17:56:39.974"} {"eventType":"delete","ServerSerial":"0","ServerName":"","deletedat":"2018-08-24 17:56:34.944"} {"eventType":"delete","ServerSerial":"0","ServerName":"","deletedat":"2018-08-24 17:56:34.944"} So I'm currently using ReplaceText to search for the value : {"eventType":"delete","ServerSerial":"0","ServerName":""[^}]+},? and replacing with empty spaces . It failed to remove the blank lines Replaced with ^[ \t]*$\r?\n --> Failed to remove . Replaced with \r\n --> Failed to remove . Replace value is not considered the regex value . What shall I do . How can i replace blank lines after removing the lines based on the Search value. ... View more

We are receiving Hourly JSON data into HDFS. The size of the data would be 7GB per hour . when matched record found on the final table then Update (or) Delete if the record not matched in the final dataset then insert the record. What is best way to do Upserts(Updates and Inserts in Hadoop) for large dataset. Hive or HBase or Nifi . What is flow . Can anyone help us in the flow .updates.txt ... View more

Hiveserver 2 log4j settings for daily rotation and delete more than 10 days logs . Can anyone share the config file to do that. I have already configured that ( attached ) but it is not working as expected . So I would like to know where exactly we need to configure it (Advanced-hive-execlog4j,Advanced hive-log4j,Advanced hive-log4j2,Advanced beeline-log4j2 ) we are usinglog.txt beeline hs2.HA enabled HS2.d ... View more

Hi, I have 3 columns , Userid , Serial no and Timestamp column(YYYY-DD-MM HH:MM:SS) in DB2 . I would like to create a incremental load using Nifi . userid,serialno are random numbers not increased primary numbers . TS is increamental. I have tried to use FLOW 1 QueryFetchTable->ExecuteSQL->PutHIveStreaming --> logattribute (to avoid duplicates)-- Not working , No Error in the logs (refer attached pic for configuration ) nf.pngnf1.pngnf2.png I'm not able to load even single data to hive using this method as I have used max_column is TS . -- Hive table is Partitioned , Bucketed , ORC and transantional FLOW 2 ExecuteSQL->PutHiveStreaming->logattribute --> I'm able to manage the data into hive , but I will not be able to incremental load the data -- Hive table is Partitioned , Bucketed , ORC and transantional Could you please help me to setup a Simple Incremental load using any flow files . Ultimately , I would like a incremental load without duplicates Hive table is Partitioned , Bucketed , ORC and transnational I'm open to any processors in Nifi for an incremental load without duplicates . Can you please create a sample workflow with sample config and sample data @Shu , Any input please . Thanks a lot in advance . Expecting a awesome answer from you 🙂 ... View more

Hi @Shu I have seen your article below and tried everything it was suggested there . You have helped me alot and thanks for that . https://community.hortonworks.com/questions/177112/nifi-puthivestreaming-processor-error.html I'm stuck with the error Refer Using QueryDatabasetable PutHiveStreaming Tables are created with same column names as like Source table , Partitioned , bucketed , and orc. Also could you please help us understand how the flow read the data and how it should be given . we are trying to connect from Db2 to Hive . Do we really need to convert inbetween from db2 to avro or anything of that sort ... View more

Kindly do the step by step to do in HDFS , NiFI and Hive and ZK I need daily log rotation and delete backup more than a week . ... View more

Hi, Unable to start the Spark Livy Server 2 ExecutionFailed: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://random_host.net:8999/sessions | grep 200 ' returned 1. curl: option --negotiate: the installed libcurl version doesn't support this curl: try 'curl --help' or 'curl --manual' for more information ) I have tried to install libcurl . but it has latest package # yum install libcurl Loaded plugins: fastestmirror, langpacks Package matching libcurl-7.29.0-35.el7.centos.x86_64 already installed. Checking for update. Nothing to do Ambari Says it is started and running .But the error wont go. Refer image attached. The port is opened and running in the background: netstat -plant | grep 8999tcp6 0 0 :::8999 :::* LISTEN 4677/java I have Spark2 livy server in two machines. 1.Machine 1 Spark livy server 2 is running. 2.Machine 2 Spark Livy server 2 is running as per Ambari and terminal . However Ambari says error. I have verified both the machines logs , it shows both error are similar (ie Hadoop folder missing , java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.) n(java.io.FileNotFoundException and File does not exist: /livy2-recovery/v1/batch/st ) the error im getting from 2nd machine not from the 1st machine. I have disabled the alert and enabled the alert . still it doesn't work.I have ambari agent restarted twice . . Still it doesn't work. sp.png Attached log : sp.txt ... View more