About raaj_muthu

raaj_muthu · ‎09-18-2018

@Shu Is it possible to add two RowIdentifier/RowIdentifier Field name in PutHbaseJson , Like ServerName,ServerNo or ${ServerName},${ServerNo} .

raaj_muthu · ‎09-18-2018

Hello everybody, I'm using Routetext processor to separate JSON records to different flow.Ultimately I need Flow1 should be inserted into HBASE table first , Flow 2 should be executed second and so on.How I can achieve this. Basically Inserts should insert data to HBase first,Updates should happen next and Delete should happen last . Attached the Sample test.xml

raaj_muthu · ‎08-30-2018

Accepeted the answer for his answer . But didn't work out well.:(

raaj_muthu · ‎08-30-2018

I'm currently trying to remove the blank line after discarding the lines in Nifi Removing those lines in Nifi {"eventType":"delete","ServerSerial":"0","ServerName":"","deletedat":"2018-08-24 17:56:34.944"} Since the serial number is 0 and empty servername . After removing there will be a blank space created in the final output file I want to replace the blanks space "eventType":"delete","ServerSerial":"1556562030","ServerName":"XYZ_U_O","deletedat":"2018-08-24 17:56:39.974"} {"eventType":"delete","ServerSerial":"0","ServerName":"","deletedat":"2018-08-24 17:56:34.944"} {"eventType":"delete","ServerSerial":"0","ServerName":"","deletedat":"2018-08-24 17:56:34.944"} So I'm currently using ReplaceText to search for the value : {"eventType":"delete","ServerSerial":"0","ServerName":""[^}]+},? and replacing with empty spaces . It failed to remove the blank lines Replaced with ^[ \t]*$\r?\n --> Failed to remove . Replaced with \r\n --> Failed to remove . Replace value is not considered the regex value . What shall I do . How can i replace blank lines after removing the lines based on the Search value.

raaj_muthu · ‎08-23-2018

Hi @Shu I did try your steps , i have few queries Answer 1: I cannot insert the partition key if the source and target tables are partitioned Updating partition not supported --> Error: Error while compiling statement: FAILED: SemanticException [Error 10292]: Updating values of partition columns is not supported (state=42000,code=10292) Answer2: 1.Raw data is in JSON and landing to HDFS 2.For Merging we are converting the data into ORC,transaction, bucketed 3. With ORC, How I can use Input_file_name . It is not possible to merge with JSON raw data files right . Also , Could you please highlight on point 3&4 .

raaj_muthu · ‎08-21-2018

Hi @Shu I followed your tutorial here: https://community.hortonworks.com/articles/191760/create-dynamic-partitions-based-on-flowfile-conten.html I'm stuck with the error attached . Also , Is it possible to do update+insert in the flow as mentioned the URL above using PutHiveQL. Am able to execute Merge option in Hiveserver2. However due to large amount of data Merge is not working for huge dataset I'm checking using the URL above , can I achieve the below use case nifi-predicateerror.jpg when matched record found on the final table then define which action you need to take either Update (or) Delete if the record not matched in the final dataset then insert the record.

raaj_muthu · ‎06-26-2018

I have 8 node cluster . NIFI is installed in only one cluster . i want to use a NiFi processor to trigger shell/python script on a remote machine . Example : Machine 1 - Nifi Installed in Machine 1 Machine 4 : Script has to be executed and files supporting the script is available in Machine 4(Script cannot be moved to Nifi node) Please tell me 1.what processors should I use and how the flow should be . . 2.How to trigger Shell Script using Nifi on a remote machine . 3.How to Log the flow if possible in case of any error / failures and trigger a mail ( optional) i need this scenario for many use cases. I have googled a lot , Execute script processor is having only ( Python,Ruby,grovvy etc..) not shell script in the list of options. How to provide SSH username and key/password. @Shu

raaj_muthu · ‎06-09-2018

Fantastic and detailed reply. I would try this out and reply if that works .Thanks a lot @Shu

raaj_muthu · ‎06-08-2018

Hi @Shu I was able to establish a connection between mysql and Hive . However the purpose is not solved yet . I was trying to update and insert values into Hive . For Example ( refer attached) . In the example : . On Initial load Hive has loaded the values of ID 1,2 and 3 . On the second load , We have to remove the values 3 and update the value 1 and insert the value 4 . How that is possible in Nifi . Is that at possible ? Could please guide me on this If you can. thanks ... ex.jpg

raaj_muthu · ‎06-07-2018

Hi, I'm unable to establish connection between Mysql and Hive due to DBCP connection pool .This is my configuration settings in DBCP connection pool ( please refer attached) . I'm getting error says (refer attached ) dbcp1.jpg dbcp2.jpg dbcp.png Query database table -> put hive streaming processors is used . and Query Database table is scheduled to run every three minutes . I have restarted Nifi by disabling the DBCP connection pool and tried . Still the same and failed . Also Is it possible to insert updated values + New values in Mysql to Hive ( which processor should I use) @Matt Clarke @Matt Burgess . Could you please have a look here please . what am I doing wrong . Thanks All

Online	Offline
Last Visited	‎09-24-2018 01:21 PM

Member Since	‎10-20-2017 12:45 PM
Last Visited	‎09-24-2018 01:21 PM
Posts	63

Cloudera Community

Re: Nifi to insert the flow one after another

Nifi to insert the flow one after another

Re: Upserts using Nifi ( updates + Inserts)

How to remove the blank lines in Nifi

Re: Upserts using Nifi ( updates + Inserts)

Upserts using Nifi ( updates + Inserts)

Executing Shell script to a remote machine in NiFi...

Re: DBCP connection pool Issue(Can't load Database...

Re: DBCP connection pool Issue(Can't load Database...

DBCP connection pool Issue(Can't load Database Dri...