About raaj_muthu

raaj_muthu · ‎07-12-2018

@Matt Burgess . I still cannot eliminate duplicates . I have changed Avro logical types to false and it allowed me to insert data into Hive with DUPLICATES . Lets assume . I have 100 records If I insert 1 record and I should be getting 101 records instead I get 201 records.Anything I'm missing right now .Kindly assist

raaj_muthu · ‎07-12-2018

Outputof the ExecuteSQL flow : "TIMESTAMP" : "2018-06-21T19:07:27.000Z" and Hive Datatype is Timestamp and not accepting the format . Is it because Maximum-value Columns aren't working..

raaj_muthu · ‎07-12-2018

@Matt Burgess Im using Maximum-value Columns as timestamp (YYYY-MM-DD hh:mm:ss) . Do you think Nifi Is unable to take the Maximum-value as a timestamp in the format as it is(YYYY-MM-DD hh:mm:ss)

raaj_muthu · ‎07-12-2018

@Matt Burgess Is that QueryDatabasetable or GeneratetableFetch. Let's say I have 100 records in my source table(DB2) . When I add one record in my source I should be getting 101 records ,instead i get around 201 records . I get duplicated records and keep on adding duplicates for every new record @Shu or @Matt Burgess Kindly assist . Im missing a small thing . Kindly clarify.nf11.png Refer attached for the flow .

raaj_muthu · ‎07-12-2018

There was no error . But I can't see data in hive . Are my configuration for flow 1 is right ? Without Queryfetchtable I'm able to put data in hive . But can't incremental load to Hadoop . I'm using NiFi 1.2 version.

raaj_muthu · ‎07-11-2018

Hi, I have 3 columns , Userid , Serial no and Timestamp column(YYYY-DD-MM HH:MM:SS) in DB2 . I would like to create a incremental load using Nifi . userid,serialno are random numbers not increased primary numbers . TS is increamental. I have tried to use FLOW 1 QueryFetchTable->ExecuteSQL->PutHIveStreaming --> logattribute (to avoid duplicates)-- Not working , No Error in the logs (refer attached pic for configuration ) nf.pngnf1.pngnf2.png I'm not able to load even single data to hive using this method as I have used max_column is TS . -- Hive table is Partitioned , Bucketed , ORC and transantional FLOW 2 ExecuteSQL->PutHiveStreaming->logattribute --> I'm able to manage the data into hive , but I will not be able to incremental load the data -- Hive table is Partitioned , Bucketed , ORC and transantional Could you please help me to setup a Simple Incremental load using any flow files . Ultimately , I would like a incremental load without duplicates Hive table is Partitioned , Bucketed , ORC and transnational I'm open to any processors in Nifi for an incremental load without duplicates . Can you please create a sample workflow with sample config and sample data @Shu , Any input please . Thanks a lot in advance . Expecting a awesome answer from you 🙂

raaj_muthu · ‎06-26-2018

I have 8 node cluster . NIFI is installed in only one cluster . i want to use a NiFi processor to trigger shell/python script on a remote machine . Example : Machine 1 - Nifi Installed in Machine 1 Machine 4 : Script has to be executed and files supporting the script is available in Machine 4(Script cannot be moved to Nifi node) Please tell me 1.what processors should I use and how the flow should be . . 2.How to trigger Shell Script using Nifi on a remote machine . 3.How to Log the flow if possible in case of any error / failures and trigger a mail ( optional) i need this scenario for many use cases. I have googled a lot , Execute script processor is having only ( Python,Ruby,grovvy etc..) not shell script in the list of options. How to provide SSH username and key/password. @Shu

raaj_muthu · ‎06-09-2018

Fantastic and detailed reply. I would try this out and reply if that works .Thanks a lot @Shu

raaj_muthu · ‎06-08-2018

Hi @Shu I was able to establish a connection between mysql and Hive . However the purpose is not solved yet . I was trying to update and insert values into Hive . For Example ( refer attached) . In the example : . On Initial load Hive has loaded the values of ID 1,2 and 3 . On the second load , We have to remove the values 3 and update the value 1 and insert the value 4 . How that is possible in Nifi . Is that at possible ? Could please guide me on this If you can. thanks ... ex.jpg

raaj_muthu · ‎06-07-2018

Hi, I'm unable to establish connection between Mysql and Hive due to DBCP connection pool .This is my configuration settings in DBCP connection pool ( please refer attached) . I'm getting error says (refer attached ) dbcp1.jpg dbcp2.jpg dbcp.png Query database table -> put hive streaming processors is used . and Query Database table is scheduled to run every three minutes . I have restarted Nifi by disabling the DBCP connection pool and tried . Still the same and failed . Also Is it possible to insert updated values + New values in Mysql to Hive ( which processor should I use) @Matt Clarke @Matt Burgess . Could you please have a look here please . what am I doing wrong . Thanks All

Online	Offline
Last Visited	‎09-24-2018 01:21 PM

Date Registered	‎10-20-2017 12:45 PM
Last Visited	‎09-24-2018 01:21 PM
Total Messages Posted	63

Re: Ingest data incrementally using Nifi from DB2

Re: Ingest data incrementally using Nifi from DB2

Re: Ingest data incrementally using Nifi from DB2

Re: Ingest data incrementally using Nifi from DB2

Re: Ingest data incrementally using Nifi from DB2

Ingest data incrementally using Nifi from DB2

Executing Shell script to a remote machine in NiFi...

Re: DBCP connection pool Issue(Can't load Database...

Re: DBCP connection pool Issue(Can't load Database...

DBCP connection pool Issue(Can't load Database Dri...