Hi Team,   I am desperately looking for a helping hand here. My flow will infuse/replicate data from different small databases into a single big database. I am using changecapturemysql processor to capture changes in small databases and putdatabase record to put into destination big database. The table structure is same for all databases. but in order to differentiate in the big database there will be an additional column.  Example: (let me take only a simple and single table in all databases) in sources, city_name is the PK. in destination,  city_name and db_name  both will form the PK.   DB1.city table city_name New york Mumbai   DB2.city table city_name Paris New york     Destination DB city table city_name    db_name New york.     DB1 Mumbai        DB1 Paris             DB2 New york.     DB2   I am able to capture the Inserts and Updates and put them in destination database. but when I delete a record in source, nothing is happening in the destination database. There is no error also. How can I achieve that when i delete a record in any source DB, it needs to delete that particular record in destination DB. How to make nifi identify the record properly to delete. I also wonder how Nifi able to identify the record that need to be updated but not the record that need to be deleted. Please help. ... View more

Hi All,   I am currently building a pipeline that fetches all changes from 2 mysql DBs into a final mysql warehouse.   I am using changecapturemysql processor to identify inserts,updates and deletes and then sending them to putdatabaserecord processor. All inserts work good. My source has a PK (primary key) column. My destination has the same column + another column as PK . I am able to replicate inserts. But when it comes to updates and deletes,  updates on non primary keys were getting captured, but if i update primary key column in source, it is unchanged in destination. same with deletes. a delete with where clause on PK, doesnot deleting the record in destination. As i read the documentation this is expected behaviour - "MPORTANT: If the Statement Type is UPDATE, then the incoming records must not alter the value(s) of the primary keys (or user-specified Update Keys). If such records are encountered, the UPDATE statement issued to the database may do nothing (if no existing records with the new primary key values are found), or could inadvertently corrupt the existing data (by changing records for which the new values of the primary keys exist)." as per nifi official documentation. Now my question is how can i acheive end-end replication into the destination DB table by using changepaturemysql processor ? is there any alternative that i can get the updates and deletes happen even though they are based on primary keys...? does putsql here helps here ? Please guide me. ... View more

Hello Nifi'ers, Good Day..   I am trying to create parameter contexts from CLI toolkit.  I have some 100 parameters (non sensitive) and 100 DB passwords (sensitive) I created a json file and they got successfully imported into Nifi.   I stored this file in my Nifi Instance as plain json. But I am thinking is there any way that I can keep a passcode or encrypt this json file and send to the CLI command :   "nifi import-param-context -i '/abc/def/xyz/nifi-input-files/parameter_contexts.json'"   because I feel insecure storing DB passwords in a json file and placing it in a server.   Is my thinking correct or is there any better way to input 100's of  DB passwords to Nifi. obviously I dont want to type them in UI.. and i dont know when I need to change the passwords I need to change at different job levels in UI. So i feel its not a correct way. Please feel free to correct me and Guide me. Thanks a lot!!     Regards Santhosh.N   ... View more

Hello Supporters,   I am trying to use NIFI CLI toolkit so that I can load 100 parameters using parameter context. (doing it using UI is a tedious job and not appropriate for deployments as well)  I installed Nifi toolkit on remote server. I generally ssh into this server from local and run my commands. To see the UI, I use ssh tunnel.   There is only one user for this Nifi Instance (admin) and it don't have https certificates. (Single User Authentication)   When i start CLI prompt it works fine (below)   # sh cli.sh CLI v1.16.3 Type 'help' to see a list of available commands, use tab to auto-complete. Session loaded from /root/.nifi-cli.config #>     But when I use any Nifi commands, it throws below error:   #> nifi list-param-contexts -u http://localhost:8443/ ERROR: Error executing command 'list-param-contexts' : Unexpected end of file from server   if I use https ,  #> nifi list-param-contexts -u https://localhost:8443/ ERROR: Error executing command 'list-param-contexts' : truststore, truststoreType, and truststorePasswd are required when using an https url   But I don't use truststore or passwords for single admin user.       Also commands related to Registry working fine.   #> registry current-user -u http://localhost:18080 anonymous #> registry list-buckets -u http://localhost:18080 # Name Id Description - ------------ ------------------------------------ ----------- 1 First-bucket 8a3da253-f635-4b01-941f-bfb6437cead7 2 delete 4df052fd-fddc-4e7b-b4ea-bb3b8b691385     Is this something related to https/URI  ? I don't find much information online for this issue and I was struck from many days.. Please help me solve this.     @TimothySpann  @araujo Sincere request to help me find the problem.. ... View more

Hi All, Good Day.   I am trying to work with NiFi cli toolkit. Installation is successful, and I am able to enter the CLI prompt. But When I use CLI commands it throws errors like below: #> nifi list-param-contexts ERROR: Error executing command 'list-param-contexts' : Unexpected end of file from server     My cli.properties looks like below:   baseUrl=http://localhost:8443/ keystore= keystoreType= keystorePasswd= keyPasswd= truststore= truststoreType= truststorePasswd= proxiedEntity=   even i explicitly mention the baseUrl the error is same:   #> nifi list-param-contexts -baseUrl http://localhost:8443/ ERROR: Error executing command 'list-param-contexts' : Unexpected end of file from server     My Nifi instance don't have certificate authorization (I actually don't need it, because I do ssh into the isntance and don't  login from outside world)  and I have only admin user.   I had Nifi Registry as well and it working good. but i want to interact with Nifi instance not with registry. #> registry list-buckets -baseUrl http://localhost:18080 # Name Id Description - ------------ ------------------------------------ ----------- 1 First-bucket 8a3da253-f635-4b01-941f-bfb6437cead7 2 delete 4df052fd-fddc-4e7b-b4ea-bb3b8b691385     complete error when using verbose:   #> nifi list-param-contexts --verbose ERROR: Error executing command 'list-param-contexts' : Unexpected end of file from server org.apache.nifi.toolkit.cli.api.CommandException: Error executing command 'list-param-contexts' : Unexpected end of file from server at org.apache.nifi.toolkit.cli.impl.command.nifi.AbstractNiFiCommand.doExecute(AbstractNiFiCommand.java:65) at org.apache.nifi.toolkit.cli.impl.command.AbstractPropertyCommand.execute(AbstractPropertyCommand.java:74) at org.apache.nifi.toolkit.cli.impl.command.CommandProcessor.processCommand(CommandProcessor.java:252) at org.apache.nifi.toolkit.cli.impl.command.CommandProcessor.processGroupCommand(CommandProcessor.java:233) at org.apache.nifi.toolkit.cli.impl.command.CommandProcessor.process(CommandProcessor.java:188) at org.apache.nifi.toolkit.cli.CLIMain.runInteractiveCLI(CLIMain.java:124) at org.apache.nifi.toolkit.cli.CLIMain.main(CLIMain.java:68) Caused by: java.net.SocketException: Unexpected end of file from server at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:866) at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689) at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:863) at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1615) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520) at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:378) at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:267) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:318) at org.apache.nifi.toolkit.cli.impl.client.nifi.impl.JerseyParamContextClient.lambda$getParamContexts$0(JerseyParamContextClient.java:51) at org.apache.nifi.toolkit.cli.impl.client.nifi.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:76) at org.apache.nifi.toolkit.cli.impl.client.nifi.impl.JerseyParamContextClient.getParamContexts(JerseyParamContextClient.java:49) at org.apache.nifi.toolkit.cli.impl.command.nifi.params.ListParamContexts.doExecute(ListParamContexts.java:46) at org.apache.nifi.toolkit.cli.impl.command.nifi.params.ListParamContexts.doExecute(ListParamContexts.java:31) at org.apache.nifi.toolkit.cli.impl.command.nifi.AbstractNiFiCommand.doExecute(AbstractNiFiCommand.java:63) ... 6 more     I am not able to get any clue where going wrong.. Can someone please help me fix it.  Thanks in advance. ... View more

Hi Everyone, Good day... I hope someone can educate me on my basic understandings.. After some basic research on Nifi on test servers, I want to try it to production, Now I get this basic question on how do we provide the passwords of databases in processors.. because we provide them in the GUI in general, but is it the same way we do for production servers ? taking an example of database DBCPConnectionPool, we provide the connection details like username, password etc in GUI as attached in the image, but I want to understand is there a different way to provide these details in production or it's also with GUI ?  I am thinking about change management and security of passwords etc. Please help me understand..   ... View more