Member since
11-14-2019
51
Posts
6
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1178 | 05-26-2023 08:40 AM | |
1297 | 05-22-2023 02:38 AM | |
1198 | 05-15-2023 11:25 AM |
03-21-2023
02:46 AM
Hi Team, I am desperately looking for a helping hand here. My flow will infuse/replicate data from different small databases into a single big database. I am using changecapturemysql processor to capture changes in small databases and putdatabase record to put into destination big database. The table structure is same for all databases. but in order to differentiate in the big database there will be an additional column. Example: (let me take only a simple and single table in all databases) in sources, city_name is the PK. in destination, city_name and db_name both will form the PK. DB1.city table city_name New york Mumbai DB2.city table city_name Paris New york Destination DB city table city_name db_name New york. DB1 Mumbai DB1 Paris DB2 New york. DB2 I am able to capture the Inserts and Updates and put them in destination database. but when I delete a record in source, nothing is happening in the destination database. There is no error also. How can I achieve that when i delete a record in any source DB, it needs to delete that particular record in destination DB. How to make nifi identify the record properly to delete. I also wonder how Nifi able to identify the record that need to be updated but not the record that need to be deleted. Please help.
... View more
Labels:
- Labels:
-
Apache NiFi
02-26-2023
11:30 AM
@mburgess Can you please help here..
... View more
02-24-2023
09:39 AM
Hi All, I am currently building a pipeline that fetches all changes from 2 mysql DBs into a final mysql warehouse. I am using changecapturemysql processor to identify inserts,updates and deletes and then sending them to putdatabaserecord processor. All inserts work good. My source has a PK (primary key) column. My destination has the same column + another column as PK . I am able to replicate inserts. But when it comes to updates and deletes, updates on non primary keys were getting captured, but if i update primary key column in source, it is unchanged in destination. same with deletes. a delete with where clause on PK, doesnot deleting the record in destination. As i read the documentation this is expected behaviour - "MPORTANT: If the Statement Type is UPDATE, then the incoming records must not alter the value(s) of the primary keys (or user-specified Update Keys). If such records are encountered, the UPDATE statement issued to the database may do nothing (if no existing records with the new primary key values are found), or could inadvertently corrupt the existing data (by changing records for which the new values of the primary keys exist)." as per nifi official documentation. Now my question is how can i acheive end-end replication into the destination DB table by using changepaturemysql processor ? is there any alternative that i can get the updates and deletes happen even though they are based on primary keys...? does putsql here helps here ? Please guide me.
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
10-25-2022
05:31 AM
Hello Nifi'ers, Good Day.. I am trying to create parameter contexts from CLI toolkit. I have some 100 parameters (non sensitive) and 100 DB passwords (sensitive) I created a json file and they got successfully imported into Nifi. I stored this file in my Nifi Instance as plain json. But I am thinking is there any way that I can keep a passcode or encrypt this json file and send to the CLI command : "nifi import-param-context -i '/abc/def/xyz/nifi-input-files/parameter_contexts.json'" because I feel insecure storing DB passwords in a json file and placing it in a server. Is my thinking correct or is there any better way to input 100's of DB passwords to Nifi. obviously I dont want to type them in UI.. and i dont know when I need to change the passwords I need to change at different job levels in UI. So i feel its not a correct way. Please feel free to correct me and Guide me. Thanks a lot!! Regards Santhosh.N
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
10-13-2022
08:05 AM
Update: So i copied the details of nifi.security.keystore nifi.security.truststore nifi.security.truststorePasswd nifi.security.truststoreType from nifi.properties file of my nifi installation and pasted them in the properties file of toolkit nifi-toolkit-1.16.3/conf/cli.properties Now when i try to run the cli commands i get a new error.. seems I am close but not yet done... below is the error: "ERROR: Error executing command 'list-param-contexts' : Error retrieving parameter contexts: Client certificate for (CN=localhost) is expired." How can I renew this expired certificate ? I wonder why CLI is not so straight forward!!! Please help me in this regards..
... View more
10-12-2022
06:04 AM
I am using CLI version 1.16.3 So, I was thinking in this direction that this CLI supports only HTTPS connections. and I must use truststore and without it there is no way.. (But I don't want to complicate things by using truststore etc. as the remote server can only be accessed by secure ssh.) Please let me know if i am thinking in right direction...
... View more
10-12-2022
03:23 AM
Hello Supporters, I am trying to use NIFI CLI toolkit so that I can load 100 parameters using parameter context. (doing it using UI is a tedious job and not appropriate for deployments as well) I installed Nifi toolkit on remote server. I generally ssh into this server from local and run my commands. To see the UI, I use ssh tunnel. There is only one user for this Nifi Instance (admin) and it don't have https certificates. (Single User Authentication) When i start CLI prompt it works fine (below) # sh cli.sh CLI v1.16.3 Type 'help' to see a list of available commands, use tab to auto-complete. Session loaded from /root/.nifi-cli.config #> But when I use any Nifi commands, it throws below error: #> nifi list-param-contexts -u http://localhost:8443/ ERROR: Error executing command 'list-param-contexts' : Unexpected end of file from server if I use https , #> nifi list-param-contexts -u https://localhost:8443/ ERROR: Error executing command 'list-param-contexts' : truststore, truststoreType, and truststorePasswd are required when using an https url But I don't use truststore or passwords for single admin user. Also commands related to Registry working fine. #> registry current-user -u http://localhost:18080 anonymous #> registry list-buckets -u http://localhost:18080 # Name Id Description - ------------ ------------------------------------ ----------- 1 First-bucket 8a3da253-f635-4b01-941f-bfb6437cead7 2 delete 4df052fd-fddc-4e7b-b4ea-bb3b8b691385 Is this something related to https/URI ? I don't find much information online for this issue and I was struck from many days.. Please help me solve this. @TimothySpann @araujo Sincere request to help me find the problem..
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
10-07-2022
09:45 AM
Hi All, Good Day. I am trying to work with NiFi cli toolkit. Installation is successful, and I am able to enter the CLI prompt. But When I use CLI commands it throws errors like below: #> nifi list-param-contexts ERROR: Error executing command 'list-param-contexts' : Unexpected end of file from server My cli.properties looks like below: baseUrl=http://localhost:8443/ keystore= keystoreType= keystorePasswd= keyPasswd= truststore= truststoreType= truststorePasswd= proxiedEntity= even i explicitly mention the baseUrl the error is same: #> nifi list-param-contexts -baseUrl http://localhost:8443/ ERROR: Error executing command 'list-param-contexts' : Unexpected end of file from server My Nifi instance don't have certificate authorization (I actually don't need it, because I do ssh into the isntance and don't login from outside world) and I have only admin user. I had Nifi Registry as well and it working good. but i want to interact with Nifi instance not with registry. #> registry list-buckets -baseUrl http://localhost:18080 # Name Id Description - ------------ ------------------------------------ ----------- 1 First-bucket 8a3da253-f635-4b01-941f-bfb6437cead7 2 delete 4df052fd-fddc-4e7b-b4ea-bb3b8b691385 complete error when using verbose: #> nifi list-param-contexts --verbose ERROR: Error executing command 'list-param-contexts' : Unexpected end of file from server org.apache.nifi.toolkit.cli.api.CommandException: Error executing command 'list-param-contexts' : Unexpected end of file from server at org.apache.nifi.toolkit.cli.impl.command.nifi.AbstractNiFiCommand.doExecute(AbstractNiFiCommand.java:65) at org.apache.nifi.toolkit.cli.impl.command.AbstractPropertyCommand.execute(AbstractPropertyCommand.java:74) at org.apache.nifi.toolkit.cli.impl.command.CommandProcessor.processCommand(CommandProcessor.java:252) at org.apache.nifi.toolkit.cli.impl.command.CommandProcessor.processGroupCommand(CommandProcessor.java:233) at org.apache.nifi.toolkit.cli.impl.command.CommandProcessor.process(CommandProcessor.java:188) at org.apache.nifi.toolkit.cli.CLIMain.runInteractiveCLI(CLIMain.java:124) at org.apache.nifi.toolkit.cli.CLIMain.main(CLIMain.java:68) Caused by: java.net.SocketException: Unexpected end of file from server at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:866) at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689) at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:863) at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1615) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520) at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:378) at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:267) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:675) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:674) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:318) at org.apache.nifi.toolkit.cli.impl.client.nifi.impl.JerseyParamContextClient.lambda$getParamContexts$0(JerseyParamContextClient.java:51) at org.apache.nifi.toolkit.cli.impl.client.nifi.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:76) at org.apache.nifi.toolkit.cli.impl.client.nifi.impl.JerseyParamContextClient.getParamContexts(JerseyParamContextClient.java:49) at org.apache.nifi.toolkit.cli.impl.command.nifi.params.ListParamContexts.doExecute(ListParamContexts.java:46) at org.apache.nifi.toolkit.cli.impl.command.nifi.params.ListParamContexts.doExecute(ListParamContexts.java:31) at org.apache.nifi.toolkit.cli.impl.command.nifi.AbstractNiFiCommand.doExecute(AbstractNiFiCommand.java:63) ... 6 more I am not able to get any clue where going wrong.. Can someone please help me fix it. Thanks in advance.
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
09-07-2022
04:22 AM
Thank you @araujo for helping me out, So the parameters are also created and updated manually in the UI and referred in the processors. If there is a change in password needed in production, we will update it manually in UI ? or do we have any alternative way than doing it through UI.. I mean any kind of deployment support..?
... View more
09-07-2022
03:11 AM
Hi Everyone, Good day... I hope someone can educate me on my basic understandings.. After some basic research on Nifi on test servers, I want to try it to production, Now I get this basic question on how do we provide the passwords of databases in processors.. because we provide them in the GUI in general, but is it the same way we do for production servers ? taking an example of database DBCPConnectionPool, we provide the connection details like username, password etc in GUI as attached in the image, but I want to understand is there a different way to provide these details in production or it's also with GUI ? I am thinking about change management and security of passwords etc. Please help me understand..
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
- « Previous
- Next »