Member since
09-14-2023
11
Posts
0
Kudos Received
0
Solutions
10-03-2023
06:14 PM
I am trying to run a merge query in PUTSQL after using ExecuteSQL to pull from source database. I then pass the results through ConvertAvrotoJSON and then to convertJSONToSQL (set to INSERT since there is no UPSERT for SQL server database). I want to now run the merge query to update if the primary key is matched or insert if not, but I have been getting all sorts of errors Input flowfile has the following attributes sql.args.1.type 12 sql.args.1.value A-001 sql.args.2.type 12 sql.args.2.value DDD sql.args.3.type 12 sql.args.3.value 2B sql.args.4.type 12 sql.args.4.value AAA sql.args.5.type 12 sql.args.5.value 2B sql.args.6.type 12 sql.args.6.value BBB sql.args.7.type 93 sql.args.7.value 1954-08-01 22:46:50.73 sql.catalog CUSDB sql.table CUSTOMERS uuid 50a3daf9-e5d8-422b-b6cb-9c92da723c55 I created the following attributes in UpdateAtributes I directed this flowfile to UpdateAttribute where I created above properties like this sql.args.1.type = 12; sql.args.1.value = ${vkey} sql.args.2.type = 12; sql.args.2.value = ${AP} sql.args.3.type = 12; sql.args.3.value = ${BP} ........ sql.args.7.type = 93; sql.args.7.value = ${LAST_MOD_DATE} But when I check the results of the queue of the UpdateAtributes, I see that sql.args.1.type, sql.args.2.type, ... sql.args.7.type return blank values. I am doing something wrong in the way I have defined those properties? Merge query in PutSQL looks like this MERGE INTO CUSDB.dbo.CUSTOMERS as t USING (select ? as vkey, ? as ap, ? as bp, ? as lm, ? as lnum, ? as wn, ? as LAST_MOD_DATE) s ON t.EKEY = s.EKEY WHEN MATCHED THEN UPDATE SET t.ap = s.ap, t.bp = s.bp, t.lm = s.lm, t.lnum = s.lnum, t.wn = s.wn, t.LAST_MOD_DATE = s.LAST_MOD_DATE WHEN NOT MATCHED THEN INSERT (vkey, ap, bp, lm, lnum, wn, LAST_MOD_DATE) VALUES (?, ?, ?, ?, ?, ?, to_date(?, 'YYYY-MM-DD HH34:MI:SS')) I get the error that all parameters are blank. Can someone please point me in the direction to resolve this issue. Why are the attributes going blank when I set them in UpdateAttributes?
... View more
Labels:
- Labels:
-
Apache NiFi
10-03-2023
03:35 PM
@mburgess I am trying to run a merge query in PUTSQL after using ExecuteSQL to pull from source database. I then pass the results through ConvertAvrotoJSON and then to convertJSONToSQL (set to INSERT since there is no UPSERT for SQL server database). I want to now run the merge query to update if the primary key is matched or insert if not, but I have been getting all sorts of errors Input flowfile has the following attributes sql.args.1.type 12 sql.args.1.value A-001 sql.args.2.type 12 sql.args.2.value DDD sql.args.3.type 12 sql.args.3.value 2B sql.args.4.type 12 sql.args.4.value AAA sql.args.5.type 12 sql.args.5.value 2B sql.args.6.type 12 sql.args.6.value BBB sql.args.7.type 93 sql.args.7.value 1954-08-01 22:46:50.73 sql.catalog CUSDB sql.table CUSTOMERS uuid 50a3daf9-e5d8-422b-b6cb-9c92da723c55 I created the following attributes in UpdateAtributes I directed this flowfile to UpdateAttribute where I created above properties like this sql.args.1.type = 12; sql.args.1.value = ${vkey} sql.args.2.type = 12; sql.args.2.value = ${AP} sql.args.3.type = 12; sql.args.3.value = ${BP} ........ sql.args.7.type = 93; sql.args.7.value = ${LAST_MOD_DATE} But when I check the results of the queue of the UpdateAtributes, I see that sql.args.1.type, sql.args.2.type, ... sql.args.7.type return blank values. I am doing something wrong in the way I have defined those properties? Merge query in PutSQL looks like this MERGE INTO RCDB.dbo.RCDB_WELLHEADER_HZ_NIFI_V2 as t USING (select ? as vkey, ? as ap, ? as bp, ? as lm, ? as lnum, ? as wn, ? as LAST_MOD_DATE) s ON t.EKEY = s.EKEY WHEN MATCHED THEN UPDATE SET t.ap = s.ap, t.bp = s.bp, t.lm = s.lm, t.lnum = s.lnum, t.wn = s.wn, t.LAST_MOD_DATE = s.LAST_MOD_DATE WHEN NOT MATCHED THEN INSERT (vkey, ap, bp, lm, lnum, wn, LAST_MOD_DATE) VALUES (?, ?, ?, ?, ?, ?, to_date(?, 'YYYY-MM-DD HH34:MI:SS')) I get the error that all parameters are blank. Can someone please point me in the direction to resolve this issue. Why are the attributes going blank when I set them in UpdateAttributes?
... View more
09-25-2023
12:03 PM
@cotopaul I have tried all the suggestions above and none of them worked. I have also tried to use the ExecuteSQL settings to normalize column and table but it messed up the actual values in those columns. Not sure why presenting column names with spaces (or alphanumeric characters) should be so painful as I have been on this for days now. I'll appreciate any solution out there from anyone who has encountered similar issue
... View more
09-24-2023
07:24 AM
I am building a workflow to extract data from a MS SQL server table using the ExecuteSQL processor with this simple SQL: select [06 QQQ], [11 JJJ], [12 KKK], [13 JJJ], from DB_TEST.dbo.FGRID But I get the error below. routing to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.avro.SchemaParseException: Illegal initial character: 06 QQQ
- Caused by: org.apache.avro.SchemaParseException: Illegal initial character: 06 QQQ Apparently, NiFi can't recognize column names with spaces and other special characters. I have tried using single and double quotes but still get same error. Question is - what can I use to reference column names with special characters. In DBeaver, I am able to query those column names with []. Any assistance will be appreciated. Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
09-17-2023
02:54 AM
@raobelina_toky_ Looking at the resolution of your NiFi issues on LDAPS, I have the following questions around Keystore/truststore 1. I am currently using the same keystore/truststore that was internally generated by our identity management team when I had configured for singleuser mode for my LDAP configuration. Does a new one need to be generated for my initial admin user to be able to connect to NiFi? Right now I can't connect with initial admin user. My knowledge of the way keystore/truststore work is very scanty. 2. Can you share the present settings of your authorizers.xml, login-identity-provider.xml that is working (without passwords, of course). I am able to start NiFi with my settings but just can't connect with initial admin user. I have a feeling that it's keystore/truststore issue but can't seem to resolve it
... View more
09-16-2023
05:53 AM
@Yemre Hello Bro, I am trying to set up LDAP and have been loosing hair on getting the right configuration for over a week now. I am trying to configure for LDAPS in NiFi and already have my keystore and trust generated internally so my issue seems quite similar to yours. Can I see the copy of your nifi.properties, login-identity-providers, authorizers.xml (without any sensitive information, of course) that finally worked? I will really appreciate any assistance for you. Thanks!
... View more
09-15-2023
02:19 PM
Thanks @MattWho The ldap listing without the port number was a typo in my text message here but not in the server I just checked the nifi-user log (after failed log on attempts) as advised and this is the error I got. So apparently, even though Nifi starts, the user set up is not right. Can you suggest what my ldap parameters should be based on sample I provided above? Your help will be appreciated 2023-09-15 15:57:58,201 ERROR [NiFi Web Server-25] o.a.n.w.a.c.AdministrationExceptionMapper org.apache.nifi.admin.service.AdministrationException: Unable to validate the supplied credentials. Please contact the system administrator.. Returning Internal Server Error response. org.apache.nifi.admin.service.AdministrationException: Unable to validate the supplied credentials. Please contact the system administrator..
... View more
09-15-2023
09:40 AM
Yes, user_admin exists in LDAP server but when I try to use it's password from the server to log on, NiFi rejects even though I know it's correct. Should I change this: sAMAccountName to just {0}? Also, how do I use the contents of the user.xml and authorizations.xml that were generated by NiFi? What do those mean? Sorry for my very basic questions. I really appreciate your help
... View more
09-15-2023
05:53 AM
Hello All, I am new to NiFi. I have set up NiFi with LDAP authentication and NiFi starts with no issues but I don't know how to log into NiFi as "Initial Admin Identity" user "user_admin" as shown in my setup below. What should I use as the login credentials as NiFi presently rejects user_admin/<password>? nifi.properties nifi.security.user.authorizer=file-provider nifi.security.allow.anonymous.authentication=false nifi.security.user.login.identity.provider=ldap-provider login-identity-providers.xml <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">START_TLS</property> <property name="Manager DN">uid=test_admin,OU=users,DC=abc,DC=com</property> <property name="Manager Password">*******</property> <property name="TLS - Keystore">./conf/keystore.p12</property> <property name="TLS - Keystore Password">*****</property> <property name="TLS - Keystore Type">PKCS12</property> <property name="TLS - Truststore">./conf/truststore.p12</property> <property name="TLS - Truststore Password">******</property> <property name="TLS - Truststore Type">PKCS12</property> <property name="TLS - Client Auth">REQUIRED</property> <property name="TLS - Protocol">TLS</property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldaps://test.com</property> <property name="User Search Base">DC=abc,DC=net</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Expiration">12 hours</property> ----------------------- authorizer.xml <authorizer> <identifier>file-provider</identifier> <class>org.apache.nifi.authorization.FileAuthorizer</class> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Users File">./conf/users.xml</property> <property name="Initial Admin Identity">uid=user_admin,OU=USERS,DC=abc,DC=com</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1"></property> </authorizer>
... View more
Labels:
- Labels:
-
Apache NiFi
09-15-2023
04:40 AM
@jgeorge I have already set up NiFi to use LDAP but need to log in as admin. The certificate was not generated by me and I can't view the content because it is protected How can I configure my admin user to log in as NiFi administrator? Your assisted will be appreciated as I am new to NiFi.
... View more