Member since
09-25-2021
3
Posts
0
Kudos Received
0
Solutions
09-09-2024
02:36 PM
I think you issue may be with using the SingleUserAuthorizer and Single user login provider. These out of the box providers were built so that NiFi could be HTTPS enabled securely out of the box. They are not designed to support clustering, nor are they suitable for production NiFi. You'll want to configure your NiFi cluster to use a production ready authorizer (managed authorizer) and user authentication method other then single user so you can have granular access controls per user/team. Most common is the ldap-provider. The documentation provides examples for authorizer.xml setup: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-based-ldap-authentication https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-based-kerberos-authentication https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap-based-usersgroups-referencing-user-dn ETC.... You cluster is most likely not forming completely due to node to node authentication and authorization issue resulting from using the single user authorizer. In a NiFi cluster the node identities (derived from clientAuth certificates in the Mutual TLS exchange) need to be authorized against some NiFi policies like "Proxy user requests". Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-28-2021
04:37 AM
Hi @asish , I have uploaded the Nifi flow definition, so you can reproduce my issue.(In txt file because I can't upload json extension, just change the extension) It generates a csv: ``` allstring,mixedtype "foo","bar" "foo2",2 "foo3","bar2" ``` Then convert it to Parquet (V1). The issue comes from UpdateHive processor : Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.UnsupportedOperationException: Unknown field type: uniontype<int,string> The issue is not present in Nifi when I convert to another format (Avro for example). But is present when I tried to read back the data from the table (from Hue): AnalysisException: Could not load table open_data.test from catalog CAUSED BY: TableLoadingException: Could not load table open_data.test from catalog CAUSED BY: TException: TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL, error_msgs:[TableLoadingException: Unsupported type 'uniontype' in column 'mixedtype' of table 'test']), lookup_status:OK) Currently, I am on the last version of Nifi CFM in CDP: 7.2.11 - Flow Management Light Duty with Apache NiFi, Apache NiFi Registry Thank you for your help, Best regards,
... View more