About alvinuw

alvinuw · ‎09-01-2017

Hi @Wynner, Thanks for your comments. You are right, it's the authorizers.xml format issue. Actually, I am using nifi-1.4-snapshot. That's why my authorizers.xml is different from the 1.3 one. When I switched to the 1.3 version authorizers.xml with nifi-1.4-snapshot, the above issue was gone.This confuses me. Then, when I have below in authorizers.xml file <propertyname="Node Identity 1">CN=nifi-0, OU= NIFI.COM</property> I saw errors from nifi-app.log, it seems the default OU of node identity is "NIFI". 2017-09-01 14:09:08,854 DEBUG [NiFi Web Server-19] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi. at org.apache.nifi.web.api.AccessResource.createAccessTokenFromTicket(AccessResource.java:349) 2017-09-01 14:09:09,045 INFO [NiFi Web Server-127] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=nifiadmin, OU=NIFI.COM><CN=nifi-0, OU=NIFI>) GET https://nifi-0:9443/nifi-api/flow/current-user (source ip: 10.244.1.95) 2017-09-01 14:09:09,048 WARN [NiFi Web Server-127] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=nifi-0, OU=NIFI 2017-09-01 14:09:09,048 WARN [NiFi Web Server-127] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=nifi-0, OU=NIFI But whenIf I set it as "CN=nifi-0, OU= NIFI" in above file, it works.

alvinuw · ‎08-31-2017

Hi guys, When I secure a three nodes nifi cluster, I got below errors during start up. I use TLS tool to create certificates, and use client-certificate for authentication. The setting in my authorizers.xml is below: <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">file-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">CN=nifiadmin, OU=NIFI.COM</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1">CN=nifi-0, OU=NIFI.COM</property> <property name="Node Identity 2">CN=nifi-1, OU=NIFI.COM</property> <property name="Node Identity 3">CN=nifi-2, OU=NIFI.COM</property> </accessPolicyProvider> From nifi-user.log ouput, it seems NiFi can locate the DN set in authorizers.xml. However, I don't understand why nifi-bootstrap.log says NiFi can't initial admin. Are users.xml and authorizations.xml auto-created and populated by NiFi? After checking, it seems "Initial Admin Identity" user and administrative policies are NOT added to the users.xml and authorizations.xml files. $ cat nifi-user.log 2017-08-31 18:54:28,424 INFO [main] o.a.n.a.FileUserGroupProvider Creating new users file at /opt/nifi/nifi-1.4.0-SNAPSHOT/./conf/users.xml 2017-08-31 18:54:28,453 INFO [main] o.a.n.a.FileUserGroupProvider Users/Groups file loaded at Thu Aug 31 18:54:28 UTC 2017 2017-08-31 18:54:28,458 INFO [main] o.a.n.a.FileAccessPolicyProvider Creating new authorizations file at /opt/nifi/nifi-1.4.0-SNAPSHOT/./conf/authorizations.xml 2017-08-31 18:54:28,475 WARN [main] org.apache.nifi.authorization.FlowParser Flow Configuration does not exist or was empty 2017-08-31 18:54:28,475 INFO [main] o.a.n.a.FileAccessPolicyProvider Populating authorizations for Initial Admin: CN=nifiadmin, OU=NIFI.COM $ cat nifi-bootstrap.log 2017-08-31 18:54:13,151 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 41600 2017-08-31 18:54:29,443 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Error creating bean with name 'niFiWebApiSecurityConfiguration': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire method: public void org.apache.nifi.web.NiFiWebApiSecurityConfiguration.setJwtAuthenticationProvider(org.apache.nifi.web.security.jwt.JwtAuthenticationProvider); nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin CN=nifiadmin, OU=NIFI.COM to seed policies 2017-08-31 18:54:29,444 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down... 2017-08-31 18:54:30,576 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi

alvinuw · ‎08-22-2017

@Wynner Thank you. I am clear on it now.

alvinuw · ‎08-21-2017

Hi @Wynner, Thank you for your clarification. May I understand the Volatile Content Repository only keeps the content for the flowfiles running in the flow? Once the flowfiles are finished, the contents are wipe out from memory, and can't be found in Repository. In contrast, the File-based Content repository will keep the contents for flowfiles even out of the flow, based on the retention setting. Thanks.

alvinuw · ‎08-18-2017

Hi @Wynner, Thank you for your answers. You said that If the configured memory limit is reached, and more content data can't be added to Volatile Content Repository. But how do we free memory for Volatile Content Repository, if it can't auto clean up? Since this day will come sooner or later. It could be an issue to make the Volatile Content Repository useless. Thanks.

alvinuw · ‎08-18-2017

I am curious to know how much risks to use the volatile content repository? My understanding is: If there is a node failure/restart, For data has already been processed/persisted through the flow, no impact on our business or downstreams. But users cannot view and/or replay content via the provenance UI, since the content are gone due to restart. For the content of flowfiles are still in the middle of flow during node failure/restart, we can't replay them from where it fails, when the node is back to normal. Instead, we have to fetch the same files from source again, and reprocess them end to end through the flow. If above is correct, I would say as long as we have source data permanently persisted in somewhere out of NiFi, we can always reprocess it when data in volatile content repository is lost. The only loss is the ability to view/replay them via Provenance UI. BTW, what happens when content exceeds the maximum size of repository? Out of memory exception? Auto purged from memory? auto archived in disk? If I set nifi.content.repository.implementation=org.apache.nifi.controller.repository.VolatileContentRepository Does that mean below properties are auto-disabled? nifi.content.claim nifi.content.repository.archive nifi.content.viewer.url Any comments are appropriated. Thanks.

alvinuw · ‎08-17-2017

Hi @Andy LoPresto, I am curious to know how much risks to use the volatile content repository? My understanding is: If there is a node failure/restart, For data has already been processed/persisted through the flow, no impact on our business or downstreams. But users cannot view and/or replay content via the provenance UI, since the content are gone due to restart. For the content of flowfiles are still in the middle of flow during node failure/restart, we can't replay them from where it fails, when the node is back to normal. Instead, we have to fetch the same files from source again, and reprocess them end to end through the flow. If above is correct, I would say as long as we have source data permanently persisted in somewhere out of NiFi, we can always reprocess it when data in volatile content repository is lost. The only loss is the ability to view/replay them via Provenance UI. BTW, what happens when content exceeds the maximum size of repository? Out of memory exception? Auto purged from memory? auto archived in disk? If I set nifi.content.repository.implementation=org.apache.nifi.controller.repository.VolatileContentRepository Does that mean below properties are auto-disabled? nifi.content.claim nifi.content.repository.archive nifi.content.viewer.url Any comments are appropriated. Thanks.

alvinuw · ‎08-14-2017

Hi @Andy LoPresto, Thank you for your work on the encrypted repositories tickets. We considered about "volatile content repository", but it affects all the workflows with data loss risk. "Decrypt and Re-encrypt on the fly" sounds like a better one for us. We can extract the non-sensitive fields as attributes, while leave the sensitive data in payload. Thanks.

alvinuw · ‎08-14-2017

Hi Guys, I noticed NiFi has encrypted provenance repository in v1.3. May I ask the timeline to release the encrypted content repository feature? Since we fetch encrypted financial data to NiFi, then decrypt them for some fields transformations before encrypting them again with another algo. Based on my understanding, the Decryption Processor will leave a copy of unencrypted data in disk, which is not acceptable for our compliance. Any idea about that? Thanks.

alvinuw · ‎08-10-2017

Awesome! Thanks.

Online	Offline
Last Visited	‎01-18-2018 07:36 PM

Member Since	‎07-19-2016 02:09 AM
Last Visited	‎01-18-2018 07:36 PM
Posts	91
Kudos received	10

Cloudera Community

Re: Two NiFi Instances/Clusters Communication Time...

Re: NiFi AuthorizerCreationException: Unable to lo...

NiFi AuthorizerCreationException: Unable to locate...

Re: NiFi Volatile Content Repository

Re: NiFi Volatile Content Repository

Re: NiFi Volatile Content Repository

NiFi Volatile Content Repository

Re: Encrypted NiFi Content Repository

Re: Encrypted NiFi Content Repository

Encrypted NiFi Content Repository

Re: How can we update NIFI workflow in production