Member since
07-19-2016
91
Posts
10
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1663 | 08-12-2016 05:05 PM |
01-18-2018
07:36 PM
Hi @Andy LoPresto, Just want to follow up on ticket https://issues.apache.org/jira/browse/NIFI-3834 Is it prioritized for this year? Thanks.
... View more
01-18-2018
07:14 PM
Hi @Matt Clarke, Kubernetes master ip is different with nifi-0.svc, which is an external pod dns. So nifi-0.svc can't been access outside of Kubernetes. I feel that nifi.web.proxy.host may work for this case. Is there a way to disable the Request Header Check? Thanks.
... View more
01-18-2018
06:32 PM
Hi @Matt Clarke, Thank you for your clarification. Yes, I use the secure mode. In NiFi v1.4, I setup the two properties as below. nifi.web.https.host= nifi-0.svc nifi.web.https.port= 9443 Please note nifi-0.svc is the pod name of the first nifi node. Then, we have nifi-1.svc, nifi-2.svc in the setting of the other nifi nodes. Normally, if we want to access NiFi cluster outside of Kubernetes, we have to use the master node ip of Kubernetes. e.g. https://master-ip:9443/nifi/ Above worked perfectly before. Now my question is in NiFi v1.5. What should I set for nifi.web.https.host? 1. If I don't do any changes, it gives me "Invalid Host Header" error. 2. If I set it as nifi.web.https.host = master-ip in all three nifi nodes in the cluster. The error in log is: 2018-01-18 17:10:28,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Cannot assign requested address One thing confuses me is: I have three nifi nodes in a cluster. Should I set "nifi.web.https.host" in three nifi.property file to the same ip value? BTW, are we able to disable this request header check? Thanks.
... View more
01-18-2018
04:00 PM
Hi @Matt Clarke, Thank you for your response. Is it nifi.web.http(s).host or nifi.web.proxy.host? I have already put nifi.web.https.host = the-nifi-node-ip We deploy nifi cluster in Kubernete, What should I change it to? Thanks.
... View more
01-18-2018
02:31 PM
2 Kudos
Hi Guys, I just upgraded NiFi from v1.4 to v1.5. The cluster is set in secure mode and running in Kubernetes. But I found below error in Web UI. System Error
The request contained an invalid host header [server:port] in the request [/nifi/]. Check for request manipulation or third-party intercept. Is it due to nifi.web.proxy.host property? How do I setup it? the IP of the nifi node? Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
01-10-2018
03:37 PM
Hi @Benoit Rousseau, Thank you for your response. For the jmx_exporter option, my main trouble is where can I find or create the "nifi.yml". I found a couple of template for Kafka and Cassandra. But How do I know the existing jmx metrics in NiFi. I didn't find any documents on it. Would you please share a template? Thanks.
... View more
01-08-2018
06:36 PM
Hi Guys, I know NiFi provides ReportTask component to export metrics to external monitoring systems. I was wondering whether we can use a generic approach to collect NiFi metrics via JMX. For example, there is a JMX exporter to Prometheus by collecting metrics from java application. May I use it to collect NiFi metrics to Prometheus? Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
10-20-2017
03:51 PM
Hi guys, I setup Volatile content repo for my NiFi cluster, the other repos are all file-based. However, It shows me "Content Repository out of space " exception, when I process only 20M dataset. It seems below settings don't work. nifi.content.repository.implementation=org.apache.nifi.controller.repository.VolatileContentRepository nifi.volatile.content.repository.max.size=1 GB
nifi.volatile.content.repository.block.size=32 KB From below code comment, I found "
If no Backup Repository has been specified, when the threshold is exceeded, an IOException will be thrown" https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/VolatileContentRepository.java My questions are: 1. How to setup the Backup Repository for volatile content repository? 2. Is my above setting for "nifi.volatile.content.repository.max.size" correct? It seems still using the default 100MB. Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
10-19-2017
08:53 PM
@Abdelkrim Hadjidj Perfect! Much clearer now. Thanks.
... View more
10-18-2017
08:41 PM
Hi @Abdelkrim Hadjidj Thank you for your response. Very helpful. I do have retrieve site-to-site details policy set for all nodes in the cluster. And the RPG can connect and refresh. However, the "Retrieve Data via site-to-site" item is grey and disabled for the Input Port. Then, I found below document. https://community.hortonworks.com/questions/67255/how-can-i-set-an-inputoutput-ports-for-a-remote-pr.html It says Input Port must be in the root level to enable S2S. I was wondering is it a limitation? How about I want to put my entire flow in a processor group? BTW, after S2S is enabled. I still find the file distribution is not even. One node processes more than the others. Is there a way to control the distribution? Thanks.
... View more
10-17-2017
08:14 PM
Hi @Matt Burgess Thank you for your response. I setup RPG and an Input Port. It seems RPG can connect the remote nifi instance. However, when I click Manage Remote Ports, it can't display the input port I added. BTW, I use a secured 3 node cluster with S2S enabled. Please see the attached screenshot. Did I miss anything? Thanks.
... View more
10-17-2017
06:28 PM
Hi Guys, I have a flow to ingest files from sftp server, by ListSFTP(primary node) -> FetchSFTP(all nodes). I uploaded 20 files(5MB each) to sftp server. I found all files are fetched and processed by the primary node. I was wondering how to make the tasks evenly distributed among all nodes, rather than only primary node. Is it because 20 files is small number or each file is too small? Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
10-17-2017
02:28 PM
Hi @Wynner, When I set Volatile content repository properties: nifi.volatile.content.repository.max.size=1 GB nifi.volatile.content.repository.block.size=32 KB It shows me "Content Repository out of space " exception, when I process a 20M dataset. From the code comment, I found " If no Backup Repository has been specified, when the threshold is exceeded, an IOException will be thrown" https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/VolatileContentRepository.java 1. How to setup the Backup Repository for volatile content repository? 2. Is my above setting for "nifi.volatile.content.repository.max.size" correct? It seems still using the default 100MB. Thanks.
... View more
09-28-2017
06:50 PM
Hi @Bryan Bende I have a new issue when I tried to login Initial Admin Identity by using username and password. In previous test, I successfully made the client cert login work. I have an principle "admin@NIFI.COM" in kerberos, and "admin@NIFI.COM" as Initial Admin in authorizers.xml When I login as "admin", the user log shows: 2017-09-28 14:27:06,112 INFO [NiFi Web Server-120] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for admin@NIFI.COM 2017-09-28 14:32:13,193 ERROR [NiFi Web Server-19] o.a.nifi.web.security.jwt.JwtService There was an error validating the JWT io.jsonwebtoken.JwtException: Unable to validate the access token. Caused by: io.jsonwebtoken.SignatureException: JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted. When the nifi cluster is up, I can login in as "admin". After logout, and login again, NiFi gives me above error. Is it caused by the browser cache the token? Thanks.
... View more
09-25-2017
07:53 PM
Hi @bkosaraju, Below command works to generate a token. $ curl -k 'https://<nifi-server>:9091/nifi-api/access/token'-H 'Accept-Encoding: gzip, deflate, br'-H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8'-H 'Accept: */*'--data 'username=<username>&password=<password>'--compressed However, $ curl -k -X GET 'https://<nifi-server>:9091/nifi-api/cluster/summary' -H 'Authorization: Bearer $token' --compressed returns: Unable to validate the access token. Any idea about it? Do I need to run above commands in one nifi node, or any machine? Thanks.
... View more
09-21-2017
03:39 PM
Hi @Egor Pushkin, Since we build a secured NiFi cluster with Kerberos for authentication in production. Does this tool support that? I didn't find it from your document. https://github.com/Glympse/terraform-provider-nifi/blob/develop/docs/syntax.md Thanks.
... View more
09-20-2017
01:39 PM
hi @bkosaraju, Thank you for your response. I did enable the kerberos-provider section. I guess I need to install krb5-user on the machine I will run the above commends, right? I was just wondering whether I need to use keytool to import nifi-cert.pem in my machine? What's the usage of nifi-cert.pem generated by the tls-tool? Thanks.
... View more
09-19-2017
06:40 PM
Hi guys, I build a secured NiFi cluster authenticated by Kerberos and use TLS tool standalone mode to generate client certificates. I am able to use NiFi Rest Api to access the unsecured cluster. When I want to use username/password of principle created in Kerberos to build a REST call for a secured cluster in curl, I got below error. Unknown user with identity 'anonymous'. Contact the system administrator. I was wondering whether I should install some certificate in my local machine to use curl. Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
09-19-2017
06:34 PM
Hi @Arti Wadhwani, Could we use username/password of the principle created in Kerberos to login? Since I don't have kinit in nifi node. Thanks.
... View more
09-15-2017
07:12 PM
Hi @Bryan Bende, I deleted my previous replies since it's misleading. Let me restate the current issue clearly. The reason I saw below error "Unknown user with identity 'alvin@NIFI.COM'. Contact the system administrator." is due to I login Initial Admin account to add user "CN=alvin, OU=NIFI.COM" in NiFi web. The correct way is to add 'alvin@NIFI.COM' and set its policy. After that my issue is gone. Thanks.
... View more
09-08-2017
08:06 PM
Hi @Bryan Bende, It seems the authentication works now. But I have issue with authorization. I used the file-based authorization approach. After I type username and password, it says "Unknown user with identity 'alvin@NIFI.COM'. Contact the system administrator." instead of prompting out "Justification" page. Any idea? Thanks. 2017-09-08 19:48:33,294 INFO [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for alvin@NIFI.COM
2017-09-08 19:48:33,295 INFO [NiFi Web Server-29] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[alvin@NIFI.COM], groups[none] does not have permission to access the requested resource. Unknown user with identity 'alvin@NIFI.COM'. Returning Forbidden response. 2017-09-08 19:48:33,295 DEBUG [NiFi Web Server-29] o.a.n.w.a.c.AccessDeniedExceptionMapper
org.apache.nifi.authorization.AccessDeniedException: Unknown user with identity 'alvin@NIFI.COM'.
at org.apache.nifi.web.api.FlowResource.authorizeFlow(FlowResource.java:230)
at org.apache.nifi.web.api.FlowResource.getCurrentUser(FlowResource.java:316)
... View more
09-08-2017
03:52 PM
Hi @Bryan Bende, Thank you for this tutorial. Very helpful. May I ask do we need to also set up below properties in a cluster mode? nifi.kerberos.service.principal nifi.kerberos.service.keytab.location Is kerberos.service and kerberos.spnego one or another? Or both must be set? Since I saw some examples set up kerberos.service rather than kerberos.spnego, e.g. https://community.hortonworks.com/articles/34147/nifi-security-user-authentication-with-kerberos.html Thanks.
... View more
09-01-2017
02:56 PM
Hi @Wynner,
Thanks for your comments. You are right, it's the authorizers.xml format issue.
Actually, I am using nifi-1.4-snapshot. That's why my authorizers.xml is different from the 1.3 one.
When I switched to the 1.3 version authorizers.xml with nifi-1.4-snapshot, the above issue was gone.This confuses me.
Then, when I have below in authorizers.xml file <propertyname="Node Identity 1">CN=nifi-0, OU= NIFI.COM</property> I saw errors from nifi-app.log, it seems the default OU of node identity is "NIFI".
2017-09-01 14:09:08,854 DEBUG [NiFi Web Server-19] o.a.n.w.a.c.IllegalStateExceptionMapper
java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.
at org.apache.nifi.web.api.AccessResource.createAccessTokenFromTicket(AccessResource.java:349)
2017-09-01 14:09:09,045 INFO [NiFi Web Server-127] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<CN=nifiadmin, OU=NIFI.COM><CN=nifi-0, OU=NIFI>) GET https://nifi-0:9443/nifi-api/flow/current-user (source ip: 10.244.1.95)
2017-09-01 14:09:09,048 WARN [NiFi Web Server-127] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=nifi-0, OU=NIFI
2017-09-01 14:09:09,048 WARN [NiFi Web Server-127] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=nifi-0, OU=NIFI But whenIf I set it as "CN=nifi-0, OU= NIFI" in above file, it works.
... View more
08-31-2017
07:52 PM
Hi guys, When I secure a three nodes nifi cluster, I got below errors during start up. I use TLS tool to create certificates, and use client-certificate for authentication. The setting in my authorizers.xml is below: <accessPolicyProvider> <identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">file-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">CN=nifiadmin, OU=NIFI.COM</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1">CN=nifi-0, OU=NIFI.COM</property> <property name="Node Identity 2">CN=nifi-1, OU=NIFI.COM</property> <property name="Node Identity 3">CN=nifi-2, OU=NIFI.COM</property> </accessPolicyProvider> From nifi-user.log ouput, it seems NiFi can locate the DN set in authorizers.xml. However, I don't understand why nifi-bootstrap.log says NiFi can't initial admin. Are users.xml and authorizations.xml auto-created and populated by NiFi? After checking, it seems "Initial Admin Identity" user and administrative policies are NOT added to the users.xml and authorizations.xml files. $ cat nifi-user.log 2017-08-31 18:54:28,424 INFO [main] o.a.n.a.FileUserGroupProvider Creating new users file at /opt/nifi/nifi-1.4.0-SNAPSHOT/./conf/users.xml 2017-08-31 18:54:28,453 INFO [main] o.a.n.a.FileUserGroupProvider Users/Groups file loaded at Thu Aug 31 18:54:28 UTC 2017 2017-08-31 18:54:28,458 INFO [main] o.a.n.a.FileAccessPolicyProvider Creating new authorizations file at /opt/nifi/nifi-1.4.0-SNAPSHOT/./conf/authorizations.xml 2017-08-31 18:54:28,475 WARN [main] org.apache.nifi.authorization.FlowParser Flow Configuration does not exist or was empty 2017-08-31 18:54:28,475 INFO [main] o.a.n.a.FileAccessPolicyProvider Populating authorizations for Initial Admin: CN=nifiadmin, OU=NIFI.COM $ cat nifi-bootstrap.log 2017-08-31 18:54:13,151 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 41600 2017-08-31 18:54:29,443 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Error creating bean with name 'niFiWebApiSecurityConfiguration': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire method: public void org.apache.nifi.web.NiFiWebApiSecurityConfiguration.setJwtAuthenticationProvider(org.apache.nifi.web.security.jwt.JwtAuthenticationProvider); nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate initial admin CN=nifiadmin, OU=NIFI.COM to seed policies 2017-08-31 18:54:29,444 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down...
2017-08-31 18:54:30,576 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi
... View more
Labels:
- Labels:
-
Apache NiFi
08-21-2017
01:53 PM
Hi @Wynner, Thank you for your clarification. May I understand the Volatile Content Repository only keeps the content for the flowfiles running in the flow? Once the flowfiles are finished, the contents are wipe out from memory, and can't be found in Repository. In contrast, the File-based Content repository will keep the contents for flowfiles even out of the flow, based on the retention setting. Thanks.
... View more
08-18-2017
08:20 PM
Hi @Wynner, Thank you for your answers. You said that If the configured memory limit is reached, and more content data can't be added to Volatile Content Repository. But how do we free memory for Volatile Content Repository, if it can't auto clean up? Since this day will come sooner or later. It could be an issue to make the Volatile Content Repository useless. Thanks.
... View more
08-18-2017
02:29 PM
Hi @Egor Pushkin, This is really fantastic! May I ask what are the main advantages of this Terraform approach, comparing with building a tool by NiFi REST api? Thanks.
... View more
08-18-2017
01:50 PM
I am curious to know how much risks to use the volatile content repository? My understanding is: If there is a node failure/restart, For data has already been processed/persisted through the flow, no impact on our business or downstreams. But users cannot view and/or replay content via the provenance UI, since the content are gone due to restart. For the content of flowfiles are still in the middle of flow during node failure/restart, we can't replay them from where it fails, when the node is back to normal. Instead, we have to fetch the same files from source again, and reprocess them end to end through the flow. If above is correct, I would say as long as we have source data permanently persisted in somewhere out of NiFi, we can always reprocess it when data in volatile content repository is lost. The only loss is the ability to view/replay them via Provenance UI. BTW, what happens when content exceeds the maximum size of repository? Out of memory exception? Auto purged from memory? auto archived in disk? If I set nifi.content.repository.implementation=org.apache.nifi.controller.repository.VolatileContentRepository Does that mean below properties are auto-disabled? nifi.content.claim nifi.content.repository.archive nifi.content.viewer.url Any comments are appropriated. Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
08-17-2017
04:04 PM
Hi @Andy LoPresto, I am curious to know how much risks to use the volatile content repository? My understanding is: If there is a node failure/restart, For data has already been processed/persisted through the flow, no impact on our business or downstreams. But users cannot view and/or replay content via the provenance UI, since the content are gone due to restart. For the content of flowfiles are still in the middle of flow during node failure/restart, we can't replay them from where it fails, when the node is back to normal. Instead, we have to fetch the same files from source again, and reprocess them end to end through the flow. If above is correct, I would say as long as we have source data permanently persisted in somewhere out of NiFi, we can always reprocess it when data in volatile content repository is lost. The only loss is the ability to view/replay them via Provenance UI. BTW, what happens when content exceeds the maximum size of repository? Out of memory exception? Auto purged from memory? auto archived in disk? If I set nifi.content.repository.implementation=org.apache.nifi.controller.repository.VolatileContentRepository Does that mean below properties are auto-disabled? nifi.content.claim
nifi.content.repository.archive
nifi.content.viewer.url
Any comments are appropriated. Thanks.
... View more