Member since
09-29-2015
33
Posts
32
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3090 | 05-09-2017 04:00 PM | |
521 | 12-08-2016 01:04 PM | |
524 | 12-06-2016 01:57 PM | |
5630 | 11-03-2016 02:10 PM | |
2342 | 11-01-2016 05:01 PM |
02-25-2020
10:43 AM
The clientId can be any string that is unique to your client. You can generate one. If one is not specified the endpoints will generate one and return it for you. Please open your web browsers developer tools and watch some of the requests in action.
... View more
02-04-2019
02:32 PM
@Jayanth S Are the variable's your templated components referencing defined in a Process Group that is also part of this template?
... View more
03-23-2018
01:06 PM
Sure... https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#config-users-access-policies Hope this helps!
... View more
03-21-2018
07:31 PM
4 Kudos
This article assumes that you're already configured to use Ranger authorization in HDF 3.0.x or earlier and that Ranger instance is syncing the available users/groups with your Directory Server. In order to leverage policies defined with these groups, we'll need to upgrade to HDF 3.1.0 or later and migrate your configuration to utilize a new Ranger authorizer in NiFi. In short, you’ll need to update your authorizers.xml file to configure a userGroupProvider and modify your authorizer to be a ManagedRangerAuthorizer. The idea is that previously an Authorizer was configured which was responsible for making access decisions, managing policies, and managing users/groups. Whether or not the Authorizer supported managing policies and users/groups was optional. The default File base authorizer supported this. The existing Ranger authorizer did not (it only made access decisions). For HDF 3.1 we decoupled these concepts so you can independently configure the Authorizer (makes access decisions), an AccessPolicyProvider (manage policies), and a UserGroupProvider (manage users/groups). Documentation exists for these three concepts and includes examples [1]. What you’ll need to do is update your authorizers.xml and add a UserGroupProvider to bind your users/groups from your Directory Server that Ranger is syncing to. There are a couple detailed examples of binding users/groups from a Directory Server using the LdapUserGroupProvider. Searching for this term in [1] should direct you to the relevant examples. Following the concepts defined above, the ManagedRangerAuthorizer will act as the AccessPolicyProvider (since the policies are defined there) and the Authorizer (Ranger ultimately makes the access decisions). To configure your ManagedRangerAuthorizer, we can take your existing configuration for RangerNiFiAuthorizer and 1) modify the class and 2) add a reference to your LdapUserGroupProvider. The remainder of the configuration should remain unchanged. This should look something like this: <authorizer>
<identifier>ranger-provider</identifier>
<class>org.apache.nifi.ranger.authorization.ManagedRangerAuthorizer</class> <!-- 1) UPDATE CLASS NAME -->
<property name="User Group Provider">ldap-user-group-provider</property> <!-- 2) REFERENCE USER GROUP PROVIDER -->
<property name="Ranger Audit Config Path">…</property>
<property name="Ranger Security Config Path">…</property>
<property name="Ranger Service Type">…</property>
<property name="Ranger Application Id">…</property>
<property name="Ranger Admin Identity">…</property>
</authorizer> This is all that you should need to do. In totality, the structure of your authorizers.xml file should looks like this: <authorizers>
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">...</property>
<property name="Manager DN">...</property>
<property name="Manager Password">...</property>
<property name="TLS - Keystore">...</property>
<property name="TLS - Keystore Password">...</property>
<property name="TLS - Keystore Type">...</property>
<property name="TLS - Truststore">...</property>
<property name="TLS - Truststore Password">...</property>
<property name="TLS - Truststore Type">...</property>
<property name="TLS - Client Auth">...</property>
<property name="TLS - Protocol">...</property>
<property name="TLS - Shutdown Gracefully">...</property>
<property name="Referral Strategy">...</property>
<property name="Connect Timeout">...</property>
<property name="Read Timeout">...</property>
<property name="Url">...</property>
<property name="Page Size">...</property>
<property name="Sync Interval">...</property>
<property name="User Search Base">...</property>
<property name="User Object Class">...</property>
<property name="User Search Scope">...</property>
<property name="User Search Filter">...</property>
<property name="User Identity Attribute">...</property>
<property name="User Group Name Attribute">...</property>
<property name="Group Search Base">...</property>
<property name="Group Object Class">...</property>
<property name="Group Search Scope">...</property>
<property name="Group Search Filter">...</property>
<property name="Group Name Attribute">...</property>
<property name="Group Member Attribute">...</property>
</userGroupProvider>
<authorizer>
<identifier>ranger-provider</identifier>
<class>org.apache.nifi.ranger.authorization.ManagedRangerAuthorizer</class>
<property name="User Group Provider">ldap-user-group-provider</property>
<property name="Ranger Audit Config Path">...</property>
<property name="Ranger Security Config Path">...</property>
<property name="Ranger Service Type">...</property>
<property name="Ranger Application Id">...</property>
<property name="Ranger Admin Identity">...</property>
</authorizer>
</authorizers> [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizers-setup
... View more
- Find more articles tagged with:
- Data Ingestion & Streaming
- How-ToTutorial
- NiFi
Labels:
12-21-2017
10:19 PM
7 Kudos
Introduction into Process Group Variables The ability configure variables at runtime in the UI was introduced in Apache NiFi 1.4 and will subsequently land in HDF 3.1. Variables can be referenced in property values of Processors and Controller Services that support expression language. This provides a centralized place to configure common values that may be referenced by multiple components and values that may be environmentally dependent. This guide provides a brief introduction to their usage. Configuration Variables are configured in a dialog that is accessed through the context menu. The menu item is available when the selection is a single Process Group. Below is an example of what the variable dialog would look like. The table will list all variables that are available in the scope of the current Process Group, the variable name, and variable value. Only variables defined in the current Process Group are editable. The table also contains a navigation link to go to the Process Group where inherited variables are defined. On the right side of the dialog, the variable selected in the table is shown along with all components that reference it. Scoping Processors and Controller Services encapsulated in the current Process Group will have access to variables defined at that Process Group and all ancestor Process Groups. If a nested Process Group defines a variable with the same name as one defined at a higher scope, it will override that value. This is visualized in the Variable dialog using a strikethrough. This overriding process can be summarized by saying that when a component references a variable, the value will come from the closest definition.
Additionally, variables defined in the properties files specified using the nifi.variable.registry.properties configuration can be overridden by Process Group Variables. Referencing Referencing a variable is done using expression language. If a Processor or Controller Service property supports expression language, a variable name can be used as the subject. The order of precedence for expression language subjects is flow file attributes, Process Group variables, variables defined in the properties file, system environment variables, and finally JVM system property. For more information regarding Expression Language, please refer to the Expression Language Guide available in Apache NiFi.
... View more
- Find more articles tagged with:
- Data Ingestion & Streaming
- How-ToTutorial
- NiFi
- variable-registry
Labels:
07-19-2017
01:45 PM
2 Kudos
When running Apache NiFi behind a proxy there are a couple of key items to follow during deployment.
1) NiFi is comprised of a number of web applications (web ui, web api, documentation, custom ui's, data viewers, etc). So the mapping needs to be configured for the root path. That way all context paths are passed through accordingly. For instance, if only the /nifi context path was mapped, the custom ui for UpdateAttribute will not work since it's available at /update-attribute-ui-<version>.
2) NiFi's REST API will generate URI's for each component on the graph. Since requests are coming through a proxy, certain elements of the URI's being generated need to be overridden. Without overriding the users will be able to view the dataflow on the canvas but will be unable to in modify existing components. Requests will be attempting to call back directly to NiFi, not through the proxy. The elements of the URI can be overridden by adding the following HTTP headers when the proxy generates the HTTP request to the NiFi instance: X-ProxyScheme - the scheme to use to connect to the proxy X-ProxyHost - the host of the proxy X-ProxyPort - the port the proxy is listening on X-ProxyContextPath - the path configured to map to the NiFi instance
3) If NiFi is running securely, any proxy needs to be authorized to proxy user requests. These can be configured in the NiFi UI through the Global Menu. Once these permissions are in place proxies can begin proxying user request. The end user identity must be relayed in a HTTP header. For example, if the end user sent a request to the proxy, the proxy must authenticate the user. Following this the proxy can send the request to NiFi. In this request an HTTP header should be added as follows. X-ProxiedEntitiesChain: <end-user>
If the proxy is configured to send to another proxy, the request to NiFi from the second proxy should contain a header as follows. X-ProxiedEntitiesChain: <end-user-identity><proxy-1-identity>
... View more
- Find more articles tagged with:
- apache-nifi
- Data Ingestion & Streaming
- How-ToTutorial
- proxy
Labels:
05-10-2017
01:14 PM
Awesome! Glad everything is working. Just to follow with extra details so others that come across this post will benefit... There are a couple of key items to know when standing up NiFi behind a proxy. 1) NiFi is comprised of a number of web applications (web ui, web api, documentation, custom ui's, etc). So you'll need to set up your mapping to the root path. That way all context paths are pass through accordingly. For instance, if you only mapped the /nifi context path, the custom ui for Update Attributes will not work since it's available at /update-attribute-ui-<version>. 2) NiFi's rest api will generate uri's for each component on the graph. Since your coming through a proxy, you'll need to override certain elements of the uri's being generated. This is why your able to view the graph, but you cannot modify existing Processors. It's attempting to call back directly to your NiFi, not through your proxy. You can override the elements of the uri by adding the following HTTP headers when your proxy generates the HTTP request to the NiFi instance: X-ProxyScheme - the scheme to use to connect to your proxy (https in this case) X-ProxyHost - the host of your proxy X-ProxyPort - the port your proxy is listening on X-ProxyContextPath - the path you've configured to map to the NiFi instance
... View more
05-09-2017
04:00 PM
1 Kudo
@Jatin Kheradiya Is there a proxy in front of the NiFi cluster? What is listening on the public IP and what is listening on the private IP?
... View more
03-22-2017
04:18 PM
6 Kudos
In Apache NiFi, Controller Services are shared services that can be used by Processors, Reporting Tasks, and other Controller Services. In order to modify a Controller Service, all referencing components must be stopped. This means that the user attempting to modify the Controller Service, must also have permissions to modify all referencing components. With the introduction of multi-tenant authorization, Controller Service scoping was updated to ensure service usage does not become so broad that it becomes impossible to update the Controller Service because no users are allowed to modified all components that reference it. Within the dataflow, to scope a Controller Service and limit it's availability, a Controller Service can be created within any Process Group. Controller Services created within a Process Group will be available to all descendent components. This means that the service will be available to all Processors and Controller Services defined in that Process Group and below. In a typical multi-tenant environment users permissions will typically be bound to a Process Group. By scoping the Controller Service in the same manner, it's usage is limited accordingly.
Within the Controller (the infrastructure that executes the dataflow), Controller Service availability is limited to Reporting Tasks and other services defined here. This scoping works in hand with the scoping implemented within the dataflow context. Users with permissions to Reporting Tasks will also have access to services defined here. Realizing these differences in Controller Service availability was often difficult to understand. However, upcoming UX changes to Controller Services helps alleviate some of these challenges. Below are screenshots for the different contexts and highlight the changes which help understand the service availability.
... View more
- Find more articles tagged with:
- apache-nifi
- Data Ingestion & Streaming
- FAQ
- NiFi
- nifi-controller-service
- ux
Labels:
03-21-2017
03:11 PM
1 Kudo
@Dan Rogers NiFi runs in a single JVM. As such, all Processors in the data flow will share the resources of the same JVM. These details can be monitored through the system diagnostics. They are access in the UI by clicking on the system diagnostics link in the Summary page off of the global context menu. They are loaded from the system diagnostics rest endpoint which can be accessed here: http://localhost:8080/nifi-api/system-diagnostics
Also check out the docs [1]. Click the System Diagnostics at the bottom to expand. Hope this helps! [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
... View more
03-03-2017
08:42 PM
Sure, the UI polls from the bulletin-board here [1]. I discussed those endpoints a little bit below. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/bulletin-board/nf-bulletin-board.js#L252
... View more
03-03-2017
08:36 PM
The website only contains the documentation for the current release. However, you can always access the documentation within the application. You should be able to click the help link in the upper right hand corner. From there, scroll down to the bottom on the left and access the Rest API docs.
... View more
02-23-2017
05:28 PM
@Ramakrishnan V Just to clarify, is the password in your code encrypted? If so then yes, you would need to decrypt the password being sent in the request to /nifi-api/access/token. NiFi only supports username/password login's over HTTPS so that communication will be secure.
... View more
12-21-2016
09:00 PM
The documentation is the link that you already have. However, I would suggest checking out the Developer Tools in your browser. The UI uses the REST API for all interactions with the framework. The Developer Tools even allows for you to copy the command as a cUrl command. Here's the command for starting a Process Group. curl 'http://localhost:8080/nifi-api/flow/process-groups/232d4ab5-0159-1000-1962-8a2afbbe9dc9' -X PUT -H 'Accept-Encoding: gzip, deflate, sdch, br' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8080/nifi/' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' --data-binary '{"id":"232d4ab5-0159-1000-1962-8a2afbbe9dc9","state":"RUNNING"}' --compressed
... View more
12-20-2016
01:20 PM
If the processor in question is only running every 10 seconds, I would suggest running the Reporting Task more frequently. However, if a single invocation of the processor will result in more than 5 bulletins, I think your going to have trouble capturing them all with this approach. Scrapping the logs may be your safest bet.
... View more
12-19-2016
07:21 PM
The Reporting Task would follow the same pattern as the REST API. The only difference is that you'll be able to schedule it to run much more frequently than you could poll the REST API. You do need to maintain some local state, specifically the last bulletin seen. Then in your on trigger you can findBulletins starting from that id. Update the last bulletin seen. Repeat in the next onTrigger. Since we're limited to 5 per component, that means that this will need to run at least once before the component is able to generate more than 5 bulletins. The REST API follows a similar patten for the Bulletin Board but it's not guaranteed to pull back every single bulletin. It is guaranteed to pull back up to the last 5. Once the next bulletin is triggered, the oldest one falls out of scope. Polling the REST API will definitely be slower than running a Report Task and will increase the risk of possibly missing one. Alternatively you can scrap the logs. The bulletins are essentially notifications of log messages.
... View more
12-19-2016
01:33 PM
1 Kudo
This is by design. A bulletin is simply a notification about a passing event. To get access to every event you'd either need to monitor the log files or implement a custom Reporting Task to exfil the bulletins as they occur. The reporting context provided to the task has access to the bulletin repository. We have a similar Reporting Task for sending provenance events [1]. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java
... View more
12-08-2016
01:04 PM
The initial value that is shown in the URL field is a placeholder to provide an example of the URL that is expected. All fields that do not have focus do have a light bluish color background. But once a field has focus the field should turn white. Additionally, once you start typing in place of the placeholder that example value should disappear. What browser are you using? If your using an unsupported browser [1] you could see unexpected behavior. Assuming the browser version is good, you might also try clearing your browser cache to ensure it's not picking up old styles from a previous NiFi version you may have used. Feel free to respond with some screenshots if these don't help. Thanks! [1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#browser-support
... View more
12-06-2016
01:57 PM
What is the behavior your currently seeing? The endpoint your invoking looks correct. However, it appears there may be an extra forward slash after nifi-api. Matt
... View more
11-09-2016
08:58 PM
Was the HTML content actually valid XML? Did the content viewer open? If so, is the content viewer unable to show the content in it's 'formatted' form? What about the 'original' or 'hex' forms?
... View more
11-09-2016
01:14 PM
As was previously mentioned, the content type is specified in the flowfile attribute mime.type. Some Processors will automatically set this value. However, UpdateAttribute can be used to explicitly set it. There is no content viewer for HTML bundled with Apache NiFi currently. This is an extension point so one could be developed and dropped into your instance. Alternatively, you could just use application/xml or text/plain.
... View more
11-03-2016
02:10 PM
3 Kudos
Indranil Roy, For Apache NiFi 1.0.0 the REST API was completely refactored to promote multi-tenancy. Specifically, the endpoints have been reorganized to align with the resources being authorized and component specific revisions have been introduced. I would highly recommend using the Developer Tools in your browser to see the APIs in action. Here is an example curl commend for starting a processor. curl 'http://localhost:8080/nifi-api/processors/2a817541-0158-1000-771e-93ca6d670429' -X PUT -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' --data-binary '{"revision":{"clientId":"2a81087c-0158-1000-1362-5452a699615c","version":3},"component":{"id":"2a817541-0158-1000-771e-93ca6d670429","state":"RUNNING"}}' In order to get the revision details for this component it would be available in the flow returned for a given process group. Here's an example of that curl commend. curl 'http://localhost:8080/nifi-api/flow/process-groups/<process-group-id>'
... View more
11-01-2016
05:08 PM
Technically, IE 11 is not a supported browser [1]. Would it be possible to try Chrome, Firefox, or Microsoft Edge? [1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#browser-support
... View more
11-01-2016
05:01 PM
What browser are you using? Is the browser you're using also on the Windows Server? In your browser, are you able to open up the Developer Tools and look at the network traffic? It would help to narrow down where the 'slowless' is being introduced. It could be due to client side (browser) performance specifically if your flow has many components. It could be introduced simply by network latency. It could also be introduced server side while NiFi is handling the request and generating a response (particularly if clustered).
... View more
09-16-2016
06:21 PM
1 Kudo
The documentation for a given release can be accessed directly from within the application. Navigate to your NiFi UI and click the 'help' link in the upper right hand corner. Scroll to the bottom of the pane on the left hand side and click 'Rest Api'.
... View more
09-15-2016
12:50 PM
The documentation on the website will be for the most recently released version of Apache NiFi (specifically 1.0.0). When this question was asked Apache NiFi was still on the 0.x baseline. In 1.x bulletins are available through the bulletin-board endpoint /nifi-api/flow/bulletin-board or bundled with each component. For instance, /nifi-api/flow/process-groups/{id} for all components under some process group or /nifi-api/processors/{id} for a specific processor.
... View more
09-08-2016
05:26 PM
What do you mean by local authentication? Using Kerberos authentication, Apache NiFi supports using a local ticket via SPNEGO. Apache NiFi 0.x (which maps to HDF 1.x) supports authentication using client certificates, LDAP, and Kerberos. This also applies to Apache NiFi 1.x.
... View more
06-27-2016
02:46 PM
1 Kudo
James, Sorry for the delay response here. The issue your describing was a bug and it has been addressed in the upcoming NiFi 0.7.0 release. Here's the JIRA in question [1]. Sorry for the inconvenience. Matt [1] https://issues.apache.org/jira/browse/NIFI-1564
... View more
03-28-2016
01:54 PM
2 Kudos
The bulletins are available through the Rest Api. They are accessed primarily through two different endpoints. /controller/status Return controller level status. This will include status and bulletins for cluster/node issues, reporting tasks, and controller services. /controller/process-groups/{id}/status Return process group specific status. This will include status and bulletins for all components in the specified process group. This endpoint supports a recursive flag to indicate whether status for all descendent components is included.
... View more
11-05-2015
05:54 PM
Requests to NiFi's REST API to change the data flow (mutable requests) require a revision to be included. Since multiple users may be working with the same NiFi instance at the same time, this is used as a simple check to verify the user who submitted the request acknowledges they have the most current version of the data flow before applying their change. The revision is comprised of two pieces. The first is the version. The version is a one-up number that is updated by NiFi during a mutable request and is included in the response. The next mutable request should include that updated version. This works well until a user wants to submit multiple requests concurrently. To support this case the revision also includes a clientId. If the clientId in the revision of the current request matches the clientId of the last successful mutable request, the version does not need to match. It is up to the client to manage the revision version and check with http://{host}:{port}/nifi-api/controller/revision in order to see if another client has updated the flow. The clientId can be any string. It's accepted in most endpoints so it could be added to requests in $.ajaxSetup for instance. If the clientId is not specified one will be generated. There is no session, it's simply generates a UUID. You could use any string you may want to represent your client. In the NiFi UI, we GET the flow http://{host}:{port}/nifi-api/controller/process-groups/root?verbose=true and use the clientId that it generates for all subsequent requests.
... View more