About MattWho

MattWho · ‎02-01-2017

@Narasimma varman Make sure the user the NiFi process is running as on your server has the necessary permissions to access that directory path and remove files from it. Matt

MattWho · ‎01-31-2017

@bhumi limbu NiFi FlowFile attributes/metadata lives in heap. The List based processors return a complete listing from the target and then creates a FlowFile for each File in that returned listing. The FlowFiles being created are not committed to the list processor's success relationship until all have been created. So you end up running out of NiFi JVM heap memory before that can happen because of the size of your listing. As NiFi stands now, the only option is to use multiple list processors with each producing a listing of on a subset of the total files from your source system. You could use the "Remote Path", "Path Filter Regex" and/or "File Filter Regex" properties in the listSFTP to filter what data is selected to help reduce the heap usage. You could also increase the available heap to your NiFi's JVM in the bootstrap.conf file; however, I find it unlikely considering the number of FlowFiles you are listing that you will not still run out of heap memory. I logged a Jira in Apache NiFi with a suggested change to how these processors produce FlowFiles from the listing returned by these types of processors: https://issues.apache.org/jira/browse/NIFI-3423 Thanks, Matt

MattWho · ‎01-31-2017

@Raj B Not all NiFi Processors will write attributes to FlowFiles about failures or errors. The documentation for each processor should include what attributes are written by that processor and what information those attributes will contain. There is no global enforcement by the NiFi controller on what attributes a processor must create. This is completely in the control of the developer who wrote each processor. That being said, it is good practice that any processor that has a "Failure" relationship should output an "Error" level log message that dictates the nature of the failure. This Error log message would contain the specific processor that produced the ERROR as well as information on the specific FlowFile that was routed to failure and the nature of the failure. It is possible to build a dataflow that monitors NiFi's nifi-app.log (TailFile processor) for ERROR log messages, parses out the relevant information and pass that along to some monitoring system. Thanks, Matt

MattWho · ‎01-31-2017

@Joshua Adeleke There is obviously something else going on within your system that is affecting leader election. When you start you NiFi, do you see a leader election/Cluster coordinator count down timer running? Is your NiFi having trouble talking to your Zookeeper? Looks like you are having timeout issues talking to your zookeeper. I still don't understand why you are running your NiFi as a 1 node cluster if all you want is a single standalone instance of NiFi. A NiFi configured as a standalone instance does not need zookeeper and also does not perform election of cluster coordinator or primary node. Setting the following property in your nifi.properties and restarting will make you NiFi a truly standalone instance: nifi.cluster.is.node=false Matt

MattWho · ‎01-31-2017

@Anishkumar Valsalam The intent of an "Admin" account in NiFi is to setup users who can do the following: - Access the UI - Setup NiFi controller level Controller Services and Reporting Tasks - Add new users and groups - Set Access policies for those users When it comes to building dataflows on the canvas, that is more of a dataflow managers role. The "Initial Admin Identity" by default does not even get this roles capabilities/accesses, but has the ability through the policies he was granted to grant himself or other users the access needed to build dataflows. In order to enable the dataflow building icons along the top of the UI, those users will need to be granted the "view the component" and "modify the component" access policies on the specific process group in which the want to build their dataflows. For more information on the various "Access policies" and what capabilities they provide to the assigned users, the NiFi Admin Guide can be found under help within your installed NiFi's UI (Most accurate for whichever version you have installed) or at the following link: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization Thanks, Matt

MattWho · ‎01-30-2017

@Anishkumar Valsalam Glad to hear you got it setup. The "Access all Policies" access policy willnot work if you have not also granted the users the "access users/user groups" access policy. They need to be able to view users in order to grant them access policies. If this answer was helpful to solving your issue, will you please accept it. Thank you, Matt

MattWho · ‎01-30-2017

@Anishkumar Valsalam Hello, During initial setup of a secured NiFi installation NiFi allows you to specify a single "initial Admin Identity". Upon first startup, NiFi will use that "Initial Admin Identity" to setup that user and grant them the "Access Policies" needed to administer that NiFi instance/cluster. That identity will be able to log in and add new users and grant "Access Policies" to those users. The default "Access Policies" that are given to that "Initial Admin Identity" include: NiFi File Based Policies: Ranger based Policies: view the UI view the user interface /flow view the controller access the controller (view) /controller (read) modify the controller access the controller (modify) /controller (write) view the users/groups access users/user groups (view) /tenants (read) modify the users/groups access users/user groups (modify) /tenants (write) view policies access all policies (view) /policies (read) modify policies access all policies (modify) /policies (write) Granting these same "Access Policies" to other users you have added will affectively make them an Admin as well. Thanks, Matt

MattWho · ‎01-30-2017

@Saminathan A The PutSQL processor expects that each FlowFile contains a single SQL statement and does not support multiple insert statements as you have tried above. You can have the GetFile Processor route its success relationship twice with each success going to its own ReplaceText processor. Each ReplaceText processor is then configured to create either the "table_a" or "table_b" insert statement. The success from both ReplaceText processors could then be routed to the same PutSQL processor. Thanks, Matt

MattWho · ‎01-30-2017

@Joshua Adeleke The message you are seeing here indicates that your NiFi instance has been setup as a cluster (possibly a 1 node cluster). NiFi cluster configurations require zookeeper in order handle cluster coordinator elections and cluster state management. If you truly only want a standalone NiFi installation with no dependency on these things, you need to make sure the following property in your nifi.properties file is set to false: nifi.cluster.is.node=false Even as a single node NIFi cluster, if zookeeper (internal or external) was setup, the election should complete eventually and the node will then become accessible. How long it takes for the cluster coordinator to get elected is controlled by the following lines in your nifi.properties file: nifi.cluster.flow.election.max.wait.time=5 mins nifi.cluster.flow.election.max.candidates= The above shows the defaults. If the max.candidates is left blank (Normally set to the number of nodes in your NiFi cluster), the election process will take the full 5 minutes to complete before the UI will become available. If max.candidates is set, then election will complete if either all nodes check in or 5 minutes (whichever occurs first). Thanks, Matt

MattWho · ‎01-27-2017

With HDF 2.x, Ambari can be used to deploy a NiFi cluster. Lets say you deployed a 2 node cluster and want to go back at a later time and add an additional NiFi node to the cluster. While the process is very straight forward when your NiFi cluster has been setup non-secure (http), the same is not true if your existing NiFi cluster has been secured (https). Below you will see an existing 2 node secured NiFi cluster that was installed via Ambari: STEP 1: Add new host through Ambari. You can skip this step if the host you want to install the additional NiFi node on is already managed by your Ambari. STEP 2: Under "Hosts" in Ambari click on the host form the list where you want to install the new NiFi node. The NiFi component will be in a "stopped" state after it is installed on this new host. *** DO NOT START NIFI YET ON NEW HOST OR IT WILL FAIL TO JOIN CLUSTER. *** STEP 3: (This step only applies if NiFi's file based authorizer is being used) Before starting this new node we need to clear out some NiFi Configs. This step is necessary because of how the NiFi application starts. When NiFi starts it looks for the existence of a users.xml and authorizations.xml files. If they do not exist, it uses the configured "Initial Admin Identity" and "Node identities (1,2,3, etc...)" to build the users.xml and authorizations.xml files. This causes a problem because your existing clusters users.xml and authorizations.xml files likely contain many more entires by now. Any mismatches in these files will prevent a node from being able to join the cluster. If these configurations are not present, the new node will grab them from the cluster it joins. Below shows what configs need to be cleared in NiFi: *Note: Another option is to simply copy the users.xml and authorizations.xml files from an existing cluster node to the new node before starting the new node. STEP 4: (Do this step if using Ambari metrics) When a new node is added by Ambari and Ambari metrics are also enabled, Ambari will create a flow.xml.gz file that contains just the ambari reporting task. Later when this node tries to join the cluster, the flow.xml.gz files between this new node and the cluster will not match. This mis-match will trigger the new node to fail to join cluster and shut back down. In order to avoid this problem the flow.xml.gz file must be copied from one of the cluster's existing nodes to this new node. STEP 5: Start NiFi on this new node. After the node has started, it should successfully join your existing cluster. If it fails, the nifi-app.log will explain why, but will likely be related to one of the above configs not being cleared out causing the users.xml and authorizations.xml files to get generated rather then inherited from the cluster. If that is the case you will need to fix the configs and delete those files manually before restarting the node again. STEP 6: While you cluster is now up and running with the additional node, but you will notice you cannot open the UI of that new node without getting an untrusted proxy error screen. You will however still be able to access your other two node's UIs. So we need to authorize this new node in your cluster. A. If NiFi handles your authorizations, follow this procedure: 1. Log in to the UI of one of the original cluster nodes. The "proxy user requests" access policies is needed to allow users to access the UI of your nodes. NOTE: There may be additional component level access policies (such as "view the data" and "modify the data") you may also want to authorize this new node for. B. If Ranger handles your NiFI authorizations, follow this procedure: 1. Access the Ranger UI: 2. Click Save to create this new user for your new node. Username MUST match exactly with the DN displayed in the untrusted proxy error screen. 3. Access the NiFi service Manager in Ranger and authorize your new node to your existing access policies as needed: You should now have a full functional new node added to your pre-existing secured NiFi cluster that was deployed/installed via Ambari.

Member Since	‎07-30-2019 10:41 AM
Last Visited
Posts	3,021
Kudos received	1494

Cloudera Community

Re: Unable to upload process group through REST A...

Re: NIFI Best Practice - Rest API endpoints

Re: Error while trying to upgrade nifi-registry fr...

Re: Nifi counters just an empty list

Re: how create user and add to nifi groups using n...

Re: How to load data from local system file to HDF...

Re: Listsftp taking a long time,

Re: How to identify source processor for failed Ni...

Re: Issue with NiFi

Re: Nifi Admin Access.

Re: Nifi Admin Access.

Re: Nifi Admin Access.

Re: How to execute multiple SQL query in NIFI PUT ...

Re: Issue with NiFi

HDF 2.x - Adding a new NiFi Node to an existing se...