@Hadoop User The ExtractText processor will extract the text that matches your regex and assign it to an attribute matching the property name on the FlowFile. The content of the FlowFile remains unchanged. Then you update a FlowFiles Attribute and finally use PutHDFS to write the content (which at this time you have not changed at all) to HDFS. If your intent is to write the modified string to HDFS, you need to update the actual content of the FlowFile and nit just create and modify attributes. For that use case, you would want to use ReplaceText processor instead. You would configure ReplaceText similar to the following: The above will result in the actual content of the FlowFile being changed to: [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] Thanks, Matt ... View more

@Foivos A The banner is a NiFi core feature and is not tied in anyway to the dataflows you select or build on your canvas. You are correct that the best approach for identifying which dataflows on a single canvas are designated dev, test, or production is through the use of "labels". In a secure NiFi setup, you can use NiFi granular multi-tenancy user authorization to control what components a user can interact with an view. If you use labels, you should set a policy allowing all user to view that specific component, so even if they are not authorized to access the labeled components, they will be able to see why via the label text. Thanks, Matt ... View more

@Alvin Jin I am not familiar with what K8 is... I suggest starting a new question rather then adding to this existing question so that it gets full exposure to the community. I would also suggest providing as much detail as you can about the use case for your question. Thanks, Matt ... View more

@Sanaz Janbakhsh You should look in your nifi-user.log file. When you attempt to perform the "List Queue", What log entries do you see? Unfortunately, the attachment you provided does not tell me much since it does not include the "NiFi Resource Identifier" or users assigned to each of those policy names. Did you create a policy that uses the "NiFi Resource Identifier" of "/data/*" and assign your single node's DN to it? Another place you could check is the Ranger Audit. Filter on Result:Denied and try to list your queue. Do you see any Denied audit lines for any "/data" resource similar to the below: The above is the result of me trying to perform list queue for my user "nifiuser1" when the node has not been properly authorized to READ the data. As you can see Ranger is reporting there is no policy authorizing my node's DN for the resource listed. The UUID in the resource is the UUID of the processor which owns the connection I was trying to list. Once I added my policy that gives the node's DN READ/WRITE to "/data/*", i was able to list and empty this queue. Thanks, Matt ... View more

@Richard Corfield The Provenance repo has not impact on the functionality of your dataflow. All the FlowFiles currently queued in your dataflow are directly tied to the content in the FlowFile and Content repositories. The data stored in your provenance repository has a configured lifespan (default 24 hours or 1 GB disk usage) and should be cleared automatically based on those threshold by NiFi. ... View more

@Jobin George The issue here is caused by the following.... 1. Ambari metrics have been enabled. 2. On start of a NiFi Node, If Ambari detects a flow.xml.gz fiel does not exist, it creates a flow.xml.gz that contains only the AmbariReportingTask to support the enabling of Ambari metrics from this NiFi. 3. Then NiFi is started and NiFi's normal startup procedure occurs. During that process NiFi detects the flow.xml.gz on this new node does not match the flow.xml.gz on the cluster. Node will shut back down. Aside from just manually copying the flow.xml.gz from an existing cluster node, another workaround is to make sure the flow.xml.gz fiel is not there and start the new node via NiFi's command line start manually to bypass the Ambari flow.xml.gz file generation. Thanks, Matt ... View more

@AnjiReddy Anumolu Just to add a little more detail to the above response from @zblanco. When NiFi ingest data, that data is turned in to NiFi FlowFiles. A NiFi FlowFile consists of Attributes (Metadata) about the actual data and the physical data. The FlowFile metadata is stored in the FlowFile repository as well as JVM heap memory for faster performance. The FlowFile Attributes includes things like filename, ingest time, lineage age, filesize, what connection the FlowFile currently resides in dataflow, any user defined metadata, or processor added metadata, etc....). The physical bytes that make up the actual data content is written to claims within the NiFi content repository. A claim can contain the bytes for 1 to many ingest data files. For more info on the content repository and how claims work, see the following link: https://community.hortonworks.com/articles/82308/understanding-how-nifis-content-repository-archivi.html Thanks, Matt ... View more

NiFi 1.2.0 added some alignment tools, but nothing is available NiFi 1.1.0. IN NiFi 1.20 you can select multiple components, right click to open context menu and select to align them vertically or horizontally. ----> These tools can't be used to align an entire canvas at once, but it gives you the ability to easily line up single rows or columns of processor components. In the future try to keep unrelated questions to different Hortonworks Community Connection (HCC) posts. Other HCC contributors are likely to miss questions asked within the context of an answer to another question. Thanks, Matt ... View more