Skip to main content
ManishR
ManishR
Silver ⭐️⭐️
March 27, 2023
Question

How to find existing processor/files/integrations in Apache Nifi?

I am new to Apache NiFi and need to know how to find a processor or processor group by its integration name, folder, or event name, as well as how to find the specific source and destination file details within any given processor or processor group.

6 replies

ManishR
ManishRAuthor
Silver ⭐️⭐️
March 27, 2023

How to find source and target files in existing processor in apache nifi?

Translate
cotopaul
cotopaul
Platinum ⭐️
March 27, 2023

@ManishRcan you please describe your problem a little bit better? What did you try to do? Why did it not work? What should have been the expected outcome? What are you actually trying to achieve?

 

What you have provided so far is quite general like " which is the best car in the world " / " how can I find the best milk " / "who came first? the chicken or the egg".


As you described your question, the answer is: you can find the source files in the source and the target files in the target, where the source is what you have defined your source to be and the target is the target you have set.

Translate
cotopaul
cotopaul
Platinum ⭐️
March 27, 2023

@ManishRcan you please describe your problem a little bit better? What did you try to do? Why did it not work? What should have been the expected outcome? What are you actually trying to achieve?

 

What you have provided so far is quite general like " which is the best car in the world " / " how can I find the best milk " / "who came first? the chicken or the egg".


As you described your question, the answer is: your existing processors are present on the canvas. The files are in the source, the target or the queues between the processors, whereas the integrations are the integrations which you have constructed or used so far.

Translate
ManishR
ManishRAuthor
Silver ⭐️⭐️
March 27, 2023

Hi,

I am very new in Apache nifi where my question is as follow:-

1. How to find the processor or processor group by Integration Name/Folder/Event Name?
Ans - As per my understanding, by simply type the name of files name in search bar in the top right of screen

Please elaborate your expert advice on this.

2. How to find the source and destination file details in any of the processor or processor group?

Translate
cotopaul
cotopaul
Platinum ⭐️
March 27, 2023

Hi @ManishR,


1. How to find the processor or processor group by Integration Name/Folder/Event Name?

What do you mean by integration name, event name and folder? Folder I assume you refer to Processor Group (external or normal) but integration name and event name are unknown to me. If you are trying it identify Processors, Processor Group, Queues or any other objects from within your Canvas Board, you mostly have two options: 1-you either use the NiFi REST Api (https://nifi.apache.org/docs/nifi-docs/rest-api/index.html ) or 2-you use the search bar in the top right of the screen.

2. How to find the source and destination file details in any of the processor or processor group?
What do you mean with this question? In NiFi there are a series of processors which can be used to retrieve SOURCE files (ListSFTP/FetchSFTP/GetSFTP,GenerateFlowFile,InvokeHTTP,ConsumeKafka,ExecuteSQLRecord,etc) and there is a series of processors used to save your data in a TARGET location (like PutFile,PublishKafka, PutGCSObject, PutHDFS, PutDatabaseRecord,PutS3Object, etc). If you are to speak about a certain processor, the source data is within the queue linked to the specific processor, whereas the target data is located within the queue linked from your specific processor to another processor. Yet again, your question is quite general and ambiguous to get a specific answer.

 

Sorry but I do not understand what exactly you are trying to achieve so if you require a more technical and more elaborate answer, you will have to come with a more detailed questions, based on a specific use case - maybe even your NiFi Flow.

Translate
    MattWho
    Titanium ⭐️⭐️⭐️
    MattWho
    Titanium ⭐️⭐️⭐️
    March 27, 2023

    @ManishR 

    NiFi offers many components (processors, controller services, reporting tasks, etc) that can be used to construct a flow based program on the NiFi canvas (Referred to as a NiFi dataflow).  While this list of default available components may be different depending on the release of NiFi being used, NiFi has embedded documentation found under help within the NiFi UI that shows all components available in that installed release.

    Apache NiFi also publishes the same info for the most current released version here:
    https://nifi.apache.org/docs/nifi-docs/
    Selecting a component from the documentation with open a description of the component and all list configurable properties.

    Building a dataflow on the NiFi canvas consist of dragging and dropping new component processors to the canvas.  You can then drag connection between these components to construct your end-to-end dataflow.  There are 100s of component processors available out of the box and even more that you can download and add to your NiFi from the apache community.

    Once a dataflow is built and configured, starting those components would result in the creation of FlowFile (for testing, you can add a GenerateFlowFile processor that generates a FlowFile rather then ingesting content from an external source like the local file system, kafka, DB, etc.

    As each component executes against a FlowFile, that FlowFile is routed to one of the available relationships the particular processor offers.  These relationships would be assigned to one of the connection exiting the processor and connecting to another downstream processor.

    The following Apache NiFi docs explain how to build a dtaflow:
    https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#building-dataflow

    This covers how to search for a component in yoru dataflow(s):
    https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#search

     

    Then when it comes to looking at the detailed lineage of an individual FlowFile, you can use NiFi's data Provenance for that.  Individual processor components generate provenance events as they execute on FlowFile (create, route, drop, etc...). You can look at the entire lineage from create to drop of a FlowFile (assuming you configure NiFi provenance with enough storage to store all the lineage).  BY default NiFI is configured to only use 10GB for Provenance and only store Provenance for 24 hours, but this can be configured in the nifi.properties file. 

    You can write click on NiFi processor component in your dataflow and Select data provenance from the pop-up context menu.  This will open a provenance search query result set that show FlowFile that traversed the component.  You can select one and even expand the lineage of that select component.  The lineage of a FlowFile will show all events associated to that FlowFile created by the processor components that FlowFile traversed. 

    This covers how to use NiFi's Data Provenance:
    https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data_provenance

    If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

    Thank you,

    Matt

    Translate
      Terms & ConditionsAccessibility statement