Silver ⭐️⭐️⭐️

Question

Unable to read remote file in nifi by using minifi

15 replies
9494 views
Translate All

Hi Team, hoping for the best that someone will help me out from this issue ASAP.

I am working with nifi in local machine from couple of months to develop some jobs.now as part of developing one job I need to read remote file in nifi by using minifi,as like tailfile processor how it reads nifi log files continuously I have to read remote log files.in the process of finding solution I seen that by using minifi we can do that.so firstly I installed minifi remote server because I have to read remote log files by providing local machine path of outputfolder those log files will place in that path.

After installing minifi in remote server.unzippped it and tried to start the services by using ./minifi sh start .now I could see services are started and next firstly tried to take and input port in nifi UI and connected that with putfile processor by providing output path and provided connection for input and putfile .nextly taken a processor group and inside that processor group taken tailfile processor and provided the tailing mode as single file and files to tail as "logs\nifi-app.log".and next taken a remote process group(RPG) and in the url given as "http://localhost:8443/nifi" which is localhost nifi url and port number.and provided connection to tailfile processor and RPG.

Now that process group we have to Select and convert into template and again we have to download that template by using converter toolkit we have to convert file from XML to YML .I was unable to download and use converter toolkit to convert file. i tried to convert manually but that does not worked.

Regards,

Sartha

Apache MiNiFi

VidyaSargur

Community Manager

@Sartha, Welcome to our community! To help you get the best possible answer, I have tagged our NiFi experts @cotopaul @SAMSAL @MattWho who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Translate

Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.

MattWho

Titanium ⭐️⭐️⭐️

@Sartha

Let me make sure I am clear on what you have built.

MiNiFI:
1. You have deployed MiNiFi on a remote server. You built a dataflow on a NiFi install consisting of a TailFile processor feeding a Remote ProcessGroup. That flow was converted in to a yml and used in your deployed MiNifi.
2. The TailFile has been configured to tail some log file(s) on the remote server. I see you said you configured it to tail nifi-app.log; however, MiNiFi does not have a nifi-app.log?
3. The Remote Process Group (RPG) will be used to send the FlowFiles generated by the tailFile processor to your NiFi instance over the network. I see you configured the RPG with an http url. Is your NiFi install running over http or https? Out-of-the-box apache NiFi runs secured over https. So did you remove that security so it is running un-secure over http? If not the RPG is not going to be able to connect to your NiFi. If you want to keep your NiFi secured, you will need to configure your MiNiFi with a proper keystore and truststore that can be used to facilitate a mutual TLS connection with your secured NiFi. If MiNiFi is on a different server then your NiFi, "localhost" is not going to work. If it is on same server, what is the point of using MiNiFi?

NiFi:
1. Is your NiFi a multi-node cluster setup or single standalone instance? If it is a single standalone instance you would probably be better off building a different flow between your MiNiFi and NiFi. (MiNiFi: TailFile --> PostHTTP or InvokeHTTP. NiFI: ListenHTTP --> other processors). If it is a NiFi cluster stick with RPG as it will distribute FlowFiles across all nodes. NOTE: With the postHTTP or InvokeHTTP to ListenHTTP dataflow, security is handled directly via the configuration of those processors, so it can be used without needing to care if MiNiFi and/or NiFi is secured or not.
2. If your NiFi is secure, what NiFi authorizer are you using? Out-of-the-box Nifi uses a Single-User authentication provider and Single-user-Authorizer. These out of the providers do not support setting up additional clients/users authorizations so will not support RPGs. So first you would to need to change to a different authorizer like FIle-authorizer. Secured communication between MiNiFi RPG and NiFi's Remote Input Port will require authentication (handle via mutual TLS exchange) and authorization (handled by NiFi authorizer). There are several authorization you would need to setup for your MiNiFi client before sending FlowFiles securely to NiFi will be possible (retrieve site-to-site details and receive data via site-to-site).

You did not share what you are seeing in the MiNiFi logs or NiFi logs when this dataflow is running to understand the issue(s) you may be having.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

SarthaAuthor

Silver ⭐️⭐️⭐️

Thanks @MattWho for your response

2. The TailFile has been configured to tail some log file(s) on the remote server. I see you said you configured it to tail nifi-app.log; however, MiNiFi does not have a nifi-app.log?_-------as you said is correct minifi does not have nifi-app.log,here my case is I want to read log files presented in nifi not from minifi so provided file to tail path as nifi-app.log.
3. The Remote Process Group (RPG) will be used to send the FlowFiles generated by the tailFile processor to your NiFi instance over the network. I see you configured the RPG with an http url. Is your NiFi install running over http or https? Out-of-the-box apache NiFi runs secured over https. So did you remove that security so it is running un-secure over http? If not the RPG is not going to be able to connect to your NiFi. If you want to keep your NiFi secured, you will need to configure your MiNiFi with a proper keystore and truststore that can be used to facilitate a mutual TLS connection with your secured NiFi. If MiNiFi is on a different server then your NiFi, "localhost" is not going to work. If it is on same server, what is the point of using MiNiFi?_-------my nifi install is running over http.
---->I followed Apache MiNiFi administration Guide ,but still it doesn't worked

1.Nifi is running on localhost and minifi is installed on server .

2.my task is how to read remote files in Nifi??----- could you please provide me solution for this it would be helpful

Even my task is also similar as you provided solution earlier to one question "Pull logs from a remote server to nifi"

Please provide me clear steps of processor flow with configuration of every processor in detail

Translate

MattWho

Titanium ⭐️⭐️⭐️

@Sartha
I am very confused on your use case.

here my case is I want to read log files presented in nifi not from minifi so provided file to tail path as nifi-app.log.

If you are just trying to tail the nifi-app.log, what do you need MiNiFi for here...

Let step back and just get a clear understanding of yoru use case. What are you trying to accomplish? Remove NiFi from the equation and define your use case.
i.e - I have server1 and server2, I have some service running on server 2 writing log files. I want to read those logs as they are being written to and send that log output to server 1. etc... As more detail the better.

As far as processors go, there is detailed documentation for each one.

https://nifi.apache.org/docs/nifi-docs/

Understanding which NiFi components you need to use starts with a detailed use case.
I am still not clear on your use case and where MiNiFi fist in to it if you are trying to tail the nifi-app.log. Tailing the nifi-app.log implies you have nifi runing on the server where the nifi-app.log is being written to, thus you could just use NiFi to tail its own nifi-app.log. But why do this? What are you trying to capture from nifi-app.log? What are planning to so with this log output once you tail it?

Thank you,

Matt

Translate

MattWho

Titanium ⭐️⭐️⭐️

@Sartha

localhost means same host on which the service is running, so you can't use localhost in a postHTTP processor running on MiNiFi (host B) and expect it to sent FlowFiles to ListenHTTP running on another host (host A).

The postHTTP is going to transmit FlowFiles to another host, so why would you configure postHTTP running on your MiNiFi Host with the MiNiFi IP and port?

in the PostHTTP url used is server hostname and port number because MiNiFi is running on this server

Maybe I am just misunderstanding what you are saying in above statement.

It also does not appear that the URL is correct. It appears to be missing the "Base path" (default contentListener) as described in the processors documentation.

To help you out, I build this flow so I can share screenshots of how it is configured for you:

Here is PostHTTP configuration which would be used on MiNiFi (host B):

Here is configuration of ListenHTTP used on the NiFi instance (host A) that will receive FlowFiles from above PostHTTP on your MiNiFi (host B):

In my configuration I used port 9933, but you can use any unused non privileged port available on your NiFi instance (host A).

I am also still confused by your screenshots with regards to below:

Image 1 of the tailFile processor. This the processor that is feeding the PostHTTP processor on the MiNiFi instance installed on (host B), correct? You configured it to tail a nifi-app.log (which is not configured with the absolute path) which is generated by NiFi (host A). As I mentioned before, the tailFile processor can not tail a file located on a different host form where the tailFile is executing).

Image 3 showing dataflow. Why is ListenHTTP success relationship connected to PostHTTP?

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

SarthaAuthor

Silver ⭐️⭐️⭐️

@MattWho

As per the above screenshots I have configured tailfile processor as below

IMG20240119181010~2.jpg

Nextly connected tailfile with PostHTTP and configured as below as provided hostname of nifi and port of nifi url

IMG20240119181101~2.jpg

Still it is not working.could you please provide any link that I can go with the data flow for success relationship that to read remote file in nifi using minifi.

Thanks for all the help

Thanks,

Sartha

Translate

MattWho

Titanium ⭐️⭐️⭐️

@Sartha

You are still not using the correct URL in your postHTTP as I stated in last response:
It needs to be based on what you shared in yoru last response:

http://10.73.121.84:5026/contentListner

The ListenHTTP processor defines the "Base Path" on which it is listening for inbound connections. The default value is "contentListener". So the PostHTTP needs to post to that Base Path.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

VidyaSargur

Community Manager

@Sartha, Did @MattWho's responses assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.

Translate

Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.