question Re: Unable to read remote file in nifi by using minifi in Support Questions

Unable to read remote file in nifi by using minifi

Sartha — Mon, 08 Jan 2024 08:47:19 GMT

Hi Team, hoping for the best that someone will help me out from this issue ASAP.

I am working with nifi in local machine from couple of months to develop some jobs.now as part of developing one job I need to read remote file in nifi by using minifi,as like tailfile processor how it reads nifi log files continuously I have to read remote log files.in the process of finding solution I seen that by using minifi we can do that.so firstly I installed minifi remote server because I have to read remote log files by providing local machine path of outputfolder those log files will place in that path.

After installing minifi in remote server.unzippped it and tried to start the services by using ./minifi sh start .now I could see services are started and next firstly tried to take and input port in nifi UI and connected that with putfile processor by providing output path and provided connection for input and putfile .nextly taken a processor group and inside that processor group taken tailfile processor and provided the tailing mode as single file and files to tail as "logs\nifi-app.log".and next taken a remote process group(RPG) and in the url given as "http://localhost:8443/nifi" which is localhost nifi url and port number.and provided connection to tailfile processor and RPG.

Now that process group we have to Select and convert into template and again we have to download that template by using converter toolkit we have to convert file from XML to YML .I was unable to download and use converter toolkit to convert file. i tried to convert manually but that does not worked.

Regards,

Sartha

Re: Unable to read remote file in nifi by using minifi

VidyaSargur — Mon, 08 Jan 2024 15:30:21 GMT

@Sartha, Welcome to our community! To help you get the best possible answer, I have tagged our NiFi experts @cotopaul @SAMSAL @MattWho who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Unable to read remote file in nifi by using minifi

MattWho — Mon, 08 Jan 2024 17:13:58 GMT

@Sartha

Let me make sure I am clear on what you have built.

MiNiFI:
1. You have deployed MiNiFi on a remote server. You built a dataflow on a NiFi install consisting of a TailFile processor feeding a Remote ProcessGroup. That flow was converted in to a yml and used in your deployed MiNifi.
2. The TailFile has been configured to tail some log file(s) on the remote server. I see you said you configured it to tail nifi-app.log; however, MiNiFi does not have a nifi-app.log?
3. The Remote Process Group (RPG) will be used to send the FlowFiles generated by the tailFile processor to your NiFi instance over the network. I see you configured the RPG with an http url. Is your NiFi install running over http or https? Out-of-the-box apache NiFi runs secured over https. So did you remove that security so it is running un-secure over http? If not the RPG is not going to be able to connect to your NiFi. If you want to keep your NiFi secured, you will need to configure your MiNiFi with a proper keystore and truststore that can be used to facilitate a mutual TLS connection with your secured NiFi. If MiNiFi is on a different server then your NiFi, "localhost" is not going to work. If it is on same server, what is the point of using MiNiFi?

NiFi:
1. Is your NiFi a multi-node cluster setup or single standalone instance? If it is a single standalone instance you would probably be better off building a different flow between your MiNiFi and NiFi. (MiNiFi: TailFile --> PostHTTP or InvokeHTTP. NiFI: ListenHTTP --> other processors). If it is a NiFi cluster stick with RPG as it will distribute FlowFiles across all nodes. NOTE: With the postHTTP or InvokeHTTP to ListenHTTP dataflow, security is handled directly via the configuration of those processors, so it can be used without needing to care if MiNiFi and/or NiFi is secured or not.
2. If your NiFi is secure, what NiFi authorizer are you using? Out-of-the-box Nifi uses a Single-User authentication provider and Single-user-Authorizer. These out of the providers do not support setting up additional clients/users authorizations so will not support RPGs. So first you would to need to change to a different authorizer like FIle-authorizer. Secured communication between MiNiFi RPG and NiFi's Remote Input Port will require authentication (handle via mutual TLS exchange) and authorization (handled by NiFi authorizer). There are several authorization you would need to setup for your MiNiFi client before sending FlowFiles securely to NiFi will be possible (retrieve site-to-site details and receive data via site-to-site).

You did not share what you are seeing in the MiNiFi logs or NiFi logs when this dataflow is running to understand the issue(s) you may be having.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Unable to read remote file in nifi by using minifi

Sartha — Tue, 09 Jan 2024 10:01:41 GMT

Thanks @MattWho for your response

2. The TailFile has been configured to tail some log file(s) on the remote server. I see you said you configured it to tail nifi-app.log; however, MiNiFi does not have a nifi-app.log?_-------as you said is correct minifi does not have nifi-app.log,here my case is I want to read log files presented in nifi not from minifi so provided file to tail path as nifi-app.log.
3. The Remote Process Group (RPG) will be used to send the FlowFiles generated by the tailFile processor to your NiFi instance over the network. I see you configured the RPG with an http url. Is your NiFi install running over http or https? Out-of-the-box apache NiFi runs secured over https. So did you remove that security so it is running un-secure over http? If not the RPG is not going to be able to connect to your NiFi. If you want to keep your NiFi secured, you will need to configure your MiNiFi with a proper keystore and truststore that can be used to facilitate a mutual TLS connection with your secured NiFi. If MiNiFi is on a different server then your NiFi, "localhost" is not going to work. If it is on same server, what is the point of using MiNiFi?_-------my nifi install is running over http.
---->I followed Apache MiNiFi administration Guide ,but still it doesn't worked

1.Nifi is running on localhost and minifi is installed on server .

2.my task is how to read remote files in Nifi??----- could you please provide me solution for this it would be helpful

Even my task is also similar as you provided solution earlier to one question "Pull logs from a remote server to nifi"

Please provide me clear steps of processor flow with configuration of every processor in detail

Re: Unable to read remote file in nifi by using minifi

MattWho — Tue, 09 Jan 2024 15:25:37 GMT

@Sartha
I am very confused on your use case.

here my case is I want to read log files presented in nifi not from minifi so provided file to tail path as nifi-app.log.

If you are just trying to tail the nifi-app.log, what do you need MiNiFi for here...

Let step back and just get a clear understanding of yoru use case. What are you trying to accomplish? Remove NiFi from the equation and define your use case.
i.e - I have server1 and server2, I have some service running on server 2 writing log files. I want to read those logs as they are being written to and send that log output to server 1. etc... As more detail the better.

As far as processors go, there is detailed documentation for each one.

https://nifi.apache.org/docs/nifi-docs/

Understanding which NiFi components you need to use starts with a detailed use case.
I am still not clear on your use case and where MiNiFi fist in to it if you are trying to tail the nifi-app.log. Tailing the nifi-app.log implies you have nifi runing on the server where the nifi-app.log is being written to, thus you could just use NiFi to tail its own nifi-app.log. But why do this? What are you trying to capture from nifi-app.log? What are planning to so with this log output once you tail it?

Thank you,

Matt

Re: Unable to read remote file in nifi by using minifi

Sartha — Tue, 09 Jan 2024 16:16:11 GMT

1. My issue is "I have to read remote file in nifi".--- for this query the solution as far i know is by using minifi we can read remote file.if yes could you please provide me the steps of data flow of how to read .

2.as I have nifi installed in local ,unable to install nifi in server ,I want to read log files from server by using minifi can we read it.if yes can you provide me the nifi processors data flow of how to read it by using tailfile processor.

Thanks for your reply @MattWho

Re: Unable to read remote file in nifi by using minifi

MattWho — Wed, 10 Jan 2024 21:23:46 GMT

@Sartha

I think i may now be understanding the confusion here. MiNiFi does not alter in any way the functionality of the components (NiFi processor, controller services, etc). They all function exactly as they do in NiFi.

As I understand it now, your use case is to tail a file located on a different server from where NiFi or MiNiFi is installed. Correct?

Neither NiFi or MiNiFi can tail a log file located on another server. The TailFile processor can only tail files local to the same server on which NiFi or MiNiFi is running. So MiNiFi would need to be installed on the server where the log file exists in order to tail that file. MiNiFi installed on that server would then be able to transfer the generated FlowFile over the network to your NiFi running on your local machine.

MiNiFi is essentially NiFi without the UI (few other differences).

Either NiFi or MiNiFi would need to be installed in that server.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Unable to read remote file in nifi by using minifi

Sartha — Tue, 16 Jan 2024 05:53:55 GMT

@MattWho

As I understand it now, your use case is to tail a file located on a different server from where NiFi or MiNiFi is installed. Correct? _--------yes you are correct ,MiNiFi is installed in server where NiFi is running in my local machine.

Neither NiFi or MiNiFi can tail a log file located on another server. The TailFile processor can only tail files local to the same server on which NiFi or MiNiFi is running. So MiNiFi would need to be installed on the server where the log file exists in order to tail that file. MiNiFi installed on that server would then be able to transfer the generated Flow File over the network to your NiFi running on your local machine_---------- what you said here is exactly I required.

Installed MiNiFi in server and NiFi in local machine I want to tail files from server to local .

So please provide me the clear steps of how to tail files from server to local .

Thank you,

Sartha.

Re: Unable to read remote file in nifi by using minifi

MattWho — Wed, 17 Jan 2024 14:58:09 GMT

@Sartha

Your MiNiFi needs a simple dataflow that consists of a TailFile processor and then either a Remote Process Group (RPG) (recommended if your NiFi is a multi-node cluster, but can be used with single instance NiFi as well.) or a PostHTTP processor (Use if NiFi a single instance and not a cluster). Neither the RPG not the PostHTTP can be configured with a target URL with "localhost". Localhost would be the MiNiFi server. It needs to be the hostname of the server where your NiFi is running. Make sure if using RPG, you have configured the Site-to-Site properties in the nifi.properties file.

Your NiFi would need a Remote Input Port (this is what your MiNiFi RPG will transmit FlowFiles to) or a ListenHTTP processor (if you used PostHTTP on your MiNiFi, this is what it can be configured to send to). The outbound connection from either of these components needs to feed whatever downstream processors you need to do within NiFi.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Unable to read remote file in nifi by using minifi

Sartha — Thu, 18 Jan 2024 04:53:27 GMT

@MattWho

As per your guidance firstly I have taken TailFile processor and configured as below

Next taken PostHTTP processor with below Configurations,in the PostHTTP url used is server hostname and port number because MiNiFi is running on this server and NiFi is running on localhost .correct me if I am wrong here.

Nextly I have taken ListenHTTP and configured the listening port as the server port number where MiNiFi is running which is provided in PostHTTP.as below

Eneded the data flow here as per your guidance ,when I ran TailFile processor it picked the log file and next when started PostHTTP processor it throwed below error

Could you guide me whether my data flow is correct,if it is wrong what needs to be changed to execute the remote file.

Thanks

Sartha

Re: Unable to read remote file in nifi by using minifi

MattWho — Thu, 18 Jan 2024 14:54:35 GMT

@Sartha

localhost means same host on which the service is running, so you can't use localhost in a postHTTP processor running on MiNiFi (host B) and expect it to sent FlowFiles to ListenHTTP running on another host (host A).

The postHTTP is going to transmit FlowFiles to another host, so why would you configure postHTTP running on your MiNiFi Host with the MiNiFi IP and port?

in the PostHTTP url used is server hostname and port number because MiNiFi is running on this server

Maybe I am just misunderstanding what you are saying in above statement.

It also does not appear that the URL is correct. It appears to be missing the "Base path" (default contentListener) as described in the processors documentation.

To help you out, I build this flow so I can share screenshots of how it is configured for you:

Here is PostHTTP configuration which would be used on MiNiFi (host B):

Here is configuration of ListenHTTP used on the NiFi instance (host A) that will receive FlowFiles from above PostHTTP on your MiNiFi (host B):

In my configuration I used port 9933, but you can use any unused non privileged port available on your NiFi instance (host A).

I am also still confused by your screenshots with regards to below:

Image 1 of the tailFile processor. This the processor that is feeding the PostHTTP processor on the MiNiFi instance installed on (host B), correct? You configured it to tail a nifi-app.log (which is not configured with the absolute path) which is generated by NiFi (host A). As I mentioned before, the tailFile processor can not tail a file located on a different host form where the tailFile is executing).

Image 3 showing dataflow. Why is ListenHTTP success relationship connected to PostHTTP?

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Unable to read remote file in nifi by using minifi

Sartha — Fri, 19 Jan 2024 13:02:46 GMT

@MattWho

As per the above screenshots I have configured tailfile processor as below

Nextly connected tailfile with PostHTTP and configured as below as provided hostname of nifi and port of nifi url

Still it is not working.could you please provide any link that I can go with the data flow for success relationship that to read remote file in nifi using minifi.

Thanks for all the help

Thanks,

Sartha

Re: Unable to read remote file in nifi by using minifi

MattWho — Fri, 19 Jan 2024 18:15:46 GMT

@Sartha

You are still not using the correct URL in your postHTTP as I stated in last response:
It needs to be based on what you shared in yoru last response:

http://10.73.121.84:5026/contentListner

The ListenHTTP processor defines the "Base Path" on which it is listening for inbound connections. The default value is "contentListener". So the PostHTTP needs to post to that Base Path.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Unable to read remote file in nifi by using minifi

VidyaSargur — Thu, 25 Jan 2024 08:47:48 GMT

@Sartha, Did @MattWho's responses assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.

Re: Unable to read remote file in nifi by using minifi

Sartha — Thu, 01 Feb 2024 12:41:38 GMT

@MattWho

I followed the flow as per your guidance but still it doesn't worked.

Even tried of following as Administration Guide of Apache MINIFI.that wasn't worked.

Is there any other way that to configure this issue.

Thank you soo much for all the support.

Regards,

Sartha

Re: Unable to read remote file in nifi by using minifi

MattWho — Thu, 01 Feb 2024 14:44:54 GMT

@Sartha

I don't know how to respond to "I followed the flow as per your guidance but still it doesn't worked."

The flow I provided works.
What exceptions/errors are you encountering?
You have not provided much detail on what you are seeing exception/issue wise.

Thank you,
Matt