Unable to read remote file in nifi by using minifi

[#1]

Hi Team, hoping for the best that someone will help me out from this issue ASAP.

I am working with nifi in local machine from couple of months  to develop some jobs.now as part of developing one job I need to read remote file in nifi by using minifi,as like tailfile processor how it reads nifi log files continuously I have to read remote log files.in the process of finding solution  I seen that by using minifi we can do that.so firstly I installed minifi remote server because I have to read remote log files by providing local machine path of outputfolder those log files will place in that path.

After installing minifi in remote server.unzippped it and tried to start the services by using ./minifi sh start .now I could see services are started and next firstly tried to take and input port in nifi UI and connected that with putfile processor by providing output path and provided connection for input and putfile .nextly taken a processor group and inside that processor group taken tailfile processor and provided the tailing mode as single file and files to tail as "logs\nifi-app.log".and next taken a remote process group(RPG)  and in the url given as "http://localhost:8443/nifi" which is localhost nifi url and port number.and provided connection to tailfile processor and RPG.

Now that process group we have to Select and convert into template and again we have to download that template by using converter toolkit we have to convert file from XML to YML .I was unable to download and use converter toolkit to convert file. i tried to convert manually but that does not worked.

Regards,

Sartha

[#2]

@Sartha, Welcome to our community! To help you get the best possible answer, I have tagged our NiFi experts @cotopaul @SAMSAL @MattWho  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

@Sartha 

Let me make sure I am clear on what you have built.

MiNiFI:
1. You have deployed MiNiFi on a remote server.  You built a dataflow on a NiFi install consisting of a TailFile processor feeding a Remote ProcessGroup. That flow was converted in to a yml and used in your deployed MiNifi.
2. The TailFile has been configured to tail some log file(s) on the remote server.  I see you said you configured it to tail nifi-app.log; however, MiNiFi does not have a nifi-app.log?
3. The Remote Process Group (RPG) will be used to send the FlowFiles generated by the tailFile processor to your NiFi instance over the network.  I see you configured the RPG with an http url.  Is your NiFi install running over http or https?  Out-of-the-box apache NiFi runs secured over https.  So did you remove that security so it is running un-secure over http?  If not the RPG is not going to be able to connect to your NiFi.  If you want to keep your NiFi secured, you will need to configure your MiNiFi with a proper keystore and truststore that can be used to facilitate a mutual TLS connection with your secured NiFi.   If MiNiFi is on a different server then your NiFi, "localhost" is not going to work.  If it is on same server, what is the point of using MiNiFi?

NiFi:
1. Is your NiFi a  multi-node cluster setup or single standalone instance?  If it is a single standalone instance you would probably be better off building a different flow between your MiNiFi and NiFi. (MiNiFi: TailFile --> PostHTTP or InvokeHTTP.  NiFI: ListenHTTP --> other processors).  If it is a NiFi cluster stick with RPG as it will distribute FlowFiles across all nodes.  NOTE: With the postHTTP or InvokeHTTP to ListenHTTP dataflow, security is handled directly via the configuration of those processors, so it can be used without needing to care if MiNiFi and/or NiFi is secured or not.
2. If your NiFi is secure, what NiFi authorizer are you using? Out-of-the-box Nifi uses a Single-User authentication provider and Single-user-Authorizer.  These out of the providers do not support setting up additional clients/users authorizations so will not support RPGs.   So first you would to need to change to a different authorizer like FIle-authorizer.  Secured communication between MiNiFi RPG and NiFi's Remote Input Port will require authentication (handle via mutual TLS exchange)  and authorization (handled by NiFi authorizer).   There are several authorization you would need to setup for your MiNiFi client before sending FlowFiles securely to NiFi will be possible (retrieve site-to-site details and receive data via site-to-site).


You did not share what you are seeing in the MiNiFi logs or NiFi logs when this dataflow is running to understand the issue(s) you may be having.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#4]

Thanks @MattWho for your response

2. The TailFile has been configured to tail some log file(s) on the remote server.  I see you said you configured it to tail nifi-app.log; however, MiNiFi does not have a nifi-app.log?_-------as you said is correct minifi does not have nifi-app.log,here my case is I want to read log files presented in nifi not from minifi so provided file to tail path as nifi-app.log.
3. The Remote Process Group (RPG) will be used to send the FlowFiles generated by the tailFile processor to your NiFi instance over the network.  I see you configured the RPG with an http url.  Is your NiFi install running over http or https?  Out-of-the-box apache NiFi runs secured over https.  So did you remove that security so it is running un-secure over http?  If not the RPG is not going to be able to connect to your NiFi.  If you want to keep your NiFi secured, you will need to configure your MiNiFi with a proper keystore and truststore that can be used to facilitate a mutual TLS connection with your secured NiFi.   If MiNiFi is on a different server then your NiFi, "localhost" is not going to work.  If it is on same server, what is the point of using MiNiFi?_-------my nifi install is running over http.
---->I followed Apache MiNiFi administration Guide ,but still it doesn't worked

1.Nifi is running on localhost and minifi is installed on server .

2.my task is how to read remote files in Nifi??----- could you please provide me solution for this it would be helpful  

Even my task is also similar as you provided solution earlier to one question "Pull logs from a remote server to nifi"

Please provide me clear steps of processor flow with configuration of every processor in detail

[#5]

@Sartha 
I am very confused on your use case.  

here my case is I want to read log files presented in nifi not from minifi so provided file to tail path as nifi-app.log.

If you are just trying to tail the nifi-app.log, what do you need MiNiFi for here...

Let step back and just get a clear understanding of yoru use case.  What are you trying to accomplish?  Remove NiFi from the equation and define your use case.
i.e - I have server1 and server2, I have some service running on server 2 writing log files.  I want to read those logs as they are being written to and send that log output to server 1. etc...  As more detail the better.

As far as processors go, there is detailed documentation for each one.

• https://nifi.apache.org/docs/nifi-docs/

Understanding which NiFi components you need to use starts with a detailed use case.
I am still not clear on your use case and where MiNiFi fist in to it if you are trying to tail the nifi-app.log.  Tailing the nifi-app.log implies you have nifi runing on the server where the nifi-app.log is being written to,  thus you could just use NiFi to tail its own nifi-app.log.  But why do this?  What are you trying to capture from nifi-app.log?  What are planning to so with this log output once you tail it?

Thank you,

Matt

[#6]

1. My  issue is "I have to read remote file in nifi".---  for this query the solution as far i know is by using minifi we can read remote file.if yes could you please provide me the steps of data flow of how to read .

2.as I have nifi installed in local ,unable to install nifi in server ,I want to read  log files from server by using minifi can we read it.if yes can you provide me the nifi processors data flow of how to read it by using tailfile processor.

Thanks for your reply @MattWho 

.

[#7]

@Sartha 

I think i may now be understanding the confusion here. MiNiFi does not alter in any way the functionality of the components (NiFi processor, controller services, etc).  They all function exactly as they do in NiFi.

As I understand it now, your use case is to tail a file located on a different server from where NiFi or MiNiFi is installed. Correct?

Neither NiFi or MiNiFi can tail a log file located on another server.  The TailFile processor can only tail files local to the same server on which NiFi or MiNiFi is running. So MiNiFi would need to be installed on the server where the log file exists in order to tail that file.  MiNiFi installed on that server would then be able to transfer the generated FlowFile over the network to your NiFi running on your local machine.

MiNiFi is essentially NiFi without the UI (few other differences).

Either NiFi or MiNiFi would need to be installed in that server.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#8]

@MattWho

As I understand it now, your use case is to tail a file located on a different server from where NiFi or MiNiFi is installed. Correct? _--------yes you are correct ,MiNiFi is installed in server where NiFi is running in my local machine.

Neither NiFi or MiNiFi can tail a log file located on another server.  The TailFile processor can only tail files local to the same server on which NiFi or MiNiFi is running. So MiNiFi would need to be installed on the server where the log file exists in order to tail that file.  MiNiFi installed on that server would then be able to transfer the generated Flow File over the network to your NiFi running on your local machine_---------- what you said here is exactly I required. 

Installed MiNiFi in server and NiFi in local machine I want to tail files from server to local .

So please provide me the clear steps of how to tail files from server to local .

Thank you,

Sartha.

[#9]

@Sartha 

Your MiNiFi needs a simple dataflow that consists of a TailFile processor and then either a Remote Process Group (RPG)  (recommended if your NiFi is a multi-node cluster, but can be used with single instance NiFi as well.) or a PostHTTP processor (Use if NiFi a single instance and not a cluster).  Neither the RPG not the PostHTTP can be configured with a target URL with "localhost".  Localhost would be the MiNiFi server.  It needs to be the hostname of the server where your NiFi is running.  Make sure if using RPG, you have configured the Site-to-Site properties in the nifi.properties file.

Your NiFi would need a Remote Input Port (this is what your MiNiFi RPG will transmit FlowFiles to) or a ListenHTTP processor (if you used PostHTTP on your MiNiFi, this is what it can be configured to send to).  The outbound connection from either of these components needs to feed whatever downstream processors you need to do within NiFi.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#10]

@MattWho 

As per your guidance firstly I have taken TailFile processor and configured as below 

Next taken PostHTTP processor with below Configurations,in the PostHTTP url used is server hostname and port number because MiNiFi is running on this server and NiFi is running on localhost .correct me if I am wrong here.

Nextly I have taken ListenHTTP and configured the listening port as the server port number where MiNiFi is running which is provided in PostHTTP.as below 

 Eneded the data flow here as per your guidance ,when I ran TailFile processor it picked the log file and next when started PostHTTP processor it throwed below error 

Could you guide me whether my data flow is correct,if it is wrong what needs to be changed  to execute the remote file.

Thanks 

Sartha