Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

nifi connector broken

avatar
Super Collaborator

I am following this tutorial and I imported the template successfully but when i try to run it its throwing error

"port INFO logs not in valid state."

the tutorial iam following is

http://hortonworks.com/hadoop-tutorial/how-to-refine-and-visualize-server-log-data/

btw its also asking me to download a python file "generate_logs.py" but i don't see where its used ?

1 ACCEPTED SOLUTION

avatar
Super Mentor

@Sami Ahmad

I am not saying it will not work as is; however, without a defined path forward from those two output ports the data will just Queue. Looking at your screenshot, it does look like the dataflow is producing FlowFiles.

If you look at the stats on the process groups "log Generator" and "Data Enrichment" you will see data being produced and queued. The problem is that none of the components inside "Data Enrichment" are running. If you double click on the "Data Enrichment" process group, you will be taken inside of it. There you will see the stopped components, the invalid components, and the ~20,000 queued FlowFiles. You will need to start all the valid stopped components in this process group to get your data flowing all the way to your two putHDFS processors outside this process group. There are two output ports in this "Data Enrichment" process group that are invalid. They are not necessary for this tutorial to work. I suggest you stop the "Filter WARN" and "Filter INFO" processors and delete the connections feeding these invalid output ports. If you have already run this flow and data exists queued on these connections you wish to delete, you will need to right click on the connection and select "Empty queue" before you will be able to delete it.

7172-screen-shot-2016-08-31-at-15006-pm.png

These example were not put out by Apache. I will try to find the correct person who wrote this tutorial and see if i can get them to update it.

Thanks,

Matt

View solution in original post

11 REPLIES 11

avatar
Super Collaborator

after i clicked on the run green arrow inside the log generator processor now its not giving any error when i click on the main run icon but it does nothing and produces no files . see the screen shot

avatar
Super Mentor

@Sami Ahmad

For starters, I have to agree that the "generate_logs.py" script is not being used in that NiFi template anywhere. The NiFi flow itself has been built to generate some fake log data.

Invalid components:

Components like NiFi processors, input ports, output ports, controller services, and reporting tasks all have minimum defined requirements that must be met before they are in a "valid" state. Only components that are in a Valid state can be started. Floating your cursor over the invalid icon on a component will show why it is not valid.

7166-screen-shot-2016-08-31-at-123932-pm.png

The Data Enrichment process group in this template has two output ports that have no defined connections making the invalid.

Despite the warning you were presented with, all valid components should have been started. You can fix the issue by creating the two missing output connections: 7167-screen-shot-2016-08-31-at-124507-pm.png

So here you can see I added a new processor (UpdateAttribute with success checked for auto-terminate) and dragged a connection from the Data Enrichment process group to it twice. Once for each invalid output port it contained (Warn logs and Info logs). Now the process group no longer reflects and invalid components in it.

I am unable to see the screenshot you attached. I did start the "Log Generate" process group without making any changes to it and do see data being produced. I see data being queued in several places in the dataflow.

7169-screen-shot-2016-08-31-at-125218-pm.png

If you are not seeing any data queued, check your NiFi's nifi-app.log for errors. Also check the various GenerateFlowFile processor to see if they are producing any bulletins (This icon will be displayed on the processor if it is: 7170-screen-shot-2016-08-31-at-125511-pm.png) Floating over the bulletin will display a log message that may indicate the issue.

Thanks,

Matt


screen-shot-2016-08-31-at-125042-pm.png

avatar
Super Collaborator

ok I found the INFO logs , it was not visible in the window , when i searched for it , the whole dataflow came in view.

now If i hover my mouse over it it says "Port INFO logs is broken because the output connector is not defined"

why the output connector is not defined in a working example? do i have to use your fix or was your fix a work around?

avatar
Super Collaborator

avatar
Super Collaborator

hi Matt

you are saying that the example is not correct and would not run without your modifications ?

how can Apache put an example that is not working ?

avatar
Super Collaborator

attaching the screenshot again . showing me as attached .nifi-screenshot.jpg

avatar
Super Collaborator

also as I pointed out now its not showing the error when i hit run button but its doing nothing and no files are produced. how can i debug ?

avatar
Super Collaborator

grrrrr .. the error is back .. so sometimes i get the error and sometimes I don't ..this is frustrating.

Matt .. I don't see any "INFO logs" button/tab on the screen (as shown in my screenshot)

avatar
Super Collaborator

I don't see any "!" sign with the connector as you showing , i do see inside "Data Enrichment" connector window number 2 written next to the "!" sign