Member since
07-29-2020
574
Posts
320
Kudos Received
175
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
248 | 12-20-2024 05:49 AM | |
283 | 12-19-2024 08:33 PM | |
294 | 12-19-2024 06:48 AM | |
245 | 12-17-2024 12:56 PM | |
238 | 12-16-2024 04:38 AM |
06-11-2024
05:01 AM
2 Kudos
Hi , I need cloudera's\community's help with this because I have been struggling with it for some time. we got new VMs that going to be dedicated to Nifi. We have been using Nifi 2.0.0 M1 on an old physical servers with custom developed python custom extensions and we did not see any issues. When we moved to the new VM servers ( See spec below) at that time M2 was released and we decided to go with the latest, however once its installed and we got it working we started noticing the CPU utilization Peaks to almost 99-100% ! When I check the Processes under task manager I notice this utilization is taken by Java that is running Nifi: This happens even without running anything on Nifi. I reverted back to M1 release thinking maybe its an M2 issue since M1 did not have this issue on the old physical servers, however the problem persisted even when using the same Java & Python versions. After some troubleshooting and going through process of elimination, I found that if I dont enable the python extension or dont deploy any of those extensions ( while its enabled) the issue is gone !!! It seems there is something with the python extension that is causing the CPU to peak. One thing I noticed when I check the task manager is too many python processes even though nothing is running at that point, and they dont seem to use much of the CPU however having them in that number seem to be strange : The more I add python extension those processes kind of double in number! In the old server (M1 release ) I counted over 35 python processes running and I only have about 8 extensions! Nothing was running in Nifi at that time. Im going to try and test this with M3 release since there have been some changes and bug fixes when it comes to python extension to see if the issue persist. Has anyone run into this issue before? If so can you please advise. If not, Is there a way at least to get to the bottom of this and know exactly what is happening or what is causing it. Anything to help with providing our IT with something to work with would be highly appreciated. The server spec Im running Nifi on: @MattWho Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
06-02-2024
04:49 AM
2 Kudos
Hi @mohdriyaz , It seems the generatetablefetch by design pushes and penalize the flow to the upstream queue when there is an error that is considered none "sql query" error which per documentation only query execution error will go to the failure rel. It doesnt seem like there is anything you can do to change this behavior. The only thing I can think of to mitigate this issue is by adding another processor as a guard to help checking the connection status before calling the generateTableFetch processor. For example you can use PutSQL processors which seems from the description able to capture connectivity error through retry & maybe the failure rel per description: This is not going to solve the problem 100% but it will minimize it and it will help you capture the error. You can configure how many to retry and for how long in the relationship tab once you select to retry. Hope that helps. If it does help please accept the solution. S
... View more
06-01-2024
04:56 AM
1 Kudo
Hi @nisha2112goyal , I will try and change the Output Grouping in the JsonREcordSertWRtiter from Array to One Line Per Object and see if that helps.
... View more
05-24-2024
05:53 PM
HI @AsifArmanRahman , yes in failure relationship you could not pass attributes for some reason as far I tested in M1 , M2 seems to have similar issue , however you would be able to pass attributes in custom relaltionship. I have created valid\invalid rel and I was able to pass attributes through both . the good thing about M3 is that for python extension it works out of the box if you enable pyhthon and have the right python version installed. Before you had to rename the script folder created by the venv to bin in order to work properly. Sad part is you have to figure it on your own when you are dealing new components as development doesnt seem to interject much in this community to guide you through it. Sad 😞
... View more
05-20-2024
07:59 AM
Hi @AsifArmanRahman , Thanks for providing the url. I did not notice that before so good catch and effort on your part. There are no videos release that I know of regarding the latest release . I was told there are some that is going to be made soon.
... View more
05-20-2024
07:02 AM
Thanks @AsifArmanRahman . I thought I read it somewhere as well but I cant remember nor I can find anything about this in the traditional guides online including release notes. I hope the owners do a better job documenting these things since it saves people a lot of headache and time.
... View more
05-19-2024
07:34 AM
Hi @RAGHUY , Thanks for your reply but Im not sure you offered anything new besides what I stated in my post. The run.bat which I dont touch or recommend anyone to touch uses relative paths and checks for system variables like %JAVA_HOME% and will return an error if its not found or set incorrectly (different kind of error). Maybe there is a way to ensure that the run.bat and the referenced file "nifi-env.bat" where the NIFI_ROOT is being set is taking care of whitesapces in the path. I tried different things like surrounding the "NIFI_ROOT" with double quotes where referenced but it did not help.
... View more
05-19-2024
06:26 AM
One thing I missed is regarding the removal of ConvertAvroToJson processor in this release. not sure if my comments will make a difference but I wish if this processor stayed. It used to be very helpful specially when used alongside the ExecuteSQL processor which produces Avro format result. Such use case when I want to do something quick and light where I dont have to worry about creating reader\writer services such as for testing purposes or if I have some high level processor group and at that level I dont want create those services. To be honest, without this processor the ExecuteSQL seem to be redundant and probably need to be removed as well and substituted with the ExecuteSQLRecord. However my recommendation is for both to stay for the mentioned reasons.
I hope that makes sense and looking forward for any feedback.
Thanks
... View more
05-18-2024
04:11 PM
2 Kudos
Hi, I had some time to download and play with the latest release 2.0.0.M3 which have some major changes compared to predecessor 2.0 releases specially around introducing new UI and Color theme. Here are my preliminary observations so far: 1- It was not clear to me in the release notes the following sentence: The new Apache NiFi UI is included and available at the /nf context path! So It took me little while to figure out where are the new changes because everything looked the same using regular url ending /nifi (Example: https://localhost:8443/nifi ) before realizing oh its path change in the url which is https://localhost:8443/nf , but maybe that was me because I though initially its some configuration in one of conf files like nifi.properties. 2- Before figuring out the new url path (context) for the new UI, I was trying to figure out how to set the dark theme, initially I thought its the brush icon under the canvas Operate panel It turns out its actually to set the color theme on the selected processor ( I knew that but forgot) so I set the processor to some color I picked. After figuring out how to go to the new UI , finding the color theme is not that difficult because its under the main menu (top right with three stacked lines) you will find the Appearance sub menu item toward the bottom which you will find three options: Light , Dark & Device. I did not notice any difference between Light & Device which is I assume the standard default mode. I assume the device mode goes to whatever theme settings on them machine but Im not sure how that works under windows system. 3- One thing I noticed quickly under the new UI because I wanted to reset the color theme on the processor from the step above but I could not find the brush icon anywhere! I'm not sure if that was removed intentionally or its done differently but I could find it anywhere. Even for the notes which you would be able to see using right menu click but its no longer there as well. It appears if you have some color theme applied to the processors or notes in the old UI you probably cant change that unless Im missing something. 4- In the UI I noticed when you hover over any control button wither in the Operate Panel or in the Component Tool bar there is no label showing up. I found this is strange and I hope this is a bug of some sort and those labels are not removed because they were helpful specially for beginners. I got the same result in chrome, edge and firefox Old (Label is Showing) New: (Label is not showing) 5- Under the new UI, I no longer find the Validate processor icon when viewing a given processor properties. Again Im not sure if it was moved to somewhere else or removed completely which I hope not because it was a good feature to have. Old (Validation Option) New (No Validation Option) 6 - Under the new UI, when listing queue content the whole page is changing to the list queue view vs showing that list as some sort of pop up with close (x) button as it works in the old UI. I guess that is fine and its matter of getting used to the new way, however what struck me is that there is no navigation option to go back! yes I can click the browser go back button or click the Nifi logo top left which will take me back but its not so intuitive and I would be concerned for clicking the browser Go Back button because this is not Nifi and we dont have control over what the browser can do. 7- For some reason the dark theme is not being applied when viewing the flowfile content as I was seeing this: 8- In the new version using the legacy UI if you want to call it , I noticed something new which Im not sure what it's used for yet , there is light bulb icon between the search and the home icon , when its clicked it brings panel to the right as the following: This is not present under the new UI: I hope this is getting covered some where in the manuals or one of the coming videos. 9- I can see some improvement in the new UI but I also see things that I'm not sure its a success, for example before under a given processor properties where there is a property to create a service you used to click on the property and you would see Create New Service as one of the options in the drop down , now you have to go to the three dots to the right (...) and click there to Create New service which is not intuitive in my opinion. Same thing if you want to go to configure the service where before it used to be a visible arrow , but now its under the (...). This applies also to managing flowfile in the list queue view , and managing the control service where instead of visible controls next to the service (enable, disable, delete..etc.) now everything is wrapped up under three dots (...) which means additional click. New ( No option to create service in the dropdwon, click on ... to configure and create service) means additional clicks Old: ( create service in the drop down and straight forward arrow to configure) 10 - I guess some of the color contrast can be worked on under the new UI ( light or dark) because its not as clear as in the legacy UI. For example when the parent group is showing number of modified sub groups for versioning, going from 0 to N the contrast of the icon in legacy stands out more to me: Old (No Modified Sub groups) : Old ( some modified sub groups) Notice how the color of the icon changed form light to dark. However in the new UI New ( No modifications ) New ( After modifications) I think it went from lighter to darker (similar to legacy) so its not standing out, it should be reversed in my opinion where it goes from darker to lighter. 11- I'm going to end on a very positive note here. I think one major improvement is regarding python extension where under windows I no longer have to create the python venv bin folder manually every time new extension processor is added. Good Job guys and thank you for listening ( I guess this address the issue I raised here: https://community.cloudera.com/t5/Support-Questions/Uncommenting-python-config-in-nifi-properties-makes-Nifi/m-p/384535 ) . However I'm not sure if anything has changed in the python extension engine itself to elevate some of the issues specially with java objects that I was facing ( referred to here: https://community.cloudera.com/t5/Support-Questions/2-0-0-M1-Python-extension-issues-with-custom-processor-and/m-p/383177 ) I need to do more testing to see if anything has changed. @pvillard I made some changes in github to the extension developer guide to highlight those issues and how to work around them which as you requested then checked in but I dont see those changes made it to the developer guide online for some reason. Also in the release note it talks about loading python extension through NAR for better security and performance but that is also not being mentioned int he same developer guide (https://nifi.apache.org/documentation/nifi-2.0.0-M1/html/python-developer-guide.html ). Im hoping that will be covered in the coming videos but its also important to modify the guide as well. Overall looks great and its refreshing to see finally some UI changes even with some issues which I'm sure will be ironed out in future releases. Thanks for reading and Sorry for keeping it long. Looking forward for your feedback: @pvillard , @MattWho S
... View more
Labels:
- Labels:
-
Apache NiFi
05-17-2024
08:53 AM
1 Kudo
Hi, It seems like the new version of Apache Nifi 2.0.0 M3 was just released. Per the release notes there are some changes to how python extension are packaged and changes to the UI as well . @pvillard Are you going to make new video to discuss those changes and other important ones? It will be highly appreciated. Im planning to download and start testing but having a video from someone like you would be helpful to set the expectation and avoid running into any issues. Thanks
... View more
Labels:
- Labels:
-
Apache NiFi