@schnell  Glad you were able to find the remnant that blocked re-install.   Here is my SO reply, which gives some details about how to completely remove HDP and components from node filesystem...   With ambari, any service that is deleted with the UI, will still exist on the original node(s) the service was installed on. You would need to manually remove them from the node(s). This process is hard to find documentation on, but basically goes as follows: Delete the application from file system locations such as /etc/ /var/ /opt/ etc Remove user accounts/groups You can find some more details in this blog post here which goes into some of the detail for completely removing HDP. Just follow steps for single service. https://henning.kropponline.de/2016/04/24/completely-uninstall-remove-hdp-nodes/ https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/ch_uninstalling_hdp_chapter.html https://gist.github.com/hourback/085500397bb2588964c5   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

Amazing work here sir!   ... View more

@nhakhoaparis. I often seen questions like this and although there are many solutions, the one i prefer is to leave the source table alone, create another table with the modified schema.   In one form of this concept, you leave the old table behind, and INSERT INTO new_table SELECT * FROM old_table.   This is immutable table is also common concept for parquet.  You do not modify columns.  You read, change, then re-write.  One way to complete the above concept in hive query language:  select parquet into a non parquet table,  do your work to modify the new table, update the new column, etc, then select back into a new parquet table with the new schema.   You can also do some of the above with spark or other programming languages.   Many options, but in summary:  leave the source table alone and create a new tables.  During the course, i like to call these staging tables, and sometimes keep the new names, or drop original table, and rename new table to the old table.   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

Thats weird.. did you scroll the attributes list down (just in case its that simple)?   i cant see the full modal so not sure.   The config of each attribute looks okay so does the json. ... View more

@justenji   Great response!   @GMAN  I have some templates that may help you get a head start:   https://github.com/steven-matison/NiFi-Templates   There are 2 InvokeHttp Examples. ... View more

@jainN   Great looking flow.   The modification you need is to simply remove json route which is combined with csv.  Connect json route from Notify to FetchFile.   You may need to adjust the wait/notify so that csv is released when you want.   The wait/notify is often trickey, so i would recommend working with wait/notify until you understand their behavior.  Here is a good article:   https://community.cloudera.com/t5/Community-Articles/Trigger-based-Serial-Data-processing-in-NiFi-using-Wait-and/ta-p/248308   You may find other articles/posts here if you do some deeper research on Wait/Notify.       If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

@jainN    If you are looking to route flowfiles that end in json versus those that are not, check out RouteOnAttribute with something similar to json => ${filename:endsWith('.json')}.  You would use this after using your method of choice to list/fetch the files which would then provide a $filename for every flowfile.   With this json property added to RouteOnAttribute you can drag the json route to a triggering flow, and send everything else (not json: unmatched) to a holding flow.   NiFi Wait/Notify should be able to provide the trigger logic, but there are many other ways to do it with out wait/notify by using another datastore, map cache, etc.  For example, your non json flow could simply write to a new location and finish.  Then your json flow can process that new location some known amount of time later.  The logic there is your use case ofc ourse, the point is to use RouteOnAttribute to split your flow.   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more