I had the same issue. Did you get around to resolve this? ... View more

MiNiFi offers CPP version that is well suited for Windows event log ingestion. ... View more

ARRAYs are a bit tricky.   But JSONReader and Writer may work better.       ... View more

@schnell  Glad you were able to find the remnant that blocked re-install.   Here is my SO reply, which gives some details about how to completely remove HDP and components from node filesystem...   With ambari, any service that is deleted with the UI, will still exist on the original node(s) the service was installed on. You would need to manually remove them from the node(s). This process is hard to find documentation on, but basically goes as follows: Delete the application from file system locations such as /etc/ /var/ /opt/ etc Remove user accounts/groups You can find some more details in this blog post here which goes into some of the detail for completely removing HDP. Just follow steps for single service. https://henning.kropponline.de/2016/04/24/completely-uninstall-remove-hdp-nodes/ https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/ch_uninstalling_hdp_chapter.html https://gist.github.com/hourback/085500397bb2588964c5   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

Amazing work here sir!   ... View more

We have some background on schema evolution in Parquet in the docs - https://docs.cloudera.com/runtime/7.2.2/impala-reference/topics/impala-parquet.html. See "Schema Evolution for Parquet Tables". Some of the details are specific to Impala but the concepts are the same across engines including Hive and Spark that use parquet tables. At a high level, you can think of the data files being immutable while the table schema evolves. If you add a new column at the end of the table, for example, that updates the table schema but leaves the parquet files unchanged. When the table is queried, the table schema and parquet file schema are reconciled and the new column's values will be all NULL.   If you want to modify the existing rows and include new non-NULL values, that would require rewriting the data, e.g. with an INSERT OVERWRITE statement for a partition or a CREATE TABLE .. AS SELECT to create an entirely new table. Keep in mind that traditional Parquet tables are not optimized for workloads with updates - Apache Kudu in particular and also transactional tables in Hive3+ have support for row-level updates that is more convenient/efficient.   We definitely don't require rewriting the whole table every time you want to add a column, that would be impractical for large tables! ... View more

@jainN   Great looking flow.   The modification you need is to simply remove json route which is combined with csv.  Connect json route from Notify to FetchFile.   You may need to adjust the wait/notify so that csv is released when you want.   The wait/notify is often trickey, so i would recommend working with wait/notify until you understand their behavior.  Here is a good article:   https://community.cloudera.com/t5/Community-Articles/Trigger-based-Serial-Data-processing-in-NiFi-using-Wait-and/ta-p/248308   You may find other articles/posts here if you do some deeper research on Wait/Notify.       If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

@jainN    If you are looking to route flowfiles that end in json versus those that are not, check out RouteOnAttribute with something similar to json => ${filename:endsWith('.json')}.  You would use this after using your method of choice to list/fetch the files which would then provide a $filename for every flowfile.   With this json property added to RouteOnAttribute you can drag the json route to a triggering flow, and send everything else (not json: unmatched) to a holding flow.   NiFi Wait/Notify should be able to provide the trigger logic, but there are many other ways to do it with out wait/notify by using another datastore, map cache, etc.  For example, your non json flow could simply write to a new location and finish.  Then your json flow can process that new location some known amount of time later.  The logic there is your use case ofc ourse, the point is to use RouteOnAttribute to split your flow.   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

@toutou    From your hdfs cluster you need hdfs-site.xml and correct configuration for PutHDFS.    You may also need to satisfy creating a user with permissions on the hdfs location. Please share PutHDFS processor configuration, and error information to allow community members to respond with specific feedback required to solve your issue.   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

Yes, That's correct Answer and it works. But do we have any other workaround, as we have disabled exec due to security reasons. So how to achieve this.? ... View more

@Vamshi245  Yes,  HandleHttpRequest and HandleHttpResponse are used in tandem.  Behind the processors is a map cache which holds connection session between request/response processors.   If your flowfile (json) coming out of custom HandleHttpRequest is delivered to stock HandleHttpResponse, it will send the json to the original connecting client.   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

As suggested above, update post with your processor, its reader and writier settings.  It sounds like you have something misconfigured.  If possible show us a screen shot of your flow too.    ... View more

@Chigoz    Your issue with that sandbox cluster is likely too many services trying to run on too small of an instance/node.  You will need to strategically turn on the components you need individually starting with HDFS first.  If you have issues specific to the sandbox, or certain components starting, you should open a post with those specific errors.      To install HUE, check out my management pack:   https://github.com/steven-matison/dfhz_hue_mpack   Local search for "hue install" topics includes articles reference above hue mpack:   https://community.cloudera.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=false&q=install%20hue   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

@Makaveli  You can use  File Filter Regex in ListSFTP to control which files come out of this processor.   Otherwise, you need to use ListSftp->UpdateAttribute->RouteOnAttribute to make sure the file name ends as suggested above.  Once you have a solid route of the files you want, then connect to FetchSFTP. ... View more

@ravisro  I think the only way you can do this is to make a custom process that interacts with the NiFI Api to complete your actions.   If you use the NiFI UI,  with developer tools open,  you can watch/record all of the api calls the UI uses against the api to perform the action manually.  Next, duplicate the required actions in your processor.       I would love to see you post this same question in the NiFi Slack...  ... View more

@manas_mandal786  Thanks for providing template.  The next thing I wanted to look at was configs so you read my mind.   I was able to import your template, start the context service, and add success/failure routes for each test to an output port (so i can see results are success/fail).  With the workflow setup I execute 2 curl calls:   curl -L -X POST 'http://localhost:8009/report' -H 'Content-Type: multipart/form-data' -F test=test curl -L -X POST 'http://localhost:8009/report' -H 'Content-Type: multipart/form-data' -H 'ReportType: test' -F test=test   And my canvas looks like this:     I did not get any warnings.  Everything worked as expected.  If you are still getting issues,  i suspect it is something to do with your environment (windows) not being able to store the metadata for the StandardSSLContext Service. This store will hold the client connection until you send the response.  In your case that part does not seem to be working.   Look for any errors/warnings in nifi-app.log while testing.  You may see warnings in the Controller Services window on the StandardSSLContext.   You may need to set each processor to DEBUG in order to find more info in the nifi-app.log.        ... View more