Came here via Google. Just for other people. NiFi does support multipart with the InvokeHTTP since a few releases: https://palindromicity.blogspot.com/2020/04/sending-multipart-form-data-with.html ... View more

Hi Andrew, thanks for the tutorial. It was very useful. It's strange that the official nifi-registry documentation doesn't cover any of this. Anyway, I feel I should add a missing step to the instructions.   Chances are, the "CN=localhost,OU=nifi" is not going to be valid in real-world scenarios. To figure out what exactly the user name should be, you can check the nifi-app.log where you will see a message like this:   org.apache.nifi.registry.client.NiFiRegistryException: Error retrieving flow snapshot: Unknown user with identity 'CN=host1, OU=ACME, O=ACME, L=Canberra, ST=Australian Capital Territory, C=AU'. Contact the system administrator.   Simply use that identity instead of "CN=localhost,OU=nifi". ... View more

Hello, I'm a learner & i would like to use the method you made mentioned here to collect logs in a remote server & send to Nifi. Please, can you put me through because i have been battling with how to build a msi before the real implementation. Thank you so much. ... View more

Hi to all.    i'm having the same issue, even after following the procedure.   What i did: 1) get the ssl certificate from AWS using the ssl: penssl s_client -showcerts -connect <source>:443 </dev/null | openssl x509 -text | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' 2) Copy the result to a *.crt file 3) convert the file to DEM openssl x509  -in aws_cert.crt  -inform PEM  -out aws_cert.der -outform DER 4) Create the jks file using keytools keytool  -import  -trustcacerts -alias aws3buckets  -file aws_cert.der  -keystore truststore-amazon.jks 5) change the permissions to be accessible from nifi. 6) add the file in the  StandardSSLContextService and set password.   I receive the same e SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target   Do you have any clue how to solve this? I can use the crt file on the keytool? There is any version of keytool that we need to use?   Thanks     ... View more

@Yemre  If you have a support contract with Cloudera, I'd recommend opening a support case to assist with your issue here. Possible causes: 1. Unsuccessful Mutual TLS handshake with NiFi-Registry from the NiFi hosts resulting in NiFi node connection only being 1-way TLS connection and node treated as an "anonymous" user.  Anonymous users would could not proxy user requests and can not see anything except public buckets. --- Caused by missing complete trust chain on one or both sides of connection.  Truststore in NiFi-Registry contains complete trust chain for NiFi hosts keystore PrivateKeyEntry. --- Caused by PrivateKeyEntry not meeting minimum requirements (missing SAN with NiFi hostname,  missing EKU of clientAuth, and/or using wildcards are the most common) 2. NiFi-registry is configured with an identity mapping pattern in the nifi-registry.properties file that is matching on the DN from the the NiFi's client certificate presented in the mutual TLS handshake. The Identity mapping value  and transform is then being applied which alters the actual client string which must be then authorized for Proxy and buckets policies.   Hope this helps you, Matt ... View more

I try to install elasticsearch-6.4.2 to my cluster(HDP 3.1 Ambari 2.7.3) , intallation was completed successfully but it could not start, and the error encounterd:   Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/ELASTICSEARCH/package/scripts/es_master.py", line 168, in <module> ESMaster().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 1011, in restart self.start(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/ELASTICSEARCH/package/scripts/es_master.py", line 153, in start self.configure(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/ELASTICSEARCH/package/scripts/es_master.py", line 86, in configure group=params.es_group File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 123, in action_create content = self._get_content() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 160, in _get_content return content() File "/usr/lib/ambari-agent/lib/resource_management/core/source.py", line 52, in __call__ return self.get_content() File "/usr/lib/ambari-agent/lib/resource_management/core/source.py", line 144, in get_content rendered = self.template.render(self.context) File "/usr/lib/ambari-agent/lib/ambari_jinja2/environment.py", line 891, in render return self.environment.handle_exception(exc_info, True) File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/ELASTICSEARCH/package/templates/elasticsearch.master.yml.j2", line 93, in top-level template code action.destructive_requires_name: {{action_destructive_requires_name}} File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'hostname' was not found in configurations dictionary!   I modified the property of  discovery.zen.ping.unicast.hosts from elastic-config.xml and hostname from elasticsearch-env.xml, However, it still could not start  and the same error encountered, do you have any idea? ... View more

@Kasun, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. ... View more

Hi @stevenmatison, I don't see any data in the cassandra database. I don't see any errors either. Here are the screenshots. ... View more

For only "name", it's working fine, I get following FlowFile content:      I guess the issue is within the ControllerServices. I used JSON-TreeReader but with default configuration:    ... View more

I had the same issue. Did you get around to resolve this? ... View more

ARRAYs are a bit tricky.   But JSONReader and Writer may work better.       ... View more

@schnell  Glad you were able to find the remnant that blocked re-install.   Here is my SO reply, which gives some details about how to completely remove HDP and components from node filesystem...   With ambari, any service that is deleted with the UI, will still exist on the original node(s) the service was installed on. You would need to manually remove them from the node(s). This process is hard to find documentation on, but basically goes as follows: Delete the application from file system locations such as /etc/ /var/ /opt/ etc Remove user accounts/groups You can find some more details in this blog post here which goes into some of the detail for completely removing HDP. Just follow steps for single service. https://henning.kropponline.de/2016/04/24/completely-uninstall-remove-hdp-nodes/ https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/ch_uninstalling_hdp_chapter.html https://gist.github.com/hourback/085500397bb2588964c5   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

Amazing work here sir!   ... View more

We have some background on schema evolution in Parquet in the docs - https://docs.cloudera.com/runtime/7.2.2/impala-reference/topics/impala-parquet.html. See "Schema Evolution for Parquet Tables". Some of the details are specific to Impala but the concepts are the same across engines including Hive and Spark that use parquet tables. At a high level, you can think of the data files being immutable while the table schema evolves. If you add a new column at the end of the table, for example, that updates the table schema but leaves the parquet files unchanged. When the table is queried, the table schema and parquet file schema are reconciled and the new column's values will be all NULL.   If you want to modify the existing rows and include new non-NULL values, that would require rewriting the data, e.g. with an INSERT OVERWRITE statement for a partition or a CREATE TABLE .. AS SELECT to create an entirely new table. Keep in mind that traditional Parquet tables are not optimized for workloads with updates - Apache Kudu in particular and also transactional tables in Hive3+ have support for row-level updates that is more convenient/efficient.   We definitely don't require rewriting the whole table every time you want to add a column, that would be impractical for large tables! ... View more

@jainN   Great looking flow.   The modification you need is to simply remove json route which is combined with csv.  Connect json route from Notify to FetchFile.   You may need to adjust the wait/notify so that csv is released when you want.   The wait/notify is often trickey, so i would recommend working with wait/notify until you understand their behavior.  Here is a good article:   https://community.cloudera.com/t5/Community-Articles/Trigger-based-Serial-Data-processing-in-NiFi-using-Wait-and/ta-p/248308   You may find other articles/posts here if you do some deeper research on Wait/Notify.       If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

@jainN    If you are looking to route flowfiles that end in json versus those that are not, check out RouteOnAttribute with something similar to json => ${filename:endsWith('.json')}.  You would use this after using your method of choice to list/fetch the files which would then provide a $filename for every flowfile.   With this json property added to RouteOnAttribute you can drag the json route to a triggering flow, and send everything else (not json: unmatched) to a holding flow.   NiFi Wait/Notify should be able to provide the trigger logic, but there are many other ways to do it with out wait/notify by using another datastore, map cache, etc.  For example, your non json flow could simply write to a new location and finish.  Then your json flow can process that new location some known amount of time later.  The logic there is your use case ofc ourse, the point is to use RouteOnAttribute to split your flow.   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

@toutou    From your hdfs cluster you need hdfs-site.xml and correct configuration for PutHDFS.    You may also need to satisfy creating a user with permissions on the hdfs location. Please share PutHDFS processor configuration, and error information to allow community members to respond with specific feedback required to solve your issue.   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

Yes, That's correct Answer and it works. But do we have any other workaround, as we have disabled exec due to security reasons. So how to achieve this.? ... View more