Member since
07-30-2019
3436
Posts
1633
Kudos Received
1012
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 160 | 01-27-2026 12:46 PM | |
| 583 | 01-13-2026 11:14 AM | |
| 1292 | 01-09-2026 06:58 AM | |
| 1045 | 12-17-2025 05:55 AM | |
| 509 | 12-17-2025 05:34 AM |
05-10-2019
07:55 PM
@sho aa I strongly recommend starting a new question in the community versus adding new questions and setup scenarios under an existing question. You current "Answer" is a new question which is not directly related to the issue expressed in the original question of this post. Creating a new question gets that question exposed to more users of this forum. I am not an HBase Subject Matter Expert (SME). As far as NiFi is concerned, how the NiFi Hbase processor within HDF's NiFi and the Hbase processor in Apache NIFi is no different. The Error you are showing in the screenshot states that the CS is unable to resolve the hostname "sandbox-hdp.hortonworks.com" which means it can no figure out what the IP address is. If it cannot resolve the hostname, it is not going to be able to communicate with that endpoint. While the config come from the provided site.xml files to the CS, there may be challenges accessing ports with the sandbox from an external resources. I am also not an expert with the HDP sandbox. If above tips do not help, best option is to start a new question that will engage more forum users. Thank you, Matt
... View more
05-07-2019
01:32 PM
2 Kudos
@Soumya Ghosh 1. The DistibuteMapCacheServer (DMC server) has not direct linkage to a dataflow built in NiFi. Thus when creating a template of your flow from NiFi for the purpose of generating your MiNiFi yaml file, the DMC Server will not be included in the template. Only the DMC client service will be included. I have not tried to manually add the DMC server to the yaml file, but is likely possible. MiNiFi does not have a rest-api where you can send command to add additional components like you can do it NiFi. 2. The only way to host a DMC server is via a NIFi instance. There is not way to execute a NAR file. 3. A major limitation to using the DMC server is lack of HA. If the NiFi instance hosting the DMC server crashes you lost all your cached data (assuming crash is not recoverable. The recommended path is to configure you ListFile processor to use one of the other available external cache server that offer HA capability. These alternate cache service are setup independent of NiFi or MiNiFi and will offer you what you need to support using Entity tracking in your ListFile processor running on MiNiFi. Thank you, Matt If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.
... View more
05-08-2019
08:51 PM
@3nomis MergeContent should be using Defragment. There is no default value for Max Bin Age, so not sure what you set there. If left blank, processor will wait for ever to merge a bin unless you run out of bins. Also make sure you adjust the object and size thresholds on the connections feeding the MergeContent processors so that they are large enough to accommodate the number of splits that need to be merged. Considering the size of the FlowFiles being merged, it may take time to merge all of them. as far as bins, try setting 21 of them. Thanks, Matt
... View more
05-02-2019
04:45 PM
Thanks Matt, I didn't realize that the colored text was not a comment in this case.
... View more
04-10-2019
06:55 PM
@Samar Aarkotti *** Community Forum Tip: Try to avoid starting a new answer in response to an existing answer. Instead use comments to respond to existing answers. There is no guaranteed order to different answer which can make it hard following a discussion. It always best to leave your processor at default value for concurrent task unless there is a specific need to increment. Here is an article on this topic: https://community.hortonworks.com/articles/221808/understanding-nifi-max-thread-pools-and-processor.html and another on "Run Duration": https://community.hortonworks.com/articles/221807/understanding-nifi-processors-run-duration-functio.html
... View more
04-04-2019
11:15 AM
@Matt Clarke Thanks a lot. You are genius!
... View more
03-25-2019
12:48 PM
@Aaron R The only method you can use to systematically adjust the configured run Schedule of a NiFi processor is through the use of the NiFi rest API. You would achieve this through a series of invokeHTTP processors. - 1. Use invokeHTTP processor to stop another processor. 2. Use invokeHTTP processor to change "Run schedule" configuration on a processor. 3. Use InvokeHTTP processor to start processor again. (Note that the processor will immediately execute when you start it and then not execute again for newly configured "run schedule" duration.) - You would then need to repeat above process for all three of your sensor inputs. Since I do not know the full details of your use case, it is not clear if this solution will really meet you needs. However, I can tell that based on how NiFi was developed, this is the only method for adjusting this configuration without user manual intervention. - You may just be better off building a flow that sends a FlowFile to each of the sensors invokeHTTP processor to trigger their execution. The InvokeHTTP processor does accept an inbound connection. When setup this way the invokeHTTP processor configured "run schedule" controls how often the processor checks that inbound connection for a FlowFile. When a FlowFile exists it will execute. - When a FlowFile is generated (by your "Sensor monitor controller" for example), that FlowFile will be timestamped. First you would need to split that incoming JSON into three different FlowFiles (one for each sensor). You could build a flow that parses the various sensor setting in to NiFi FlowFile Attributes and then create a if/then loop to check when the FlowFile timestamp + sensor value is equal to or less than current time. When that loop exits you route the FlowFile to appropriate invokeHTTP processor to trigger it to collect the sensor data. - Now becomes the question of what do you want to do with "Original" FlowFile out of the invokeHTTP. You can discard it and expect a new sensor monitor input each time or you can reset its trigger time and loop it back to the start. You would also need to in this case build in a method in your flow to kick this looping FlowFile out once a new sensor input comes in. (perhaps using the DistributedMapCache to store a value you increment each time a new timer comes in and then have your looping FlowFiles fetch this value to see if it has changed and kick FlowFile out if it has). - Hope these ideas help you with your design decisions. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.
... View more
03-21-2019
02:00 PM
@TRACEY JACKSON NiFi processors are configurable to run using a Timer Driven or Cron Driven scheduling strategy. This scheduling can uniquely configured per each processors configuration. Processors are then started and the operate based on the configured scheduler from that point forward. - The NiFi API can be used to perform any action a user can perform directly from the UI. The best way to learn the rest-api calls is to use the "Developer tools" available via most browsers. - While focused on tab where NiFi UI is open, launch the developers tools. Chrome Browser example: Then perform the action via the NiFi UI and you will see the "Network" call display in list. You can right click on that call and select "Copy as curl". Now you have an example of how to execute that same request via command line. - For more info in the NIFi rest-api, you can look in "help" found within the NiFi UI Global menu (upper right corner). The Rest Api documentation can be found in the "Developer" section at bottom of list on far left side of help UI. - The danger with trying to "run" dataflows in NiFi via command line is you may end up stopping processors in the dataflow chain that results in FlowFile being unprocessed and sitting on connection queues between processors. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.
... View more
03-22-2019
05:48 PM
1 Kudo
@Benjamin Bouret The invokeHTTP processor would require you to use a SSL context service when communicating with a secure (https) endpoint. The SSLContext service can be setup with only a truststore.jks if this is only a 1-way TLS connection that does not require client authentication. - You should be able to use openssl to get the complete public certificate chain from the target secured endpoint. From command line execute following command: Openssl s_client -connect <hostname>:<port> -showcerts - The return from this command will include one or more public certificates. each public certificate will start with and end with following: -----BEGIN CERTIFICATE-----
.....
-----END CERTIFICATE----- - Copy each certificate including the above two lines and write each to separate file with a .crt extension. for example: CA-1.crt - Then import each of these public certificates in to the truststore you want to use in your SSLContext service as follows: # keytool -import -alias <unique Alias name 1> -file CA-1.crt -keystore truststore.jks
# keytool -import -alias <unique Alias name 2> -file CA-2.crt -keystore truststore.jks
etc... - Make your your NiFi service user can read this file where ever you decide to place on each of your NiFi nodes. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.
... View more