@karthikraja @Seedy    Parameter Contexts are still relatively new.  There is an existing improvement Jira for exactly what you are looking for here that is currently in development. NIFI-8487 Unfortunately, there is not workaround or alternative to what you are trying to achieve other than recreating those common parameters in every parameter context you create currently.   If you found this query addressed your question, please take a moment to login and click accept on this solution. Thank you, Matt ... View more

@SAMSAL    NiFi Site-To-SIte (S2S) components perform to things: 1. A background process runs every 30 seconds which connects to the target URL entered (In this case https://localhost:9443/nifi ) to retrieve S2S details.  These details include details like how many nodes in target NiFi cluster, hostnames for those target NiFi nodes, load on those nodes, if those nodes support http and/or RAW transport protocols, What remote input ports exist that this source node is authorized to see, etc...  S2S details are always fetched over HTTP even if you set transport protocol to RAW. 2. Then the source NiFi uses this data to actually sent content over S2S to all the target nifi nodes in a distributed fashion. Since your target is a https, the first thing that needs to happen is a Mutual TLS handshake.  That means the keystore configured in the SSL Context service must contain a PrivateKeyEntry that support an EKU with "clientAuth".  The target NiFi which is probably returned a FQDN via those S2S details, will also send its server certificate to the client.  That means the truststore configured in your sslContextService must contain the complete trust chain for that certificate.   That server certificate which comes from the keystore in the nifi.properties file on the target NiFi must contain a single PrivateKeyEntry with a EKU that supports "serverAuth" and must also have a SAN that matches the hostname used to connect with.  The target NiFi's truststore configured in its nifi.properties file must also contain the complete trust chain for that client certificate presented from the sslContextService. So if all the above is properly in place, and I would guess it is since you are trying to S2S back to same NiFi cluster/instance and are probably using same keystore and truststore in your SSLContextService as is configured in the nifi.properties file, that those files are good.  I would however be concerned with your use of "localhost" as the target URL because I doubt the server certificate sent in that TLS handshake is going to have a SAN entry that contains "localhost".  You should instead provide the actual hostname for that target NiFi.  It is fine and common to use localhost in the "instance URL"field as that is only used to identify the host that sent the FlowFile to the target InputPort. The only other statement that stands out to me is "The user created by securing the instance has the policy "retrieve site-to-site details"."  NiFi authentication and authorization is setup to control what users are allowed to do once they access a secured NiFi UI.  The components that are added by the authorized user do not execute as that authenticated user.  All components execute as the NiFi service user.  In that case of this S2S reporting task, it is executing as the NiFi service user but authenticating to the target through that mutual LS handshake, which means the DN form that clientAuth certificate is going to be the user that needs to be authorized for both the "retrieve Site-To-Site details" and "receive data via site-to-site" NiFi authorization policies. I know there is a lot of information here and hope it is clear. If you found this addressed your query, please take a moment to login and click accept on this solution. Thank you, Matt ... View more

@Acbx  You should be able to easily do this based on your example easily with the ReplaceText processor configured as follows: If you found this addressed your query, please take a moment to login and click accept on the solution. Thank you, Matt ... View more

@leandrolinof  The NiFi merge processor is working exactly as designed.  Binary concatenation simply appends the content from one FlowFile to the end of the previous FlowFIle's content. You are trying to perform a special formatted merge.  There is no way to configure the mergeContent processor with custom merge logic. I am not clear on the what the content looks like at the various stages of your dataflow.  You mention merging FlowFiles from two streams; however, the flow screenshot you shared as 3 flow streams leading to the MergeContent processor. Are you saying the content of a FlowFile down one path is exactly this: Flow 1 ( {"cod": 1} {"cod": 2} {"cod": 3}) and on the other path it is exactly this: Flow 2 ( {"error": "error 1"} {"error": "error 2"} {"error": "error 3"}) or is it just the json objects without the "flow <num> ( )" wrapped around it? I am guessing above is what is extracted from the invokeHTTP response content? What does the FlowFile content look like on that third path before it reaches MergeContent? I might be helpful with examples of content post InvokeHTTP on all three paths and then what the exact mergeContent based on that example you would want to see. What logic can be get from the above two flows that tells us the first "cod" goes with the first "error"?  So if we were to split each into three separate FlowFiles, would there be a way to determine what cod should go with what error? The ExecuteSQL processor configuration you shared is using NiFi Expression Language (NEL).  NEL does not read from a FlowFiles content.  ${cod} and ${erros} will looks for attributes with these names created on FlowFile and replace that with the values assigned to those attributes.  If those FlowFile Attributes do not exist on the FlowFile, NEL will check the variable registry for those attribute strings. Hope this helps give you some direction with your dataflow design journey here. Matt ... View more

@leandrolinof  The exception indicates "unable to unmarshal JSON to an object" which is commonly thrown when there is an issue with the source FlowFile content. For examples some unescaped Unicode characters like whitespace. I'd inspect you source content.  I see in your dataflow that you have a convertAvrotoJson processor that took in a 2.08 KB content source FlowFile and output only a 2 byte content FlowFile.  Then later a ReplaceText processor that takes that 2 byte content and output 78 bytes.  So you'll want to inspect the content output there just before your JoltTransformJson processor.   NiFi processor bulletins which you have shared will not return stack traces that may be produced by processor. Always best to check the nifi-app.log when you see a bulletin to see if maybe there is additional log output that can be helpful as well. Hope this helps, Matt ... View more

@rahul_ars  You can use the "UpdateCounter" [1] processor to create counters and update the count on them (up and down).  The "Counter Name" property accepts NiFi EL, so you can use attribute son FlowFiles to dynamically update a counter uniquely by FlowFile traversing same dataflow path.  This will allow you to consolidate within your dataflow(s) to reduce the number of processors needed.  Fewer processors can lead to better performance through reduced resource utilization. The "Counters" UI is found under the NiFi Global menu in the upper right corner of the NiFi UI. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.UpdateCounter/index.html   If you found this help with your query, please take a moment to login and click accept on this solution. Thanks, Matt     ... View more

@techNerd  I don't know anything about the Wildfly endpoint service, so I can only assume your URL is correct.  Using the IP as you have should not be an issue as long as the NiFi host can reach that network address. What you are seeing is a timeout which indicates the endpoint you are trying to reach did not respond to you post request.  There could be numerous reasons for this such as network issues, incorrect or missing headers on post request, bad endpoint URL, too many concurrent connections to the endpoint at time of this request, failed authentication, using http for a https endpoint, bad or no client authentication, etc... I would suggest you monitor the logs on Wildfly when the post request is made by NiFi to see if: 1. Wildfly acknowledges receiving the request. 2. Wildfly does not throw and exception about the request. Since the endpoint URL you shared had multipart in it, I assumed that perhaps it only accepts multi-part form data and thus may be expecting the proper header fo this type of data.  And if it is multi-part form data, you are going to have better success using the postHTTP processor instead of the InvokeHTTP processor. Hope this helps you, Matt ... View more

@Kilynn  Can you provide more detail around "nodes randomly restarting"? What do you see in the nifi-app.log and nifi-bootstrap.log when the node(s) restart? Do you see anything in /var/log/messages?  Maybe OOM Killer? I see Max heap shows 330 GB configured.  With 12 nodes that is ~27GB set per node.   Any particular reason why you set heap so high for your NiFi nodes? Larger heaps mean longer stop-the-world GC pauses. What version of Java are you using? Thanks, Matt ... View more