Member since
04-19-2023
22
Posts
2
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
927 | 05-08-2023 12:30 AM | |
2070 | 04-20-2023 07:31 AM |
09-29-2023
06:04 AM
@VLban From what you have shared, I don't think you are having any issues with yoru NiFi communicating with your zookeeper. When NiFi is running it sends a heartbeat message to ZK so that ZK knows that node is available. ZK is used to facilitate the election of two NiFi roles: 1. Cluster coordinator - Only one node in the NiFi cluster can be elected as cluster coordinator. The cluster coordinator is responsible for replicating requests made form any node to all nodes in the cluster. This allows for NiFi to support a zero master architecture meaning that users do not need to connect to the elected cluster coordinator node in order to make changes. Users can interact with the NiFi cluster form any node. 2. Primary node - Only one node at a time can be elected to this role. The node with this assigned role will be the only node that schedules component processors configured with "primary node" only execution. Your log output shared indicates that ZK is receiving these heartbeats from at least some of the 10 nodes (maybe all of them, but we know the node from which you got this log is talking to ZK fine) allowing for cluster coordinator election to be successful. We see that "sd-sagn-rtyev:9082" was elected with the cluster coordinator role. Once nodes aware of who the elected cluster coordinator is, they will start sending cluster heartbeats to that elected cluster coordinator. The initial set of heartbeats will be used to connect the nodes to the cluster (things like making sure all nodes are running the exact sam flow.xml.gz/flow.json.gz, have matching users.xml files, and authorizations.xml files). If your NiFi is secured (running over HTTPS), then all communications between nodes are over mutualTLS encrypted connections. Based on the exception you shared, it sounds like this connection between node(s) and the elected cluster coordinator is failing. 1. Make sure that all nodes can properly resolve the cluster hostnames to reachable IP addresses. 2. Make sure that the PrivateKeyEntry in each nodes keystore configured in the nifi.properties supports EKUs clientAuth and serverAuth, have required host SAN entry(s). 3. Make sure that the truststore used on every node contains the complete trust chain for all the privateKey entries being used by all 10 nodes. A PrivateKey may be signed by a root or intermediate CA (an intermediate CA may be signed by another intermediate CA or the root CA). A complete trust chain consists of ALL trusted public certificates from signer of the Private key to the root CA. If a MutualTLS handshake can not be established, typically one side or the other will simply close the connection. Most commonly as a result of lack of proper trust. Thus would explain the Broken pipe (write failed) as the client was unable to send heartbeat connection_request to the elected cluster coordinator. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-28-2023
06:08 AM
Can nifi authenticate it self in an external zookeeper using login and password? I found in the documentation only using Kerberos and ldap, and without this the usual sasl anti-fiction to zukipir, as for example, Kafka is authenticated in zookeeper from zookeeper.jaas config. Where and how is this implemented? or should the zookeeper be open and visible from the outside?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Zookeeper
06-13-2023
01:32 AM
Who tried the stream listHDFS---FetchHDFS--invokeHTTP I want to send a file from hdfs parquet to the database table, I encounter a problem at the time of loading into the database I use a syntax error, maybe I'm doing something wrong, please help with the parameters for the database
... View more
Labels:
- Labels:
-
Apache NiFi
05-30-2023
02:22 PM
@VLban MergeContent and MergeRecords handling merging of FlowFiles's content differently. Since your FlowFiles already contain Json formatted record(s), using MergeContent is not going to be the correct processor to use. MergeContent does not care about the data/content format (except for Avro) of the inbound FlowFiles. With Binary Concatenation, On flowFile's content bytes are simply write starting at the end of the last FlowFile's content. So in the case of JSON, the resulting merged FlowFile's content is not going to be valid json anymore. Both processors will bin FlowFiles each time the Processor executes based on its run schedule. At the end of each bin cycle the bins are evaluated to see if both configured mins are satisfied. If so, the bin will be merged. Setting a max does not mean that the bin will wait to get merged until the max has been met. So you would be better to set your min to 500 MB if you always want files of at least 500 MB and set you max to a value a bit larger then that. Doing so may result in bins that say have 480 MB binned and next FlowFile can't be added because it would then exceed configured max (FlowFile placed in new bin). So the Max Bin Age property when set will force a bin to merge once the bin has existed for the configured max bin age (this avoid FlowFile getting stuck in these merge based processors). If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
05-08-2023
12:30 AM
1 Kudo
problem solution due to java spec keytool -delete -alias RCA-CA -keystore /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts -storepass changeit -noprompt keytool -import -alias RCA-CA -keystore /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts -file /etc/pki/ca.crt -storepass changeit -noprompt
... View more
05-02-2023
11:18 PM
How compression works in nifi I have a consumercafcarerecord(jsontreereader+parquetrecordsetwriter)--mergecontent --putndfs(none compression) at my output, parquet files are read by spark, the problem is that the output from json is not large x2, but when I do snappy compression, the file namefile.parquet.snappy shrinks x5x6 at the output but the file cannot be opened by spark or it opens and the structure is no longer as good as it was at step 1 how to get the same structure from step 1 and x5x6 compression?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
04-28-2023
05:05 AM
another interesting point is how to implement on updateattribute the function of checking whether it was delivered to puthdfs because if hdfs runs out locally, the process continues to go and the files are not written but are thrown out of the queue and go to another file in the basket, in fact, files are lost if the meso runs out and the chain does not stop , you need to check if the file in ndfs did not arrive stop the stream or the memory ran out in ndfs stop putndfs and let the recycle bin fill up
... View more
04-20-2023
07:31 AM
1 Kudo
the issue was resolved the problem was in the settings of the version of the tls protocol
... View more