Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.
VLban
Explorer
My Accepted Solutions
Show All

Error with nifi accumulation in one file

by Explorer in Support Questions
‎05-16-2023 03:01 AM
‎05-16-2023 03:01 AM
I have two processes 1. consumerkafkarecord --mergerecord  --puthdfs 2. consumerkafkarecord --mergecontent--puthdfs when I use process 1, I have files in ndfs on the output readable spark, database, python libraries without problems, but the files are not larger than 200mb, all different sizes, although 500mb is set, but it is not filled when I use process 2 with the same parameters in mb and the number of lines, the files I get are exactly 500mb, but these files do not open, not by spark, not by any database, not by python libraries question why? I also want a large file always 500mb and so that it can be read without problems as in process 1 mergecontent settings Merge Strategy  Bin-Packing Algorithm Merge Format  Binary Concatenation Attribute Strategy  Keep Only Common Attributes Correlation Attribute Name  No value set Minimum Number of Entries  10000 Maximum Number of Entries  1000000 Minimum Group Size  100 MB Maximum Group Size  500 MB Max Bin Age  No value set Maximum number of Bins  10 Delimiter Strategy  Text Header  No value set Footer  No value set Demarcator  \n mergerecord settings Record Reader  JsonTreeReader Record Writer  ParquetRecordSetWriter Merge Strategy  Bin-Packing Algorithm Correlation Attribute Name  No value set Attribute Strategy  Keep Only Common Attributes Minimum Number of Records  10000 Maximum Number of Records 1000000 Minimum Bin Size  100 MB Maximum Bin Size  500 MB Max Bin Age  No value set Maximum Number of Bins  10 ... View more
Labels:

Re: Nifi cluster, StandardFlowService

by Explorer in Support Questions
‎05-08-2023 12:30 AM
1 Kudo
‎05-08-2023 12:30 AM
1 Kudo
problem solution due to java spec keytool -delete -alias RCA-CA -keystore /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts -storepass changeit -noprompt     keytool -import -alias RCA-CA -keystore /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts -file /etc/pki/ca.crt -storepass changeit -noprompt ... View more

Nifi cluster, StandardFlowService

by Explorer in Support Questions
‎05-04-2023 06:54 AM
‎05-04-2023 06:54 AM
I am running a cluster with tls what commands I used to create tls, perhaps something was not indicated in them:. I have crt and key my company openssl pkcs12 -export -in /etc/nifi/certs/wibe.t.crt -inkey /etc/nifi/certs/wibe.t.key -out /etc/nifi/certs/pkcs12_file.p12 -name nifi_alias -CAfile /etc/pki/CA.pem -caname root -password pass:qwerty231 keytool -importkeystore -deststorepass "keystore_password" -destkeypass "keystore_password" -destkeystore /etc/nifi/certs/nifi_keystore.jks -srckeystore pkcs12_file.p12 -srcstoretype PKCS12 -srcstorepass "pkcs12_password" -alias nifi_alias  k eytool -import -trustcacerts -alias root -file /etc/pki/CA.pem -noprompt -keystore /etc/nifi/certs/nifi_truststore.jks -storepass "truststore_password" at first I could not collect a quorum, then this problem went away and the cluster chose a leader, work is done with external zookeepers on three servers no errors, only warning 2023-05-04 16:06:34,116 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: java.net.SocketException: Broken pipe (Write failed) 2023-05-04 16:06:39,119 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 10.1.4.2:9082; will use this address for sending heartbeat messages 2023-05-04 16:06:39,119 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at 10.1.4.2:9082. Will send Cluster Connection Request to this address 2023-05-04 16:06:39,236 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: java.net.SocketException: Broken pipe (Write failed) 2023-05-04 16:06:44,239 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 10.1.4.2:9082; will use this address for sending heartbeat messages 2023-05-04 16:06:44,240 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at 10.1.4.2:9082. Will send Cluster Connection Request to this address 2023-05-04 16:06:44,373 WARN [Process Cluster Protocol Request-19] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from 10.1.4.2 due to Extended key usage does not permit use for TLS client authentication the web version does not start, such an error hangs and does not go further The Flow Controller is initializing the Data Flow. statute of nifi and zookeeper on all servers zookeepe asset in quorum ... View more
Labels:

How compression works in nifi from kafka json

by Explorer in Support Questions
‎05-02-2023 11:18 PM
‎05-02-2023 11:18 PM
How compression works in nifi I have a consumercafcarerecord(jsontreereader+parquetrecordsetwriter)--mergecontent --putndfs(none compression) at my output, parquet files are read by spark, the problem is that the output from json is not large x2, but when I do snappy compression, the file namefile.parquet.snappy shrinks x5x6 at the output but the file cannot be opened by spark or it opens and the structure is no longer as good as it was at step 1 how to get the same structure from step 1 and x5x6 compression? ... View more
Labels:

Re: Kafka-->Nifi--parquet--->HDFS

by Explorer in Support Questions
‎04-28-2023 05:05 AM
‎04-28-2023 05:05 AM
another interesting point is how to implement on updateattribute the function of checking whether it was delivered to puthdfs because if hdfs runs out locally, the process continues to go and the files are not written but are thrown out of the queue and go to another file in the basket, in fact, files are lost if the meso runs out and the chain does not stop , you need to check if the file in ndfs did not arrive stop the stream or the memory ran out in ndfs stop putndfs and let the recycle bin fill up ... View more

Re: Kafka-->Nifi--parquet--->HDFS

by Explorer in Support Questions
‎04-28-2023 03:17 AM
‎04-28-2023 03:17 AM
is it possible to make three processes per kafka and so that the output data is not repeated? ... View more

Re: Kafka-->Nifi--parquet--->HDFS

by Explorer in Support Questions
‎04-27-2023 10:20 PM
‎04-27-2023 10:20 PM
yes, it helped me, it's a pity that there is no built-in functionality for puthdfs ... View more

Re: Kafka-->Nifi--parquet--->HDFS

by Explorer in Support Questions
‎04-27-2023 01:12 AM
‎04-27-2023 01:12 AM
this method does not work, puthdfs creates a directory with the extension directory1.parquet and inside the directory there is still a file without extension, I tried like this /user/test/${ now():toNumber()}.parquet, but I need to have files in the same test directory and not parquet directories ... View more

Re: Kafka-->Nifi--parquet--->HDFS

by Explorer in Support Questions
‎04-26-2023 08:19 AM
‎04-26-2023 08:19 AM
puthdfs process can file give transform parquet to HDFS or do i need putparquet to HDFS? ... View more

Re: Kafka-->Nifi--parquet--->HDFS

by Explorer in Support Questions
‎04-26-2023 08:00 AM
‎04-26-2023 08:00 AM
I figured out this error, then I have a problem, I get files in HDFS, but they are without the parquet extension, I don’t understand what kind of files these are if I put them everywhere without compression ... View more
Contact Me
Online
Offline
Last Visited
‎05-22-2023 05:44 AM
Public Statistics
Date Registered ‎04-19-2023 03:44 AM
Last Visited ‎05-22-2023 05:44 AM
Total Messages Posted 19
Total Kudos Received 2