Member since
10-31-2021
11
Posts
0
Kudos Received
0
Solutions
01-12-2024
12:30 AM
Hello @LKB , Just saw your message. Since I have changed my job, I am not able to access those files anymore. Sorry for that.
... View more
02-23-2022
11:54 PM
Hi @araujo, Thank you for your answer. Our Kafka servers are always designed to consist of 100 partitions in my company and each partition belongs to a data of an id. I explicitly generate the partition value attribute according to an id before going into Publish Kafka processor. So, there is no chance that some values could not be valid partitions. Also, I dont know why but every flow file with same partition number does not result in same partition. This one confused me. Maybe, hash function returns a value more than 100 and that value may be processed randomly in the kafka servers since we have only 100 partitions. As a last ditch solution and if it makes sense for the community, I may open a pr to add another config in publish kafka processor whether partition value should be hashed or not.
... View more
02-23-2022
05:22 AM
Hello everyone, I want to use the Publish Kafka processors and I need to specify the partition for each flow file. But when I choose Expression Language Partitioner class and add my partition info on Partition field, it does not publish my records to appropriate partitions. I realize that, instead, it gets my partition string and hashes. Then it publishes the records to partition with hashed value. I dont understand why it works like this. I want to choose a spesific partition for each flow file. How can I achieve this? Here is the code from nifi github repo. As we can see it hashes the partition value: private Integer getPartition(final ProcessContext context, final FlowFile flowFile) {
final String partitionClass = context.getProperty(PARTITION_CLASS).getValue();
if (EXPRESSION_LANGUAGE_PARTITIONING.getValue().equals(partitionClass)) {
final String partition = context.getProperty(PARTITION).evaluateAttributeExpressions(flowFile).getValue();
final int hash = Objects.hashCode(partition);
return hash;
}
return null;
} https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-2-0-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/PublishKafka_2_0.java
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
11-03-2021
05:55 AM
Hi everyone, I'm newbie to Nifi. I have already configured and build my test server. I have different DB connections in my NiFi. However, our DB connection passwords are dynamic in my company. Because we are using CyberArk security system, our DB user passwords are not constant. Our applications send a kind of query to cyberark system to fetch the password on the fly and use that password to connect to DBs. My question is how to do this on a NiFi system. Can we configure the sensitive data on the controllers dynamically?
... View more
Labels:
- Labels:
-
Apache NiFi
11-02-2021
07:19 AM
Thanks a lot Matt. It worked finally after trying for 2 weeks. I was tired of trying to get this to work. I really appreciate it.
... View more
11-01-2021
12:43 AM
Hi, Im trying to connect LDAP from Nifi. I've done most of the instructions in the official documentation and read most posts on the Cloudera Community. However, I couldn't connect to my company's LDAP system. I configured login-entity-providers and authorizers for LDAP. When I start Nifi, it starts successfully and gives no error. When I try to log in with my initial admin it gives me "Unable to validate the supplied credentials. Please contact the system administrator." error. My questions are: 1- When we are setting up initial admin, we are not setting password of that user. How do we log in without password? 2- I'm using USE_DN for identity strategy for ldap-provider. My initial admin name and ldap user are same. So I'm trying to login with my user like: Username: CN=K015576,CN=Users,DC=tcmb,DC=gov,DC=tr Password: "my_ldap_password" What path should I follow? Is there a mistake in my configure files? I have added my conf files below: nifi.properties, login-entity-provider and authorizers. Removed sensitive data. nifi-properties login-identity-providers authorizers-1 authorizers-2
... View more
Labels:
- Labels:
-
Apache NiFi