Member since
07-30-2019
2909
Posts
1443
Kudos Received
846
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
55 | 04-23-2024 05:56 AM | |
27 | 04-22-2024 06:13 AM | |
150 | 04-17-2024 11:30 AM | |
119 | 04-16-2024 05:36 AM | |
77 | 04-15-2024 05:31 AM |
06-28-2023
12:31 PM
@agrayush Once you secure NiFi and/or NiFi-Registry (configured for HTTPS), MutualTLS based authentication will always be supported. When you access the HTTPS URL for either service, in the TLS exchange the service (NiFi or NiFi-Registry will "WANT" a client auth certificate). When a client certificate is NOT provided, the services will attempt to authenticate the user/client via another configured Authentication method. My guess here is that when you originally secured your NiFi and NiFi-Registry services, you used the TLS toolkit to create your user/client certificate which you then loaded into your browser. When you accessed the service, the browser presented that client certificate (depending on browser you may have even been prompted by the browser to confirm using the certificate). At this point the browser retains your certificate. preference for the target URL(s). Now that you have configured another authentication method, the browser is still going to present that certificate and the service us going to take it. You can not disable client certificate authentication as it is the only supported auth method for connecting between nodes in a NiFi cluster and NiFi authentication with NiFi-Registry. I suggest you remove the sys_admin certificate from your browser, clear all cookies/site data from your browser, and and then restart the service again. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
06-25-2023
12:21 PM
Apache PDFBox is supposed to allow you to merge PDF content. Since this is a Java library you can create a scripted Groovy processor to merge the files for you. https://pdfbox.apache.org https://javadoc.io/doc/org.apache.pdfbox/pdfbox/2.0.27/index.html
... View more
06-21-2023
04:47 PM
Thank you @MattWho, the trick worked! Sadly, it worked for this simple case, but for more complex cases we couldn't find a pattern to apply it. However, we found another workaround by using literals like this: Expression Value { "${literal('$${key1}')}": "${literal('$${value1}')}", "key2": "${literal('$${value2}')}", "${literal('$${key3}')}": "value3", "key4": "${literal('value4')}" } { "${key1}": "${value1}", "key2": "${value2}", "${key3}": "value3", "key4": "value4" } It's a nasty hack, but it gets the job done! Thanks again for your help and for filling a bug report.
... View more
06-16-2023
09:58 AM
@Kiranq 1. What version of NiFi and NiFi-Registry are you using? 2. How is your GitFlowPersistenceProvider configured? Matt
... View more
06-16-2023
06:25 AM
@MOUROU Is your NiFi configured to support Oauth2 based user authentication? It looks more like you are using either kerberos-provider or ldap-provider fro user authentication. My suggestion to create a client certificate and use a SSLContext service for client authentication for an automated dataflow like this is because: 1. No need to obtain an token ever. 2. Certs can be created with long expiration time. 3. Tokens are NiFi node specific (same token is not valid for a different NiFi node in a the same NiFi cluster). 4. Same certs works no matter which node in the cluster the invokeHTTP connects with. Matt
... View more
06-15-2023
11:41 AM
This simple InvokeScriptedProcessor will look for a FlowFile attribute called "ip_address" and will attempt the reverse lookup and create a new attribute called "host_name" with the resolved value. import groovy.json.JsonOutput
import groovy.json.JsonSlurper
import java.net.InetAddress
import java.net.UnknownHostException
import java.nio.charset.StandardCharsets
import org.apache.commons.io.IOUtils
class GroovyProcessor implements Processor {
PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
.name("BATCH_SIZE")
.displayName("Batch Size")
.description("The number of incoming FlowFiles to process in a single execution of this processor.")
.required(true)
.defaultValue("1000")
.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
.build()
Relationship REL_SUCCESS = new Relationship.Builder()
.name("success")
.description('FlowFiles that were successfully processed are routed here')
.build()
Relationship REL_FAILURE = new Relationship.Builder()
.name("failure")
.description('FlowFiles that were not successfully processed are routed here')
.build()
ComponentLog log
void initialize(ProcessorInitializationContext context) { log = context.logger }
Set<Relationship> getRelationships() { return [REL_FAILURE, REL_SUCCESS] as Set }
Collection<ValidationResult> validate(ValidationContext context) { null }
PropertyDescriptor getPropertyDescriptor(String name) { null }
void onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue) { }
List<PropertyDescriptor> getPropertyDescriptors() { Collections.unmodifiableList([BATCH_SIZE]) as List<PropertyDescriptor> }
String getIdentifier() { null }
JsonSlurper jsonSlurper = new JsonSlurper()
JsonOutput jsonOutput = new JsonOutput()
def reverseDnsLookup(String ipAddress) {
try {
InetAddress inetAddress = InetAddress.getByName(ipAddress)
String hostName = inetAddress.getCanonicalHostName()
return hostName
} catch (UnknownHostException e) {
return "Unknown"
}
}
void onTrigger(ProcessContext context, ProcessSessionFactory sessionFactory) throws ProcessException {
ProcessSession session = sessionFactory.createSession()
try {
List<FlowFile> flowFiles = session.get(context.getProperty(BATCH_SIZE).asInteger())
if (!flowFiles) return
Map customAttributes = [:]
flowFiles.each { flowFile ->
String ipAddress = flowFile.getAttribute("ip_address")
if (ipAddress) {
String hostName = reverseDnsLookup(ipAddress)
customAttributes["host_name"] = hostName
flowFile = session.putAllAttributes(flowFile, customAttributes)
}
session.transfer(flowFile, REL_SUCCESS)
}
session.commit()
} catch (final Throwable t) {
log.error('{} failed to process due to {}; rolling back session', [this, t] as Object[])
session.rollback(true)
throw t
}
}
}
processor = new GroovyProcessor()
... View more
06-15-2023
05:56 AM
1 Kudo
@noncitizen Welcome to the Community!!! Apache NiFi is a very large open source project. Over the 8+ years since it was originally open sources it has grown so large that the download distribution has reached the max allowable size and does not include all components that the community has developed for NiFi. There are more then 400+ unique components developed for NiFi and growing every year. Many of these "add-on" components can be found in various open source repositories and NiFi makes it every easy to add them to your NiFi (even hot loading is possible). As is true with many open source products with lots of contributors, the documentation usually comes after the development and may at times be lacking in detail. Sometimes this because the originator could not anticipate all the possible use cases for a given component or being so close to the development there is good amount of self inferred knowledge and understanding. I myself have been working with NiFi for more then 8 years and have been exposed to many use cases, bugs, improvements, etc. I look forward to seeing you more in the community as you learn and grow and begin to help others using that new found knowledge.
... View more
06-15-2023
02:22 AM
1 Kudo
Thanks @MattWho I created https://issues.apache.org/jira/browse/NIFI-11695 Hoping for the best!
... View more
06-14-2023
02:42 PM
Consume Kafka has an attribute called "Message Demarcator"...click on it hold shift+enter and instead of pulling 1 event at a time it'll create a single FlowFile with several events at a time and might make your merge even better. You can do the same thing with the Publish...merge on shift+enter and configure the same demarcator and you'll achieve greater throughput
... View more