@MattWho Thank you for your input ... View more

Apache PDFBox is supposed to allow you to merge PDF content. Since this is a Java library you can create a scripted Groovy processor to merge the files for you.  https://pdfbox.apache.org https://javadoc.io/doc/org.apache.pdfbox/pdfbox/2.0.27/index.html ... View more

@MOUROU  Is your NiFi configured to support Oauth2 based user authentication? It looks more like you are using either kerberos-provider or ldap-provider fro user authentication. My suggestion to create a client certificate and use a SSLContext service for client authentication for an automated dataflow like this is because: 1. No need to obtain an token ever. 2. Certs can be created with long expiration time. 3. Tokens are NiFi node specific (same token is not valid for a different NiFi node in a the same NiFi cluster). 4. Same certs works no matter which node in the cluster the invokeHTTP connects with. Matt ... View more

This simple InvokeScriptedProcessor will look for a FlowFile attribute called "ip_address" and will attempt the reverse lookup and create a new attribute called "host_name" with the resolved value.   import groovy.json.JsonOutput import groovy.json.JsonSlurper import java.net.InetAddress import java.net.UnknownHostException import java.nio.charset.StandardCharsets import org.apache.commons.io.IOUtils class GroovyProcessor implements Processor { PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder() .name("BATCH_SIZE") .displayName("Batch Size") .description("The number of incoming FlowFiles to process in a single execution of this processor.") .required(true) .defaultValue("1000") .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR) .build() Relationship REL_SUCCESS = new Relationship.Builder() .name("success") .description('FlowFiles that were successfully processed are routed here') .build() Relationship REL_FAILURE = new Relationship.Builder() .name("failure") .description('FlowFiles that were not successfully processed are routed here') .build() ComponentLog log void initialize(ProcessorInitializationContext context) { log = context.logger } Set<Relationship> getRelationships() { return [REL_FAILURE, REL_SUCCESS] as Set } Collection<ValidationResult> validate(ValidationContext context) { null } PropertyDescriptor getPropertyDescriptor(String name) { null } void onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue) { } List<PropertyDescriptor> getPropertyDescriptors() { Collections.unmodifiableList([BATCH_SIZE]) as List<PropertyDescriptor> } String getIdentifier() { null } JsonSlurper jsonSlurper = new JsonSlurper() JsonOutput jsonOutput = new JsonOutput() def reverseDnsLookup(String ipAddress) { try { InetAddress inetAddress = InetAddress.getByName(ipAddress) String hostName = inetAddress.getCanonicalHostName() return hostName } catch (UnknownHostException e) { return "Unknown" } } void onTrigger(ProcessContext context, ProcessSessionFactory sessionFactory) throws ProcessException { ProcessSession session = sessionFactory.createSession() try { List<FlowFile> flowFiles = session.get(context.getProperty(BATCH_SIZE).asInteger()) if (!flowFiles) return Map customAttributes = [:] flowFiles.each { flowFile -> String ipAddress = flowFile.getAttribute("ip_address") if (ipAddress) { String hostName = reverseDnsLookup(ipAddress) customAttributes["host_name"] = hostName flowFile = session.putAllAttributes(flowFile, customAttributes) } session.transfer(flowFile, REL_SUCCESS) } session.commit() } catch (final Throwable t) { log.error('{} failed to process due to {}; rolling back session', [this, t] as Object[]) session.rollback(true) throw t } } } processor = new GroovyProcessor() ... View more

@noncitizen  Welcome to the Community!!! Apache NiFi is a very large open source project.  Over the 8+ years since it was originally open sources it has grown so large that the download distribution has reached the max allowable size and does not include all components that the community has developed for NiFi.  There are more then 400+ unique components developed for NiFi and growing every year.  Many of these "add-on" components can be found in various open source repositories and NiFi makes it every easy to add them to your NiFi (even hot loading is possible).   As is true with many open source products with lots of contributors, the documentation usually comes after the development and may at times be lacking in detail.  Sometimes this because the originator could not anticipate all the possible use cases for a given component or being so close to the development there is good amount of self inferred knowledge and understanding.  I myself have been working with NiFi for more then 8 years and have been exposed to many use cases, bugs, improvements, etc.   I look forward to seeing you more in the community as you learn and grow and begin to help others using that new found knowledge. ... View more

Thanks @MattWho I created https://issues.apache.org/jira/browse/NIFI-11695 Hoping for the best! ... View more

Consume Kafka has an attribute called "Message Demarcator"...click on it hold shift+enter and instead of pulling 1 event at a time it'll create a single FlowFile with several events at a time and might make your merge even better.  You can do the same thing with the Publish...merge on shift+enter and configure the same demarcator and you'll achieve greater throughput   ... View more

Tracking here: https://issues.apache.org/jira/browse/NIFI-11682 ... View more

Tried follow accordingly, not working at the moment hit the same issue. Will look into other configuration. ... View more