Member since
02-07-2019
2690
Posts
235
Kudos Received
30
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1151 | 04-15-2025 10:34 PM | |
3336 | 10-28-2024 12:37 AM | |
1436 | 09-04-2024 07:38 AM | |
3280 | 06-10-2024 10:24 PM | |
1391 | 02-01-2024 10:51 PM |
06-15-2023
11:41 AM
This simple InvokeScriptedProcessor will look for a FlowFile attribute called "ip_address" and will attempt the reverse lookup and create a new attribute called "host_name" with the resolved value. import groovy.json.JsonOutput
import groovy.json.JsonSlurper
import java.net.InetAddress
import java.net.UnknownHostException
import java.nio.charset.StandardCharsets
import org.apache.commons.io.IOUtils
class GroovyProcessor implements Processor {
PropertyDescriptor BATCH_SIZE = new PropertyDescriptor.Builder()
.name("BATCH_SIZE")
.displayName("Batch Size")
.description("The number of incoming FlowFiles to process in a single execution of this processor.")
.required(true)
.defaultValue("1000")
.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
.build()
Relationship REL_SUCCESS = new Relationship.Builder()
.name("success")
.description('FlowFiles that were successfully processed are routed here')
.build()
Relationship REL_FAILURE = new Relationship.Builder()
.name("failure")
.description('FlowFiles that were not successfully processed are routed here')
.build()
ComponentLog log
void initialize(ProcessorInitializationContext context) { log = context.logger }
Set<Relationship> getRelationships() { return [REL_FAILURE, REL_SUCCESS] as Set }
Collection<ValidationResult> validate(ValidationContext context) { null }
PropertyDescriptor getPropertyDescriptor(String name) { null }
void onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue) { }
List<PropertyDescriptor> getPropertyDescriptors() { Collections.unmodifiableList([BATCH_SIZE]) as List<PropertyDescriptor> }
String getIdentifier() { null }
JsonSlurper jsonSlurper = new JsonSlurper()
JsonOutput jsonOutput = new JsonOutput()
def reverseDnsLookup(String ipAddress) {
try {
InetAddress inetAddress = InetAddress.getByName(ipAddress)
String hostName = inetAddress.getCanonicalHostName()
return hostName
} catch (UnknownHostException e) {
return "Unknown"
}
}
void onTrigger(ProcessContext context, ProcessSessionFactory sessionFactory) throws ProcessException {
ProcessSession session = sessionFactory.createSession()
try {
List<FlowFile> flowFiles = session.get(context.getProperty(BATCH_SIZE).asInteger())
if (!flowFiles) return
Map customAttributes = [:]
flowFiles.each { flowFile ->
String ipAddress = flowFile.getAttribute("ip_address")
if (ipAddress) {
String hostName = reverseDnsLookup(ipAddress)
customAttributes["host_name"] = hostName
flowFile = session.putAllAttributes(flowFile, customAttributes)
}
session.transfer(flowFile, REL_SUCCESS)
}
session.commit()
} catch (final Throwable t) {
log.error('{} failed to process due to {}; rolling back session', [this, t] as Object[])
session.rollback(true)
throw t
}
}
}
processor = new GroovyProcessor()
... View more
06-14-2023
02:42 PM
Consume Kafka has an attribute called "Message Demarcator"...click on it hold shift+enter and instead of pulling 1 event at a time it'll create a single FlowFile with several events at a time and might make your merge even better. You can do the same thing with the Publish...merge on shift+enter and configure the same demarcator and you'll achieve greater throughput
... View more
06-14-2023
07:10 AM
@Fredb This is a very difficult one to solve. Does anyone know what would cause the execution of the sample_Import_Load.bat to run correctly from the windows command prompt, but fail when executed via the ExecuteStreamCommand processor with these errors? This is most likely caused by permission issues. Nifi requires specific permissions against files and scripts it touches or executes from within processors. As such, the error is saying the processor does not know where any of the resources exist to run that .bat file. I do not have any experience with nifi on windows, other than to avoid it, but the solution is likely the same as other operating systems. Make sure the nifi user has full ownership of the file(s). Additionally, it is sometimes possible to find deeper errors looking at the nifi-app.log file while testing and/or setting the log level of the processor to be more aggressive.
... View more
06-14-2023
03:13 AM
@ClouderaC, Welcome to our community! This post may be able to help you with your query. Hope this helps!
... View more
06-13-2023
11:36 PM
@alangularte Welcome to our community! To help you get the best possible answer, I have tagged in our Hive experts @asish @Shelton who may be able to assist you further. Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.
... View more
06-12-2023
10:47 PM
@DFrank, have you resolved your issue? If so, can you please provide the solution here, as it will make it easier for others to find the answer in the future? If you are still experiencing the issue, can you provide the information @asish has requested?
... View more
06-12-2023
10:10 PM
1 Kudo
@nuxeo-nifi, here is the link to the recording: https://youtu.be/8cZJ9CyLYyI.
Hope this helps.
... View more
06-12-2023
09:03 PM
Tried follow accordingly, not working at the moment hit the same issue. Will look into other configuration.
... View more
06-12-2023
05:24 AM
1 Kudo
@Ansh001 Just to provide an answer here. HDP is a no longer supported product and you should not be trying to build or learn on hdp sandbox. Take a look at CDP Private Cloud Base or CDP Public Cloud. As @VidyaSargur you can start conversations around becoming a customer with sales. If you are looking for self-service you can get a trial here of CDP Public Cloud: https://www.cloudera.com/campaign/try-cdp-public-cloud.html
... View more
06-12-2023
01:34 AM
1 Kudo
@MukaAddA, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @SAMSAL @cotopaul @steven-matison who may be able to assist you further. Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.
... View more