Member since
06-26-2015
502
Posts
121
Kudos Received
113
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
236 | 09-20-2022 03:33 PM | |
782 | 09-19-2022 04:47 PM | |
410 | 09-11-2022 05:01 PM | |
509 | 09-06-2022 02:23 PM | |
627 | 09-06-2022 04:30 AM |
10-09-2022
04:01 PM
@Althotta , I tested this on 1.16.2 and the behaviour you described doesn't happen to me. Would you be able to share you flow and processor/controller services configuration? Cheers, André
... View more
09-28-2022
03:04 AM
1 Kudo
You can get the id of the root process group and import the template there as well. André
... View more
09-28-2022
12:23 AM
1 Kudo
@Kushisabishii , Which version of NiFi are you using? There's an API endpoint for this: POST /process-groups/{id}/templates/upload Cheers, André
... View more
09-28-2022
12:18 AM
Can you share your settings?
... View more
09-21-2022
03:24 PM
Is your dev cluster running the exact same version of NiFi as production, including the NiFi lib folder?
... View more
09-20-2022
03:33 PM
1 Kudo
@progowl , Yes, it is. Check out the docker compose configuration in this article: https://community.cloudera.com/t5/Community-Articles/NiFi-cluster-sandbox-on-Docker/ta-p/346271 Cheers, André
... View more
09-19-2022
04:47 PM
@SAMSAL @ChuckE , I believe parsing the schema for each flowfile that goes through the processor would be too expensive. Because of that, the schema is parsed only once when the processor is scheduled and used for every flowfile. That's why the attribute values cannot be used for this property. Having a schema hashmap<filename, parsed_schema> internally could be an interesting idea so that the processor would parse the schema onTrigger only once for every schema file name and reuse it afterwards. Obviously memory usage could be a problem if you have too many schemas, but I don't think this is likely to happen. This doesn't happen currently, but it would be a nice feature request IMO. Currently, you can either do that with a scripting processing or use RouteOnAttribute to send each message to a ValidateXML processor with the correct schema. Cheers, André
... View more
09-11-2022
05:01 PM
@sekhar1 , The CDP user that you're using to execute your job needs an "IDBroker mapping" to a valid AWS role to be able to access the contents of the S3 bucket. Please check this: https://docs.cloudera.com/cdf-datahub/7.2.10/nifi-hive-ingest/topics/cdf-datahub-hive-ingest-idbroker-mapping.html Cheers, André
... View more
09-07-2022
05:04 AM
1 Kudo
Everything you do in the NiFi UI can also be done using the NiFi REST API. So if you want/need to automate it, it's totally possible and not difficult. Cheers, André
... View more
09-07-2022
03:57 AM
@SandyClouds , Have a look at Parameters and Parameters Contexts: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Parameters Cheers André
... View more
09-06-2022
03:24 PM
@rafy , Same approach. Just create the processor(s) and let the files flow through it 🙂 Cheers, André
... View more
09-06-2022
02:23 PM
@rafy You shouldn't need to extract the content as an attribute. Instead, use ReplaceText to replace the contents of the flowfile with the SQL template, like this: INSERT INTO my_table (xml_col) VALUES ('$1') Where $1 is a reference to the default regex capture group that captures the entire content. In reality, it requires a bit more labour, since you have to escape the single quotes in the XML first. Cheers, André
... View more
09-06-2022
02:17 PM
@VenkatG , What are you trying to use this resulting JSON for? That seems a bit odd of a format to me. Nevertheless, here's a JOLT to achieve that (or close): [
{
"operation": "default",
"spec": {
"__timestamp": "${now()}"
}
},
{
"operation": "shift",
"spec": {
"custId": "Rows[0].values[0]",
"name": "Rows[0].values[1]",
"Address2": "Rows[0].values[2]",
"Address1": "Rows[0].values[3]",
"zip": "Rows[0].values[4]",
"__timestamp": "Rows[0].values[5]"
}
},
{
"operation": "default",
"spec": {
"operationType": "Insert",
"Source": "Dev"
}
}
] Cheers, André
... View more
09-06-2022
04:30 AM
1 Kudo
Hi, @code , I don't think this is actually possible. Even if there was a way to enter a literal value of NULL for the lookup value, the controller service is probably comparing the lookup value with an "equals" operation (e.g. mytable.mykey = lookup_value) and in relational databases the comparison NULL = NULL is always evaluated to FALSE. (the only way to compare values with a NULL is to use the operator IS). What you can try to do is to create a view on top of that table that converts NULLs to some string that you can use to compare to lookup values in NiFi. Then you can use the view name in the controller service configuration instead of the table name. For example: CREATE VIEW v_mytable AS
SELECT
NVL(mykey, '<NULL>') as key_without_nulls, *
FROM mytable Be aware of potential performance implications of this, since this could prevent existing table indexes from being used for lookups. Cheers, André
... View more
09-06-2022
04:16 AM
@Markyd , Functions used in Hive queries are usually passed to the Hive service as is. I don't think the ODBC driver does any particular processing or validation of those functions. At the end of the day it's a matter of whether the Hive version you're using supports that function or not, and this depends on the Hive version being used, not the driver version. Here you can see a comprehensive list of Hive functions and the associated Hive version where they were introduced: https://cwiki.apache.org/confluence/display/hive/languagemanual+udf Which version of Hive are you using? Are you using the Cloudera ODBC Drivers for Hive? Which version? Cheers, André
... View more
09-05-2022
03:54 PM
@Ahmed_Abuhaimed , Have you looked into change data capture functionality for Hana? https://help.sap.com/docs/SAP_DATA_INTELLIGENCE/1c1341f6911f4da5a35b191b40b426c8/023c75aedfdd4646934f2d9ccde5660a.html Cheers, André
... View more
09-05-2022
03:48 PM
@samrathal , Is seems that "subtract" only works for integers and your value is a long. The longSubtract function only works if longs are passed as parameters and I don't know if there's a way to specify a long literal in Jolt (I tried 60L but that doesn't work). The following does a little bit more work but also achieves what you want: [
{
"operation": "modify-overwrite-beta",
"spec": {
"currenttime": "=toLong(${now():toNumber()})",
"minute": "=toLong(60)"
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"timeOneMinu": "=longSubtract(@(1,currenttime), @(1,minute))"
}
},
{
"operation": "remove",
"spec": {
"minute": ""
}
}
] Output: {
"currenttime": 1662417935173,
"timeOneMinu": 1662417935113
} Cheers, André
... View more
09-05-2022
03:09 PM
@samrathal , What about this: [
{
"operation": "default",
"spec": {
"currenttime": "${now():toNumber()}",
"timeOneMinu": "${now():toNumber():minus(60)}"
}
}
]
... View more
09-05-2022
05:37 AM
@samrathal , Doesn't your JOLT spec solve your own problem? Cheers, André
... View more
09-04-2022
10:21 PM
@syntax_ , It seems to me that your source Kafka is a Confluent Kafka cluster and the producer uses schema registry to source the schema from it. In this case, the KafkaAvroSerializer prepends 5 bytes to every message produced to indicate the id of the schema that was used (in you case, schema id 34). If you try to read this message as a pure Avro payload the deserialization will fail because those 5 bytes are not part of the Avro payload. So, the best way to handle this in NiFi is to also use Schema Registry to deserialize Avro messages. With this, NiFi will get the schema ID from the message 5-byte prefix, use that ID to retrieve the correct schema from Schema Registry and then correctly deserialize the Avro payload. Considering that my guess is correct and you're using a Confluent Schema Registry, you should create a new ConfluentSchemaRegistry controller service and configure it with the details of your Schema Registry. Once this is done, edit the configuration of the Avro Reader controller service and set the following: After you do this, your flow should be able to correctly process the messages that you're reading from Kafka. I read the binary message that you send me with NiFi and loaded the schema in my local schema registry service (making sure it got assigned the right ID 34), and I was able to successfully convert the message from Avro to JSON using a ConvertRecord processor: Cheers, André
... View more
09-04-2022
06:15 PM
@Uday483 , The error above happens if you don't specify the domain during authentication, right? If you do specify the domain, does it work? André
... View more
09-04-2022
05:32 PM
@hebamahmoud , Did you try enabling the dependency between Kudu and Impala, as per my post above? Cheers, André
... View more
09-04-2022
05:29 PM
@syntax_ , Please try running this command: xxd message.avro Then you can copy and paste the output here. Cheers, André
... View more
09-02-2022
03:18 AM
Ok. Did you try the ldap configuration I mentioned above? Cheers André
... View more
09-02-2022
03:16 AM
Can you please send me that file in a private message. Copy and paste won't work 🙂 Cheers, André
... View more
09-01-2022
10:25 PM
@hebamahmoud , Try enabling the Kudu Service as a dependency for the Impala service, as shown below. When you do that Impala will use the selected Kudu service as the default service whenever you use a Kudu table that does not explicitly sets the Kudu masters. If you don't set the property above you can still use Kudu from Impala but when you create the table you have to specify the Kudu masters through the TBLPROPERTIES clause. Cheers, André
... View more
09-01-2022
10:15 PM
@ajignacio , What's the output of the command below if you run it from the same machine where NiFi is running? openssl s_client -connect ldap.dev.abcde:389 I know you are not using TLS, but the command above can still give us some useful information. Cheers, André
... View more
09-01-2022
10:09 PM
I actually wanted to have a look at the binary Avro data that is in Kafka, not the deserialized content. Something like this: kafka-console-consumer --from-beginning --bootstrap-server admin:9092 --topic pbs_jobs --max-messages 1 > message.avro Cheers, André
... View more