Member since
05-07-2020
32
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4506 | 03-15-2022 08:41 AM |
09-13-2022
12:34 AM
Thanks for the reply @nramanaiah. I seem to be unable to find an option 'metastore.msck.repair.enable.partition.retention' does it need to be added as a custom option and if so under which drop down? Thanks
... View more
09-08-2022
01:44 AM
I am trying to set partition retention times on existing Hive manged tables using the following: ALTER TABLE <table name> SET TBLPROPERTIES ('discover.partitions'='true'); ALTER TABLE <table name> SET TBLPROPERTIES ('partition.retention.period'='1d'); as stated on this page below, however I am still able to search partitions older than a day so it appears to not be working? It does mention on the page that this is for 'external' tables, can anyone let me know if this an 'age off' retention period is possible on managed tables? am I missing any commands etc? https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/using-hiveql/content/hive-set-partition-retention.html Thanks in advance
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
09-02-2022
03:04 AM
1 Kudo
@araujo thank you! I had gotten around the issue by using a replaceText loop to search for capitals within' the keys 1 at a time and prepend underscores, followed by a jolt to lowercase all keys. You're script returns the same result, very impressed. Thanks again.
... View more
08-22-2022
08:16 AM
I have stolen to spec below from here: https://stackoverflow.com/questions/54696540/jolt-transformation-lowercase-all-keys which works in lowercasing all keys, however I ideally need the hyphens between the words. [ { // unwrap the keys and values into literal // "key" : "A", "value" : "b" "operation": "shift", "spec": { "*": { "$": "&1.key", "@": "&1.value" } } }, { "operation": "modify-overwrite-beta", "spec": { "*": { // Now that the origional key // is on the "right hand side" // lowercase it "key": "=toLower" } } }, { // pivot back, the now lowercased keys "operation": "shift", "spec": { "*": { "value": "@(1,key)" } } } ]
... View more
08-22-2022
07:36 AM
Within my NiFi flow I have a number of datasets, with various schemas where the 'key' of key value pairs are on camelCase but i want any incoming keys to be outputted in snake case, does anyone know what jolt spec i can use to achieve this? Example input: {"aRandomFieldName":"Random Value"} Required Output: {"a_random_field_name":"Random Value"} Been struggling to achieve this using Jolt or replaceText processors, any assistance would be great. Thanks in advance Andy.
... View more
Labels:
- Labels:
-
Apache NiFi
03-15-2022
08:41 AM
Hello @Azhar_Shaikh Thanks for the reply, as it turns out it wasn't a service account problem. We found that the ListS3's output included a 'key' field, and this is what was required in The FetchS3Object processor for 'Object Key'. So the fix I applied was to split the json into individual records (SplitJson), then pull the keys out as attributes (EvaluateJsonPath) then input ${key} into the FetchS3 processor.. worked a treat.
... View more
03-15-2022
01:34 AM
Hi @VidyaSargur I do not see an option for 'Accept as Solution' below the post, I assume because I didn't ask the original question. Regards
... View more
03-11-2022
01:31 AM
Hello I am having an issue retrieving bucket contents into NiFi using the FetchS3Object processor. I have configured the ListS3 processor to pull in a json array containing all the bucket information, I'm happy that's working ok... example output: I have configured the FetchS3Object processor with the same Accesskey, Secret etc but I get the following error, I haven't been able to find much online ref this error, can anyone see where I'm going wrong? FetchS3Object Config: Any assistance would be greatly appreciated, I've seen a lot of questions asked about these two processors but haven't found anything about this particular error. Cheers Andy
... View more
Labels:
- Labels:
-
Apache NiFi
03-10-2022
08:46 AM
@araujo Thanks very useful indeed!
... View more
03-10-2022
01:34 AM
@araujo Thanks André What if I wanted to turn the key value pairs separated by '=' in json content as in the original question? for instance the following syslog: converted into: [ { "sig" : "0", "arch" : "c000003e", "syscall" : "87" }] I'm aware this can be done using regex to create attributes and then attributeToJson but some of my logs have hundreds of key value pairs so that's not an option, there must be a way to convert it using record processing? i.e convertRecord
... View more