Member since
05-07-2020
32
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 6765 | 03-15-2022 08:41 AM |
09-13-2022
12:34 AM
Thanks for the reply @nramanaiah. I seem to be unable to find an option 'metastore.msck.repair.enable.partition.retention' does it need to be added as a custom option and if so under which drop down? Thanks
... View more
09-08-2022
01:44 AM
I am trying to set partition retention times on existing Hive manged tables using the following: ALTER TABLE <table name> SET TBLPROPERTIES ('discover.partitions'='true'); ALTER TABLE <table name> SET TBLPROPERTIES ('partition.retention.period'='1d'); as stated on this page below, however I am still able to search partitions older than a day so it appears to not be working? It does mention on the page that this is for 'external' tables, can anyone let me know if this an 'age off' retention period is possible on managed tables? am I missing any commands etc? https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/using-hiveql/content/hive-set-partition-retention.html Thanks in advance
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
09-02-2022
03:04 AM
1 Kudo
@araujo thank you! I had gotten around the issue by using a replaceText loop to search for capitals within' the keys 1 at a time and prepend underscores, followed by a jolt to lowercase all keys. You're script returns the same result, very impressed. Thanks again.
... View more
08-22-2022
08:16 AM
I have stolen to spec below from here: https://stackoverflow.com/questions/54696540/jolt-transformation-lowercase-all-keys which works in lowercasing all keys, however I ideally need the hyphens between the words. [ { // unwrap the keys and values into literal // "key" : "A", "value" : "b" "operation": "shift", "spec": { "*": { "$": "&1.key", "@": "&1.value" } } }, { "operation": "modify-overwrite-beta", "spec": { "*": { // Now that the origional key // is on the "right hand side" // lowercase it "key": "=toLower" } } }, { // pivot back, the now lowercased keys "operation": "shift", "spec": { "*": { "value": "@(1,key)" } } } ]
... View more
08-22-2022
07:36 AM
Within my NiFi flow I have a number of datasets, with various schemas where the 'key' of key value pairs are on camelCase but i want any incoming keys to be outputted in snake case, does anyone know what jolt spec i can use to achieve this? Example input: {"aRandomFieldName":"Random Value"} Required Output: {"a_random_field_name":"Random Value"} Been struggling to achieve this using Jolt or replaceText processors, any assistance would be great. Thanks in advance Andy.
... View more
Labels:
- Labels:
-
Apache NiFi
03-15-2022
08:41 AM
Hello @Azhar_Shaikh Thanks for the reply, as it turns out it wasn't a service account problem. We found that the ListS3's output included a 'key' field, and this is what was required in The FetchS3Object processor for 'Object Key'. So the fix I applied was to split the json into individual records (SplitJson), then pull the keys out as attributes (EvaluateJsonPath) then input ${key} into the FetchS3 processor.. worked a treat.
... View more
03-11-2022
01:31 AM
Hello I am having an issue retrieving bucket contents into NiFi using the FetchS3Object processor. I have configured the ListS3 processor to pull in a json array containing all the bucket information, I'm happy that's working ok... example output: I have configured the FetchS3Object processor with the same Accesskey, Secret etc but I get the following error, I haven't been able to find much online ref this error, can anyone see where I'm going wrong? FetchS3Object Config: Any assistance would be greatly appreciated, I've seen a lot of questions asked about these two processors but haven't found anything about this particular error. Cheers Andy
... View more
Labels:
- Labels:
-
Apache NiFi
01-10-2022
06:21 AM
Thanks Matt!! I hadn't used the advanced tab on the processor until now, worked a treat for my use case. Apologies for the slightly confusing question. I am now able to pick out key words from the hostname attribute to direct them to the correct database. Thanks for the quick response!
... View more
01-10-2022
04:15 AM
Hello all Within NiFi, updateAttribute processor I am trying to change an attribute called 'hive_database' based on the value of another attribute called 'hostname'. For instance if: #1: hostname = Mickey Mouse #2: hostname = James Bond I want to use a 'contains' statement (or similar ) to change the output of new attribute 'hive_database' property: hive_database value: ${hostname:contains('Mickey'):<OUTPUT>('cartoon')} <OR> ${hostname:contains('Mickey'):<OUTPUT>('movie')} So that #1 output would be: property= hive_database value= cartoon The parts I'm stuck on is the OUTPUT and OR parts above, do you know if this would be possible using the update attribute processor? I am trying to avoid having to use routeOnAttribute to break out multiple different 'hosname' values to send data to the relevant 'hive_database' Hope this makes sense, unable to share screenshots etc as sensitive vales. Any help greatly appreciated.
... View more
Labels:
11-25-2021
03:18 AM
Thanks for the reply, is there a way to use Jolt to extract to flowfile attribute?
... View more
- « Previous
-
- 1
- 2
- Next »