Member since
05-07-2020
32
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4053 | 03-15-2022 08:41 AM |
01-25-2023
02:13 AM
We using PutHive3Streaming processor to send data to Hive from NiFi, I have an issue where we are getting LOTS of small delta files on our busier feeds which is causing issues with compaction etc. I have used a series of merges in NiFi to ensure each flowfile contains many thousand records but it still creates many delta files. I wondered if anyone had any advice on tuning 'Records Per Transaction' and 'Transactions per Batch' options on the PutHive3Streaming processor, I believe this could help with my issue but have had mixed/confusing results from testing. There isn't a great deal of information on best practice that I have found. Has anyone else had similar issues/found adjustments helpful?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
11-02-2022
01:57 AM
Hello all, NiFi Jolt question Does anyone know if it's possible to convert timestamp formats using Jolt using 'field.value' the way you can using update Record? I'm aware that I can use an attribute value in Jolt to convert a field, like: "date" : "${timestamp:toDate('yyyy-MM-dd HH:mm:ss'):toNumber()}" but I was wondering if this function can be done using Jolt on incoming 'field.values', have tried using the default spec as below but it doesn't change the format. Thanks in advance. Andy
... View more
Labels:
- Labels:
-
Apache NiFi
10-31-2022
09:33 AM
@steven-matison I have been trying to get ifElse working for me but the below gives me an empty string > "" And this gives me null as a string > "null" is there a way to return null not as a string?
... View more
10-31-2022
02:16 AM
Hi @steven-matison Here is a flow file example, as you can see the date field is only present in the array some of the time, what I want to achieve is, if the date is present I need it in epoch. If the date is not present for the field to be 'null,' (not "null" as a sting/in brackets) or ignored altogether. { "example": "cloudera", "risk_rating": "low", "observed" false, "sources": [ { "date": "2020-06-12T12:00:00.000Z", "unique_id": "some_value", "source_name": "cloudera", "url": "https://www.cloudera.com", "description": "example_log", },{ "unique_id": "some_other_value", "source_name": "google", "url": "https://www.google.com", "description": "example_log_without_date" }], "report_confidence": "confirmed", "base_score" 7.8, "authentication": "none" }
... View more
10-28-2022
02:39 AM
Hello all I have a slightly annoying problem in NiFi, I have some JSON records that have a field 'date' nested under 'sources' that I want to turn to Epoch time. Using updateRecord I am able to do this using the following config: As you can see where this field is present it works, however sometimes this field is not present, in these cases the record value is blank: To do deal with these blank field I did some searching online and found the following 'isBlank' could be used in updateRecord property to deal with blank fields: The outcome of this fixes the blank field issue... but, doesn't convert the present field values to epoch, see below: Does anyone know how I can get these two actions to perform at the same time? Thanks in advance. Andy
... View more
Labels:
- Labels:
-
Apache NiFi
10-14-2022
02:58 AM
Hi @Fredi Can you send a screenshot of the advanced tab rules if possible, I'm a little confused as to what you are wanting to achieve. Cheers
... View more
10-03-2022
07:16 AM
@nramanaiah have been able to run further testing and confirm that my partitions are purging as expected! thanks again for the assistance!
... View more
09-27-2022
01:53 AM
@nramanaiah I haven't had a chance to do further testing yet, I will let you know ASAP. Thanks again for the help.
... View more
09-20-2022
04:46 AM
@nramanaiah I'm still experiencing some issues with this, I have applied metastore.msck.repair.enable.partition.retention=true and restarted, all looks good as below. I have applied the ALTER table statements to set the retention of 1 day to a test table without error, but when I do a SELECT statement in Beeline I can still see data from last week? Any idea what I'm missing?
... View more
09-14-2022
08:40 AM
@nramanaiah thanks very much for the help!
... View more