Member since
06-22-2022
7
Posts
0
Kudos Received
0
Solutions
06-23-2022
12:35 AM
I am using Nifi 1.6.0. I am reading a JSON file from s3, de-compressing it, making some changes to this file, compressing it and re-uploading it to S3. I am trying to upload the file to S3 using PutS3, but I came across this error: The compressed file content is 5 MB in size. I am using CompressContent Processor with the following configuration: I have tried the same flow on a smaller file (in a few KBs) and that file was uploaded without any errors. How do I fix this error?
... View more
Labels:
- Labels:
-
Apache NiFi
06-22-2022
06:56 PM
@araujo I will try this solution, but I think the queue to PutDatabaseRecord processor will get clogged very soon because there are over 14 million rows in this file. I tried Replace text and I was able to replace '},' to '}' within seconds. Can you tell me how to remove the first line '[' and the last line ']'? Thanks
... View more
06-22-2022
06:45 PM
I have not created that part of the flow yet, but that will be an ExecuteStreamCommand processor which will run a psql command with the copy sql query: It will look something like this:
... View more
06-22-2022
06:16 PM
@araujo , Right now there is no error in Nifi flow. But when I tried to execute a copy command (from an SQL query editor) on the file, it gave me an error "Invalid JSONPath format: Member is not an object. ". When I searched about this error, I found this - https://stackoverflow.com/a/45348425 This is why I wanted to convert the file.
... View more
06-22-2022
05:57 PM
@araujo The jsons files are compressed and are placed in an S3 bucket. Here's the flow I have created to fetch s3 file and decompress it.
... View more
06-22-2022
05:52 PM
@araujo Sorry, my bad. I made a mistake in the json representation mentioned in the question. The records actually have double quotes ("). I have updated the question.
... View more
06-22-2022
06:14 AM
I am using Nifi 1.6.0. I am trying to copy an S3 file into redshift. The json file on S3 looks like this: [ {"a":1,"b":2}, {"a":3,"b":4} ] However this gives an error because of '[' and ']' in the file (https://stackoverflow.com/a/45348425). I need to convert the json from above format to a format which looks like this: {"a":1,"b":2} {"a":3,"b":4} So basically what I am trying to do is to remove '[' and ']' and replace '},' with '}'. The file has over 14 million rows (30 MB in size) How can I achieve this using Nifi?
... View more
Labels:
- Labels:
-
Apache NiFi