Member since
12-12-2023
8
Posts
0
Kudos Received
0
Solutions
01-24-2024
07:03 AM
I created a template to convert data from querying Postgres database and then convert it into xml. I want to run it by using command line arguments. Are there any way to do this in NiFi? Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
01-10-2024
10:41 AM
Hi guys! I have a question about Avro schema. My input flowfile are in json format and I want to convert it into xml. The method I use is using ConvertRecord Processor. In terms of ConvertRecord Processor, I use JsonTreeReader as record reader and XMLRecordSetWriter as record writer. Following picture is the properties of JsonTreeReader I set it up. Here is the json I want to convert: {
"article":{
"article-meta": {
"art_id": "me08001",
"title" : [ {
"lang" : "en",
"title" : "A"
}, {
"lang" : "hr",
"title" : "B"
} ]
},
"journal-meta": {
"journal_id": "me9",
"type" : "journal"
}
} If I have a lot of flowfile that contain this kind of format Json file, what Avro schema should I put inside AvroSchemaRegistry. I try a lot of online converter and it doesn't satisfy AvroSchemaRegistry when I do validation. I need the Avro schema so that the ConvertRecord can read my JSON flowfile. Thank you in advance
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
12-14-2023
06:52 AM
Another things is that if I found some record contain data that doesn't exist in my table(art_title). I don't know why. I can confirm my art_title table only contain 500 rows for sure.
... View more
12-14-2023
06:49 AM
1. These data is static and there are only 500 rows in the table. I don't need to modify them in the future and I don't need to keep processing them. I only need to get all 500 rows from table and then done. 2. I just need to process them once. Do I need to set the schedule time longer so that the ExecuteSQL only process once and this once will contain 500 rows?
... View more
12-14-2023
06:23 AM
By the way , on the upper right corner of NiFi page, there are some errors that I can't understand. Can you explain it to me? Thank you so much!!!
... View more
12-14-2023
06:17 AM
but it looks like still a lot of information, not only 500 flowfiles in the queue. Above pic is my configuration.
... View more
12-13-2023
10:14 AM
Thank you for your reply. Yes, "queue will have a lot of flowfile and it will mess up" means that there are a lot of data that I don't really want. I just want the 500 row in my table. My flow so far is ExecuteSQL and then split avro and then convert to json and then convert to xml. I want each flow file have one row in my table. For the executeSQL, do I need to set up schedule time? If I run executeSQL, will I get exactly 500 flowfile? Thank you in advance.
... View more
12-12-2023
11:31 AM
Hi, Guys. I am new to use NiFi flow. I am kind of confused about Query database table processor. So now, I am try to get all data from Postgres database and convert them into json and then convert them into xml. But sometimes, the Query database table processor will have a lot of Flow file and then the queue just mess up. I try to set the art_id to be maximum-value column but it doesn't work well because art_id is string.(not sure) This is a screenshot from a sample table called title and it has about 500 rows. How to query them just one time and split them into 500 flowfile and each of flowfile has only one row from table title without any duplicates. Can someone help me? Thank you so much. Following screenshots are my progress so far.
... View more
Labels:
- Labels:
-
Apache NiFi