Member since
04-17-2023
17
Posts
0
Kudos Received
0
Solutions
06-01-2023
06:52 AM
Dears, I have a CSV File with around 5lakh records and I am inserting these 5 lakh records into Singlestore database through putDataBaseRecord Processor It is taking around 15 minutes to insert all these 5 lakh records, I am also setting the property Maximum Batch Size to 10000 to fasten the insertion but it is still taking 15 minutes. Can you suggest me any approach to fasten the insertion of these 5 lakh records through putDataBaseRecord Processor. Thanks In Advance!
... View more
Labels:
- Labels:
-
Apache NiFi
05-12-2023
04:28 AM
@ushasri, If you are reading an Excel File, you can use an ConvertRecord processor, in which you define an Record Reader (CSVReader most likely) where you tell NiFi how to read the value (value separator). Next, with the help of a Record Writer (I assume CSV again or maybe Avro) you define how your data should look like, including the header. For both the Reader and the Writer you will have to define a schema, where you can do the replace of blank space with underline.
... View more
05-04-2023
06:30 AM
You do not install the Cloudera version on your laptop 🙂 You need the Cloudera DataFlow for Public Cloud (CDF-PC), meaning that we are talking here about a license and some services. As @steven-matison already provided you with the perfect answer for your question, he might also be in the position to further assist you with everything you need to know about the Cloudera Data Flow and their Public Cloud. Unfortunately I am still learning about what Cloudera offers and how, so I am not the best one to answer your question. If you are going to use NiFi for some real data processing, I strongly recommend you to have a look to Cloudera Data Flow, as this will solve many issues and headaches 🙂
... View more
05-04-2023
05:39 AM
1 Kudo
@ushasri Since you describe going from 1.20.0-RC1 to 1.15.3-RC1 it is safe to assume those features are not available in 1.15.3. ( Right click on canvas --> Enable all control services & disable all control services) are some of the newer UI features.
... View more
04-25-2023
04:15 AM
1 Kudo
The file can be found within your NiFi logs folder, where you have your other logs as well.
... View more
04-24-2023
12:38 AM
hi @ushasri, The file you are looking for is called flow.xml.gz and can be found on each NiFi node, within the conf folder. More information can be found here: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html flow.xml.gz: Everything the DFM puts onto the NiFi User Interface canvas is written, in real time, to one file called the flow.xml.gz. This file is located in the nifi/conf directory by default. Any change made on the canvas is automatically saved to this file, without the user needing to click a "Save" button. In addition, NiFi automatically creates a backup copy of this file in the archive directory when it is updated. You can use these archived files to rollback flow configuration. To do so, stop NiFi, replace flow.xml.gz with a desired backup copy, then restart NiFi. In a clustered environment, stop the entire NiFi cluster, replace the flow.xml.gz of one of nodes, and restart the node. Remove flow.xml.gz from other nodes. Once you confirmed the node starts up as a one-node cluster, start the other nodes. The replaced flow configuration will be synchronized across the cluster. The name and location of flow.xml.gz, and auto archive behavior are configurable. See the System Administrator’s Guide for further details.
... View more
04-20-2023
11:00 PM
Hi @ushasri, There are plenty of ways to perform such a thing, but here are three ways I have already implemented the alert in my flows so far: You can have an PutEmail at the end of each failure Queue and in that why, when a message gets rejected you will get notified by email. This can be done strictly in NiFi. You can create a monitoring system which will interrogate your Bulletin Board using NiFi's REST API and send you whatever alerts you want. This can be done either directly inside NiFi or with a shell/python script on your server. You tail the nifi-app.logs on each server and extract only the errors you are interested in. This solution can be implemented both in NiFi and on your local server.
... View more
04-20-2023
01:13 AM
Dears, How to insert same number to all the records coming from a csv file in nifi and increment that number while inserting second csv file. Can you knidly suggest me an approach to achieve this. Thank you.
... View more
Labels:
- Labels:
-
Apache NiFi
04-18-2023
05:07 AM
hi @ushasri, What processor are you using to save the data within your DB and how do you build up the query? If you want to add the filename to the record which gets inserted within your table, you should use an UpdateRecord Processor to generate a new column and populate it using the ${filename}. Next, the generated record should go further down your line up until it gets inserted in your database.
... View more
04-18-2023
02:41 AM
hi @ushasri, Can you provide some more details about your flow? Without knowing what you are doing in your flow, I can only tell you that you can use the Expression Language from NiFi and extract the current time send it into your stream. The current time can be called as an example like: ${now():toNumber():format('yyyy-MM-dd')} Next, you can use an UpdateRecord Processor, add the attribute to your newly defined column and send it to further processing. More about NiFi's Expression Language: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
... View more