hello, guys! I need to transfer about 3 billion rows to the database from impala, (about 100 million rows = 1 GB) my stream: executesqlrecord-putdatabaserecord, but the data ends at 600 million rows. please tell me the solution to the problem
... View more
Hello, guys! Would you tell me, how can I check a CSV file for the number and order of fields, the presence of headers, the absence of quotes in headers and values? nifi experts, @MattWho, @kumar, @hegdemahendra, please help!
... View more
hello! I have a CSV files, a lot of CSV files from sftp. I need to get the store name from the filename and append that to one of the columns. The filenames are very different, number of characters before and after changes for ex., ftp_123245_ECCO_160921.csv ftp_1984_remont_msk_and_spb_200721.csv taxcom_2345_ashan_17122021_2.csv also I need to get the date from the filename and append that to one of the columns as you can see there are also in different formats, I need update it to 'yyyy-mm-dd' THANKS
... View more
Hey! I built the process of getting data from s3, but now I have another problem, when converting, part of the data is lost and not written to the SQL database, and I can’t understand where my error, there are my settings. If you help me it will be very great. I use split because without it there is a lot of data and Nifi drops UpdateAttribute I use because some of the values are taken from the path where the files lie on s3
... View more
I'm just starting to learn NiFi. I need to read parquet data from s3 bucket, I don’t understand how to set up lists3 and fetchs3object processors for reading data. full path looks like this: s3://inbox/prod/export/date=2022-01-07/user=100/cro.parquet I'll write data to sql database - I don't have problems with it, but not sure)) I tried to configure the lists3 processor myself and I think is not very good bucket inbox aws_access_key_id aws_secret_access_key region US EAST endpoint override URL http://s3.wi-fi.ru:8080
... View more
Hi! I'm trying to set up a data transfer with Ni-Fi. I need to transfer data from postgresql to Clickhouse. I'm asking for help in configuring processors. There is also a problem with connecting Nifi to Clickhouse.
... View more
