I am running nifi flow in minifi in containerized environment. I am using executesqlrecord to fetch data from bigquery.It takes almost 3 hours to fetch 14GB data from bigquery table . Is there any way by which I can reduce the fetch time . Environment : Google Compute Engine  Docker container  MiniFi 0.5.0   ... View more

I want to transform a NiFi Flow to Minifi Flow using transform command by minifi toolkit . I observe that property values are not getting copied to yaml and I have to manually add them like password , file path etc. Is there any way to handle this in automated fashion ? Any help would be appreciable . TIA ... View more

Hi Team, we have observed that PUTGCSOBJECT processor in NiFi uploads the flowfile in one request which lowers down the upload speed for that file transfer . we tried to use the GSUTIL utility to upload the same file with resumable uploads option and the speed was 10 times fasted as it performs multiple parallel uploads with the following command gsutil -D -o GSUtil:parallel_composite_upload_threshold=100M cp <source file path> gs://<destination cloud bucket name>/ I tried with the ExecuteProcess processor to execute the above command but processor is not able to parse the command and landing into the parsing error. Is some one aware about this performance or anybody has any alternative approach for nifi to upload the file in parts and in parallel and then perform compose operation as gsutils does ? ... View more

I am trying to fetch big full table with 10M+ records from hive using selectHiveQL Processor and did find that the converttoCSVStream() method in the source code takes longer time than fetching the result set . Observing the code : Result set is being iterated row by row and then added to the output stream . When the table size is small it completes the process in seconds but as the data is large it takes longer time . Is there any way we can optimize the conversion ?? I have tried with the fetch size of 100000/1000/10000/1000 . while (rs.next()) { //logger.info("+++++++++++++Inside the While loop+++++++++++++++++"); if (callback != null) { callback.processRow(rs); } List<String> rowValues = new ArrayList<>(nrOfColumns); for (int i = 1; i <= nrOfColumns; i++) { final int javaSqlType = meta.getColumnType(i); final Object value = rs.getObject(i); //logger.info("+++++++++++++Entering the Switch at +++++++++++++++++"); switch (javaSqlType) { case CHAR: case LONGNVARCHAR: case LONGVARCHAR: case NCHAR: case NVARCHAR: case VARCHAR: String valueString = rs.getString(i); if (valueString != null) { // Removed extra quotes as those are a part of the escapeCsv when required. StringBuilder sb = new StringBuilder(); if (outputOptions.isQuote()) { sb.append("\""); if (outputOptions.isEscape()) { sb.append(StringEscapeUtils.escapeCsv(valueString)); } else { sb.append(valueString); } sb.append("\""); rowValues.add(sb.toString()); } else { if (outputOptions.isEscape()) { rowValues.add(StringEscapeUtils.escapeCsv(valueString)); } else { rowValues.add(valueString); } } } else { rowValues.add(""); } break; case ARRAY: case STRUCT: case JAVA_OBJECT: String complexValueString = rs.getString(i); if (complexValueString != null) { rowValues.add(StringEscapeUtils.escapeCsv(complexValueString)); } else { rowValues.add(""); } break; default: if (value != null) { rowValues.add(value.toString()); } else { rowValues.add(""); } } //logger.info("+++++++++++++Exiting the Switch at +++++++++++++++++" + System.currentTimeMillis()); } //List<String> rowValues1 = new ArrayList<>(); //rowBuffer.add(StringUtils.join(rowValues, outputOptions.getDelimiter())); //rowBuffer.add("\n"); // Write row values //logger.info("+++++++++++++Writing Row value at+++++++++++++++++" + System.currentTimeMillis()); outStream.write(StringUtils.join(rowValues, outputOptions.getDelimiter()).getBytes(StandardCharsets.UTF_8)); outStream.write("\n".getBytes(StandardCharsets.UTF_8)); nrOfRows++; /* if (rowBuffer.size() == BATCH_SIZE) { writeToOutStream( outStream,rowBuffer ); } */ if (maxRows > 0 && nrOfRows == maxRows) break; } ... View more

@Bryan Bende @Matt Burgess @Timothy Spann @Abdelkrim Hadjidj Hi , I am trying to upload a 4gb file to azure blob storage but I can see that the 4 gb file is getting uploaded in smaller files of 128mb and 64 mb .What property is splitting the 4gb file in 128mb or 64 . I have seen some of the properties like provenance.rollover.size =100mb but not sure if this splits the file . I want to understand why this big file is getting uploaded in blocks of 128Mb and not as a whole of 4gb in on file . ... View more

Is there any way where I can capture the total time taken by PutAzureBlobStorage processor to upload all the files to azure blob storage . ... View more

I have the following settings in the schedule tab of nifi selectHiveQl Processor settings.png schedule.png And I am using "select * from tablename" query in my selectHiveQlProcessor with fetchsize of 20000 and max number of rows in flow files as 20000. As soon as i schedule the selectHiveQl processor and check the app.log I can see continuous parse statements being printed in the log like below : 2018-05-15 06:29:54,919 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:54,919 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,063 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,063 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,213 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,213 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,356 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,356 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,501 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,502 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,634 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,634 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,764 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,764 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,914 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,914 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:56,046 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:56,046 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:56,195 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:56,195 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:56,334 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:56,335 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed It should be parsed one time only , multiple parsing might be using CPU more than what is expected . Can some one help me understand how to prevent this parsing multiple times . @Matt Burgess@Bryan Bende ... View more