Hi All, In source we are getting 31 files for every 5 minutes and in Nifi we used ListSFTP processor to get the files and its scheduled to run for every 5 minutes .All the files are not getting picked some times,lets say for 10th hr 15th min its processing only 25 files and not picking other 6 files even though file name format and permissions are same. All the 31 files for each 5 mint window are landing at same time . If the same file copied again(file which was not picked by listsftp initially),its picking . Any recommendations to fix this . We are using  Listing Strategy as  Tracking Timestamps. @Nifi @ListSFTP ... View more

Hi All, I have table where it have multiple columns along with time column which is  GMT epochtime(ex. 1584081205316).Now i want to perform aggregations on every 15 minutes records for previous day.  Currently i wrote script which gives last 15 minutes data whenever we execute ,but in prod whenever there is issue we will loose data with current query.Please share your thoughts or if you have any approach or solutions  ... View more

ExecuteStreamCommand is failing with below error "failed to write flow file to stdin due to broken pipe in NIFI" . I also attached the error screenshot    Note: When we process our flow with single source file we are getting output generated but when we have multiple source files then only one output file is generated with all the data and rest all other output files doesn't have any data. Please let me know how to fix this issue ... View more

Hi Team,   I'm executing query(insert select ..) which have one left join and cross join .Its creating 5 stages out of which 4 stages are completing and in 5th stage map tasks are not getting progressing (all tasks are at 0.00). It have total 11 map tasks and there is no reducers . Job have sufficient resources and memory to utilize . There is no exceptions or errors in both attempt and map tasks log . We can see only below info in attempt log  2019-10-01 03:34:52,057 INFO [IPC Server handler 14 on 9493] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1568245027185_28090_m_000007_0 is : 0.0 below info from map task syslog    2019-10-01 04:00:41,013 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: MAP[7]: records read - 10000 2019-10-01 04:08:36,216 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[5]: records written - 100000000   note : Since we are unable to use non equal conditions on join gone with cross join We can able to fetch records with select clause but when we add insert query on top of it its getting stuck  Tried same query without cross join(since impala accepting non conditions) but result is same ,working with select clause but getting stuck while inserting    Could please let us know what causing the issue and also alternative for using non equal conditions in join ON ... View more