Hi, We recently deployed NiFi to production server (04/03) and executed few dataflows. But after few hours, it automatically shut down and when i tried to stop and re-start, it's not running. There are no errors in log file nifi-app.log. And in bootstrap log i just see these details. Could you please tell what is going wrong here.   Bootstrap log file : 2020-04-04 00:28:24,785 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-04-04 00:28:24,791 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-04-04 00:28:24,857 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is not running 2020-04-04 00:28:31,274 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-04-04 00:28:31,328 INFO [main] org.apache.nifi.bootstrap.RunNiFi Runtime Java version: 1.8.0_211 2020-04-04 00:28:31,334 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2020-04-04 00:28:31,335 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /users/quant_hub/nifi/nifi-1.11.1 2020-04-04 00:28:31,335 INFO [main] org.apache.nifi.bootstrap.Command Command: /users/java/jdk1.8.0_211/bin/java -classpath /users/quant_hub/nifi/nifi-1.11.1/./conf:/users/quant_hub/nifi/nifi-1.11.1/./lib/log4j-over-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jcl-over-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-framework-api-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-properties-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jul-to-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/logback-core-1.2.3.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/slf4j-api-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-nar-utils-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/javax.servlet-api-3.1.0.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/logback-classic-1.2.3.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jetty-schemas-3.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-api-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-runtime-1.11.1.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx4096m -Xms2048m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dzookeeper.admin.enableServer=false -Dnifi.properties.file.path=/users/quant_hub/nifi/nifi-1.11.1/./conf/nifi.properties -Dnifi.bootstrap.listen.port=39921 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/users/quant_hub/nifi/nifi-1.11.1/logs org.apache.nifi.NiFi ... View more

Currently i am using ${now():format("yyyyMMdd")} to extract today's date and passing it to next flow. But i would need Yesterday's and day before's date also in similar way. How can i achieve this. Appreciate quick response on this. ... View more

@MattWho :  I'm finding a way to get the content of a file based on its filename and it should drop files to target based on date. . All of the target files are in the same directory, but I'd like to select only the files which has "{Today's YYYYMMDD} as their postfix. For example, if today is 20200326,  /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326 -> create folder with 20200326 and drop file. /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326-> Drop file into above folder /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200325-> not ok because this YYYYMMDD is not today.   Approach i followed : ListFile: Extract only the files whose name ends with "gz.20200326" UpdateAttribute: Substitute YYYYMMDD into an Attribute "today"     RouteOnAttribute: Extract only the file which matches ${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})} (where $filename is the target file name) and pass it as Relationship "target_file"   But when executing, looks like it's not matching my requirement and all files are matching. Could you please assist me on this. ... View more

I have NAS mounted on Linux server and have to copy files from location to S3, i am using ListFile→FetchFile-→PutS3.  While reading files from NAS mount location, i am getting error saying file not found but i see all required files are present at location. Is there any special configuration which needs to be set while reading files from NAS drive. ... View more

Used case :  Need to transfer entire folder of files based on date to S3. Ex :  Source Path (Linux) : /users/abc/20200312/gtry/gyyy.csv             S3    : /Users/datastore/20200312/gyyy.csv   Since dates keep changing everyday, and i need to build dataflow which would pick files from date folder ... View more

I have used case where i am reading files with timestamp and these files has to be transferred to S3 and create folder with respective dates. Ex: file names abcd.out.gz.20200303 , abcd.out.gz.20200302 and the file abcd.out.gz.20200303 need to be in S3 under /data/20200303 and file abcd.out.gz.20200302 under /data/20200302.   How can i achieve this in NiFi.  ... View more

I have triggered dataflow and its running everyday. For 1st run, ListFile is able to list files and pass it on to next starge FetchFile. But when i touch file in my input directory, ListFile is unable to list those files.  ... View more

i'm new to NiFi and have developed simple dataflow which would read 3 files from on-prem and upload into S3 bucket. i am using ListFile, FetchFile and PutS3 processor groups for this case. When i trigger dataflow, CPU consumption is spiking >100% and NiFi gets crashed.After few minutes. it automatically comes down to normal. This is happening frequently. My onprem server is Linux and has 8 CPUs. Any idea what's going on  with this. ... View more