Re: NiFi instances not running

Gubbi · ‎09-11-2020

@stevenmatison : I tried below flow and getting this error. ConvertRecord[id=7d5d0bb5-0174-1000-1a76-dc9d3a7b1b35] Failed to process StandardFlowFileRecord[uuid=4d10880f-3ce2-475a-ac53-cca01f190600,claim=,offset=0,name=userdata1.parquet,size=0]; will route to failure: org.apache.nifi.parquet.stream.NifiParquetInputFile@17badbbc is not a Parquet file (too small length: 0) Below is my ConvertRecord settings. For reading file from location and i want to convert into CSV and transfer to other location, i have built flow ListFile-->ConvertRecord-->FetchFile-->PutFile. Can you please check if the above settings is correct and flow is correct for my requirement

Gubbi · ‎09-10-2020

I haven't worked on Parquet file but i have requirement to convert Parquet file to CSV using NiFi. Appreciate if you could guide me on this.

Gubbi · ‎08-10-2020

NiFi services is shutting down abruptly with below exception in production environment. So i tried few things to troubleshoot, when i remove existing flow.xml.gz and restart services i see NiFi UI works as expected. But when i try using flow.xml.gz from archive (latest) and restart i get below error. Not sure what is causing issue here..Appreciate quick response as its impacting production run. Log file nifi-app.log doesnt have any error. Below is error from hs_err_pid*.log Internal exceptions (10 events): Event: 25.581 Thread 0x00007f6cef29e800 Exception <a 'java/lang/NoSuchMethodError': java.lang.Object.lambda$comparingInt$7b0bb60$1(Ljava/util/function/ToIntFunction;Ljava/lang/Object;Ljava/lang/Object;)I> (0x00000005bdf0e618) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/ Event: 25.665 Thread 0x00007f6cef29e800 Implicit null exception at 0x00007f6e7efdb410 to 0x00007f6e7efddb85 Event: 25.665 Thread 0x00007f6cef29e800 Implicit null exception at 0x00007f6e7d49a697 to 0x00007f6e7d49a7b5 Event: 25.784 Thread 0x00007f6ceea03800 Exception <a 'sun/nio/fs/UnixException'> (0x00000005c2dc39b8) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.784 Thread 0x00007f6ceea03800 Exception <a 'sun/nio/fs/UnixException'> (0x00000005c2dc3f60) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.794 Thread 0x00007f6a1c001000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005d838d148) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.794 Thread 0x00007f6a1c001000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005d838d6f0) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.832 Thread 0x00007f6a64002800 Implicit null exception at 0x00007f6e7e330862 to 0x00007f6e7e330959 Event: 25.832 Thread 0x00007f6b08005000 Implicit null exception at 0x00007f6e7e330862 to 0x00007f6e7e330959 Event: 25.839 Thread 0x00007f6b08005000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005db870240) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Bootstrap error log : 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut # An error report file with more information is saved as: 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut # /users/quant_hub/nifi/nifi-1.11.1/hs_err_pid20683.log 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140348715132672 also had an error] 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349382018816 also had an error] 2020-08-10 10:16:51,393 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349099882240 also had an error] 2020-08-10 10:16:51,402 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349347297024 also had an error] 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # If you would like to submit a bug report, please visit: 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # http://bugreport.java.com/bugreport/crash.jsp 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # The crash happened outside the Java Virtual Machine in native code. 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # See problematic frame for where to report the bug. 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # 2020-08-10 10:16:59,250 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-08-10 10:16:59,327 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is running at PID 20683 but is not responding to ping requests 2020-08-10 10:17:10,140 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi @dfheinz @MattWho

Gubbi · ‎06-02-2020

Currently i am having simple dataflow which has ListFile-->FetchFile-->PutS3. This flow reads data from source and uploads it to S3. Now, i have source directory /users/data/ which has abc_mmddyyy.pdf, xyz_mmddyyy.pdf and hij_mmddyyyy.pdf which uploads into S3 folder \aws\data\ . We are planning to source new files which would be published in /users/data/demo/www_mmddyyyy.csv and has to be uploaded to \aws\data\demo. How will i achieve this @stevenmatison

Gubbi · ‎04-05-2020

Issue resolved. Had to re-install NiFi and services are up and running.

Gubbi · ‎04-04-2020

Hi, We recently deployed NiFi to production server (04/03) and executed few dataflows. But after few hours, it automatically shut down and when i tried to stop and re-start, it's not running. There are no errors in log file nifi-app.log. And in bootstrap log i just see these details. Could you please tell what is going wrong here. Bootstrap log file : 2020-04-04 00:28:24,785 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-04-04 00:28:24,791 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-04-04 00:28:24,857 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is not running 2020-04-04 00:28:31,274 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-04-04 00:28:31,328 INFO [main] org.apache.nifi.bootstrap.RunNiFi Runtime Java version: 1.8.0_211 2020-04-04 00:28:31,334 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2020-04-04 00:28:31,335 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /users/quant_hub/nifi/nifi-1.11.1 2020-04-04 00:28:31,335 INFO [main] org.apache.nifi.bootstrap.Command Command: /users/java/jdk1.8.0_211/bin/java -classpath /users/quant_hub/nifi/nifi-1.11.1/./conf:/users/quant_hub/nifi/nifi-1.11.1/./lib/log4j-over-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jcl-over-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-framework-api-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-properties-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jul-to-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/logback-core-1.2.3.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/slf4j-api-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-nar-utils-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/javax.servlet-api-3.1.0.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/logback-classic-1.2.3.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jetty-schemas-3.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-api-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-runtime-1.11.1.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx4096m -Xms2048m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dzookeeper.admin.enableServer=false -Dnifi.properties.file.path=/users/quant_hub/nifi/nifi-1.11.1/./conf/nifi.properties -Dnifi.bootstrap.listen.port=39921 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/users/quant_hub/nifi/nifi-1.11.1/logs org.apache.nifi.NiFi

Gubbi · ‎04-02-2020

@stevenmatison : We dont have control on /users/data folder. Vendor would publish files in folder datewise. So there is no easy way to scan historical files without using minus expression. Minus would restrict for certain days, What if i have file a week back and i need to pick that one too.. There is no definite date when the old files would be pushed to that folder.

Gubbi · ‎04-02-2020

Used case : Transfer files from inbound directory based on date condition in filename to destination folder with date . Ex : Input directory :/user/mft/inbound/ Filename : etfbfo_20200331.csv , gfsdajaf_20200401.csv, gdfcbakfg_20200402.csv. Destination : /users/aws/outbound/20200331/ tfbfo_20200331.csv /users/aws/outbound/20200401/ gfsdajaf_20200401.csv /users/aws/outbound/20200402/gdfcbakfg_20200402.csv. @stevenmatison

Gubbi · ‎04-02-2020

I have data flow which is checking folder /users/data/${now():format("yyyyMMdd")} and it would pick all files under this folder and upload to S3 /users/aws/${now():format("yyyyMMdd")}. Suppose i get file today which is 3 days old in the folder /users/data/20200331 i need to scan that folder and pick that file as well and move to S3 user /users/aws/20200331. Note : File names doesnt have date, Files are just dropped inside date folders. @stevenmatison

Gubbi · ‎04-02-2020

@stevenmatison : Changing the Max Thread Count in Admin->Controller Services? Max Timer Driven--16 , Max Event Driven --1 Setting Concurrent Tasks in processor scheduling tab? - Set to 1 as default Increasing Min/Max Ram? - Set to default Below is my Linux configuration : Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 8 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz Stepping: 4 CPU MHz: 2294.609 BogoMIPS: 4589.21 Hypervisor vendor: VMware Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0-7

Gubbi · ‎04-01-2020

@stevenmatison @MattWho : I am still facing issue where NiFi is crashing when CPU >100%. While checking the logs i get only this error. And once CPU % comes down , GUI comes back again. But when i trigger the flow, same issue occurs. Btw, dataflow is to transfer files ( <50MB) from Linux to S3 with no complex logic. Could you please suggest. 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-4] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=0170100e-e28a-1807-249c-3f5fee9fdb0e] with Flow Registry because could not retrieve version 1 of flow with identifier 42614870-ace3-42fa-a719-01148884d252 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-1] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=01701029-e28a-1807-8de5-812b10e812f4] with Flow Registry because could not retrieve version 1 of flow with identifier 043dc40c-7f7a-4ed3-b3b5-dffcc716a332 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,003 ERROR [Framework Task Thread Thread-3] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=0170101b-e28a-1807-55d0-ba1fc7c21ae5] with Flow Registry because could not retrieve version 1 of flow with identifier 555068bb-a5b6-4506-bbe4-25023010c5e5 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-2] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=01701000-e28a-1807-386c-9c1498a492f7] with Flow Registry because could not retrieve version 1 of flow with identifier 13163161-190b-40fb-8dd9-b9adf6c55245 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,345 INFO [main] o.a.nifi.wali.LengthDelimitedJournal 21.25% of the way finished recovering journal ./flowfile_repository/journals/2460243665.journal, having recovered 69338 updates

Gubbi · ‎04-01-2020

Currently i am using ${now():format("yyyyMMdd")} to extract today's date and passing it to next flow. But i would need Yesterday's and day before's date also in similar way. How can i achieve this. Appreciate quick response on this.

Gubbi · ‎04-01-2020

@stevenmatison : On similar thread, can i do match for 2-3 days and push files to respective folder. Like ${filename:endsWith(${today -1})} and ${filename:endsWith(${today -2})}. Basically i want to keep checking 2 days previous files along with current date and push to respective folder based on date condition. And how will i transfer those files previous day files to target date folder. Currently i am using /users/${now():format("yyyyMMdd")} to push today's file. If i have to do it for previous 2 days. how will it change.

Gubbi · ‎03-27-2020

Ignore my previous reply. Solution which you provided is working. Thanks a lot.

Gubbi · ‎03-27-2020

@stevenmatison : I am not using GetFile here. i followed that approach you suggested but issue still persists. Below is dataflow i have been using. i need to route all matching files into target folder based on date. So for today's date 20200327, all files ending with _202003027 must reside in target folder 20200327. So in my fetch file i am passing this /users/FIDS_OZ_DEV/Datastore/Bloomberg/backoffice/${now():format("yyyyMMdd")}

Gubbi · ‎03-26-2020

@MattWho : I'm finding a way to get the content of a file based on its filename and it should drop files to target based on date. . All of the target files are in the same directory, but I'd like to select only the files which has "{Today's YYYYMMDD} as their postfix. For example, if today is 20200326, /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326 -> create folder with 20200326 and drop file. /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326-> Drop file into above folder /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200325-> not ok because this YYYYMMDD is not today. Approach i followed : ListFile: Extract only the files whose name ends with "gz.20200326" UpdateAttribute: Substitute YYYYMMDD into an Attribute "today" RouteOnAttribute: Extract only the file which matches ${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})} (where $filename is the target file name) and pass it as Relationship "target_file" But when executing, looks like it's not matching my requirement and all files are matching. Could you please assist me on this.

Gubbi · ‎03-26-2020

@shuhei_shogen : I had similar used case but when i tried to reproduce the same approach its not working.My file name is like equity_asia2.dif.gz.20200324 and i want these files to be in target folder 20200324. In Update Attribute i have used ${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})}. But it doesnt seem to be working. Could you please check and assist me on this.

Gubbi · ‎03-20-2020

@MattWho : Currently i am using PutS3 Object to transfer files from source to destination. I am passing secret key in property file and accessing via Credential File configuration. But our security team is not allowing this process in production as it is exposed for others to access. Can you suggest any other approach to access these keys. Appreciate quick response as we have production deployment in < 5 days.

Gubbi · ‎03-16-2020

@MattWho : Appreciate your inputs as well

Gubbi · ‎03-16-2020

@Shu_ashu : I saw you had suggested similar pattern solution before. Could you please look into this and suggest approach. https://community.cloudera.com/t5/Support-Questions/NiFi-Creating-the-output-directory-from-the-content-of-the/td-p/231335

Gubbi · ‎03-15-2020

Issue has been resolved.

Gubbi · ‎03-15-2020

I have NAS mounted on Linux server and have to copy files from location to S3, i am using ListFile→FetchFile-→PutS3. While reading files from NAS mount location, i am getting error saying file not found but i see all required files are present at location. Is there any special configuration which needs to be set while reading files from NAS drive.

Gubbi · ‎03-13-2020

Used case : Need to transfer entire folder of files based on date to S3. Ex : Source Path (Linux) : /users/abc/20200312/gtry/gyyy.csv S3 : /Users/datastore/20200312/gyyy.csv Since dates keep changing everyday, and i need to build dataflow which would pick files from date folder

Gubbi · ‎03-04-2020

I got solution for this. Had to use expression language in Object Key to fetch date from file and it worked. Below is expression :${filename:substringAfter('.gz.')}/${filename}

Gubbi · ‎03-03-2020

@AustinLiu Just to clarify, i am transferring files from linux box to S3.

Gubbi · ‎03-03-2020

@AustinLiu : But i need to transfer file abcd_20200303 to S3 folder 20200303 and respectively based on dates. Every day when the files arrive my processor should identify file based on date and push it to respective date folder in S3.

Gubbi · ‎03-03-2020

I have used case where i am reading files with timestamp and these files has to be transferred to S3 and create folder with respective dates. Ex: file names abcd.out.gz.20200303 , abcd.out.gz.20200302 and the file abcd.out.gz.20200303 need to be in S3 under /data/20200303 and file abcd.out.gz.20200302 under /data/20200302. How can i achieve this in NiFi.

Gubbi · ‎02-28-2020

Since you suggested to check on Tasks/Time taken by ListFile processor group i had mentioned 300. If you notice above screenshot Tasks shows 297. Nifi is still crashing and i am clueless on what's going on here.

Gubbi · ‎02-27-2020

Also adding to it, noticed Tasks/Time is varying drastically when new file is placed. Tasks spiked upto 300. Not sure what this means!!!!

Gubbi · ‎02-27-2020

I have triggered dataflow and its running everyday. For 1st run, ListFile is able to list files and pass it on to next starge FetchFile. But when i touch file in my input directory, ListFile is unable to list those files.

Online	Offline
Last Visited	‎05-07-2021 10:12 AM

Date Registered	‎02-25-2020 09:46 AM
Last Visited	‎05-07-2021 10:12 AM
Total Messages Posted	35
Total Kudos Received	1

Re: NiFi instances not running

Re: Fetch File unable to copy files from NAS drive

Re: Transfer files to S3 based on file timestamp

Re: Convert Parquet file to CSV using NiFi

Convert Parquet file to CSV using NiFi

NiFi shutting down abruptly

How to filter file with multiple pattern

Re: NiFi instances not running

NiFi instances not running

Re: Monitor Historical files and transfer to desti...

Transfer files based on date present in filename

Monitor Historical files and transfer to destinati...

Re: Nifi process consuming >100%CPU while executi...

Re: Nifi process consuming >100%CPU while executi...

How to derive yesterday's date using now expressio...

Re: Transfer files based on the YYYYMMDD in their ...

Re: Transfer files based on the YYYYMMDD in their ...

Re: Transfer files based on the YYYYMMDD in their ...

Transfer files based on the YYYYMMDD in their file...

Re: How to ingest files based on the YYYYMMDD in t...

Option to acces AWS3 Secret without exposing keys

Re: Move entire folder based on date to S3

Re: Move entire folder based on date to S3

Re: Fetch File unable to copy files from NAS drive

Fetch File unable to copy files from NAS drive

Move entire folder based on date to S3

Re: Transfer files to S3 based on file timestamp

Re: Transfer files to S3 based on file timestamp

Re: Transfer files to S3 based on file timestamp

Transfer files to S3 based on file timestamp

Re: Nifi process consuming >100%CPU while executi...

Re: Nifi process consuming >100%CPU while executi...

ListFile processor is not detecting all new files