About Gubbi

Gubbi · ‎09-11-2020

@stevenmatison : I tried below flow and getting this error. ConvertRecord[id=7d5d0bb5-0174-1000-1a76-dc9d3a7b1b35] Failed to process StandardFlowFileRecord[uuid=4d10880f-3ce2-475a-ac53-cca01f190600,claim=,offset=0,name=userdata1.parquet,size=0]; will route to failure: org.apache.nifi.parquet.stream.NifiParquetInputFile@17badbbc is not a Parquet file (too small length: 0) Below is my ConvertRecord settings. For reading file from location and i want to convert into CSV and transfer to other location, i have built flow ListFile-->ConvertRecord-->FetchFile-->PutFile. Can you please check if the above settings is correct and flow is correct for my requirement

Gubbi · ‎09-10-2020

I haven't worked on Parquet file but i have requirement to convert Parquet file to CSV using NiFi. Appreciate if you could guide me on this.

Gubbi · ‎08-10-2020

NiFi services is shutting down abruptly with below exception in production environment. So i tried few things to troubleshoot, when i remove existing flow.xml.gz and restart services i see NiFi UI works as expected. But when i try using flow.xml.gz from archive (latest) and restart i get below error. Not sure what is causing issue here..Appreciate quick response as its impacting production run. Log file nifi-app.log doesnt have any error. Below is error from hs_err_pid*.log Internal exceptions (10 events): Event: 25.581 Thread 0x00007f6cef29e800 Exception <a 'java/lang/NoSuchMethodError': java.lang.Object.lambda$comparingInt$7b0bb60$1(Ljava/util/function/ToIntFunction;Ljava/lang/Object;Ljava/lang/Object;)I> (0x00000005bdf0e618) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/ Event: 25.665 Thread 0x00007f6cef29e800 Implicit null exception at 0x00007f6e7efdb410 to 0x00007f6e7efddb85 Event: 25.665 Thread 0x00007f6cef29e800 Implicit null exception at 0x00007f6e7d49a697 to 0x00007f6e7d49a7b5 Event: 25.784 Thread 0x00007f6ceea03800 Exception <a 'sun/nio/fs/UnixException'> (0x00000005c2dc39b8) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.784 Thread 0x00007f6ceea03800 Exception <a 'sun/nio/fs/UnixException'> (0x00000005c2dc3f60) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.794 Thread 0x00007f6a1c001000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005d838d148) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.794 Thread 0x00007f6a1c001000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005d838d6f0) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.832 Thread 0x00007f6a64002800 Implicit null exception at 0x00007f6e7e330862 to 0x00007f6e7e330959 Event: 25.832 Thread 0x00007f6b08005000 Implicit null exception at 0x00007f6e7e330862 to 0x00007f6e7e330959 Event: 25.839 Thread 0x00007f6b08005000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005db870240) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Bootstrap error log : 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut # An error report file with more information is saved as: 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut # /users/quant_hub/nifi/nifi-1.11.1/hs_err_pid20683.log 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140348715132672 also had an error] 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349382018816 also had an error] 2020-08-10 10:16:51,393 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349099882240 also had an error] 2020-08-10 10:16:51,402 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349347297024 also had an error] 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # If you would like to submit a bug report, please visit: 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # http://bugreport.java.com/bugreport/crash.jsp 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # The crash happened outside the Java Virtual Machine in native code. 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # See problematic frame for where to report the bug. 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # 2020-08-10 10:16:59,250 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-08-10 10:16:59,327 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is running at PID 20683 but is not responding to ping requests 2020-08-10 10:17:10,140 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi @dfheinz @MattWho

Gubbi · ‎04-05-2020

Issue resolved. Had to re-install NiFi and services are up and running.

Gubbi · ‎04-04-2020

Hi, We recently deployed NiFi to production server (04/03) and executed few dataflows. But after few hours, it automatically shut down and when i tried to stop and re-start, it's not running. There are no errors in log file nifi-app.log. And in bootstrap log i just see these details. Could you please tell what is going wrong here. Bootstrap log file : 2020-04-04 00:28:24,785 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-04-04 00:28:24,791 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-04-04 00:28:24,857 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is not running 2020-04-04 00:28:31,274 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-04-04 00:28:31,328 INFO [main] org.apache.nifi.bootstrap.RunNiFi Runtime Java version: 1.8.0_211 2020-04-04 00:28:31,334 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2020-04-04 00:28:31,335 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /users/quant_hub/nifi/nifi-1.11.1 2020-04-04 00:28:31,335 INFO [main] org.apache.nifi.bootstrap.Command Command: /users/java/jdk1.8.0_211/bin/java -classpath /users/quant_hub/nifi/nifi-1.11.1/./conf:/users/quant_hub/nifi/nifi-1.11.1/./lib/log4j-over-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jcl-over-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-framework-api-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-properties-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jul-to-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/logback-core-1.2.3.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/slf4j-api-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-nar-utils-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/javax.servlet-api-3.1.0.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/logback-classic-1.2.3.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jetty-schemas-3.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-api-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-runtime-1.11.1.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx4096m -Xms2048m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dzookeeper.admin.enableServer=false -Dnifi.properties.file.path=/users/quant_hub/nifi/nifi-1.11.1/./conf/nifi.properties -Dnifi.bootstrap.listen.port=39921 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/users/quant_hub/nifi/nifi-1.11.1/logs org.apache.nifi.NiFi

Gubbi · ‎04-02-2020

@stevenmatison : Changing the Max Thread Count in Admin->Controller Services? Max Timer Driven--16 , Max Event Driven --1 Setting Concurrent Tasks in processor scheduling tab? - Set to 1 as default Increasing Min/Max Ram? - Set to default Below is my Linux configuration : Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 8 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz Stepping: 4 CPU MHz: 2294.609 BogoMIPS: 4589.21 Hypervisor vendor: VMware Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0-7

Gubbi · ‎04-01-2020

@stevenmatison @MattWho : I am still facing issue where NiFi is crashing when CPU >100%. While checking the logs i get only this error. And once CPU % comes down , GUI comes back again. But when i trigger the flow, same issue occurs. Btw, dataflow is to transfer files ( <50MB) from Linux to S3 with no complex logic. Could you please suggest. 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-4] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=0170100e-e28a-1807-249c-3f5fee9fdb0e] with Flow Registry because could not retrieve version 1 of flow with identifier 42614870-ace3-42fa-a719-01148884d252 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-1] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=01701029-e28a-1807-8de5-812b10e812f4] with Flow Registry because could not retrieve version 1 of flow with identifier 043dc40c-7f7a-4ed3-b3b5-dffcc716a332 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,003 ERROR [Framework Task Thread Thread-3] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=0170101b-e28a-1807-55d0-ba1fc7c21ae5] with Flow Registry because could not retrieve version 1 of flow with identifier 555068bb-a5b6-4506-bbe4-25023010c5e5 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-2] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=01701000-e28a-1807-386c-9c1498a492f7] with Flow Registry because could not retrieve version 1 of flow with identifier 13163161-190b-40fb-8dd9-b9adf6c55245 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,345 INFO [main] o.a.nifi.wali.LengthDelimitedJournal 21.25% of the way finished recovering journal ./flowfile_repository/journals/2460243665.journal, having recovered 69338 updates

Gubbi · ‎04-01-2020

Currently i am using ${now():format("yyyyMMdd")} to extract today's date and passing it to next flow. But i would need Yesterday's and day before's date also in similar way. How can i achieve this. Appreciate quick response on this.

Gubbi · ‎04-01-2020

@stevenmatison : On similar thread, can i do match for 2-3 days and push files to respective folder. Like ${filename:endsWith(${today -1})} and ${filename:endsWith(${today -2})}. Basically i want to keep checking 2 days previous files along with current date and push to respective folder based on date condition. And how will i transfer those files previous day files to target date folder. Currently i am using /users/${now():format("yyyyMMdd")} to push today's file. If i have to do it for previous 2 days. how will it change.

Gubbi · ‎03-27-2020

Ignore my previous reply. Solution which you provided is working. Thanks a lot.

Online	Offline
Last Visited	‎05-07-2021 10:12 AM

Member Since	‎02-25-2020 09:46 AM
Last Visited	‎05-07-2021 10:12 AM
Posts	35
Kudos received	1

Cloudera Community

Re: NiFi instances not running

Re: Fetch File unable to copy files from NAS drive

Re: Transfer files to S3 based on file timestamp

Re: Convert Parquet file to CSV using NiFi

Convert Parquet file to CSV using NiFi

NiFi shutting down abruptly

Re: NiFi instances not running

NiFi instances not running

Re: Nifi process consuming >100%CPU while executi...

Re: Nifi process consuming >100%CPU while executi...

How to derive yesterday's date using now expressio...

Re: Transfer files based on the YYYYMMDD in their ...

Re: Transfer files based on the YYYYMMDD in their ...