Cloudera Community
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Gubbi
user rank
Contributor

Re: Convert Parquet file to CSV using NiFi

by Contributor in Support Questions
‎09-11-2020 06:58 AM
‎09-11-2020 06:58 AM
@stevenmatison : I tried below flow and getting this error. ConvertRecord[id=7d5d0bb5-0174-1000-1a76-dc9d3a7b1b35] Failed to process StandardFlowFileRecord[uuid=4d10880f-3ce2-475a-ac53-cca01f190600,claim=,offset=0,name=userdata1.parquet,size=0]; will route to failure: org.apache.nifi.parquet.stream.NifiParquetInputFile@17badbbc is not a Parquet file (too small length: 0)   Below is my ConvertRecord settings.   For reading file from location and i want to convert into CSV and transfer to other location,  i have built flow ListFile-->ConvertRecord-->FetchFile-->PutFile. Can you please check if the above settings is correct and flow is correct for my requirement   ... View more

Convert Parquet file to CSV using NiFi

by Contributor in Support Questions
‎09-10-2020 12:56 PM
‎09-10-2020 12:56 PM
I haven't worked on Parquet file but i have requirement to convert Parquet file to CSV using NiFi. Appreciate if you could guide me on this. ... View more
Labels:

NiFi shutting down abruptly

by Contributor in Support Questions
‎08-10-2020 10:47 AM
‎08-10-2020 10:47 AM
NiFi services is shutting down abruptly with below exception in production environment. So i tried few things to troubleshoot, when i remove existing flow.xml.gz and restart services i see NiFi UI works as expected. But when i try using flow.xml.gz from archive (latest) and restart i get below error. Not sure what is causing issue here..Appreciate quick response as its impacting production run. Log file nifi-app.log doesnt have any error. Below is error from hs_err_pid*.log Internal exceptions (10 events): Event: 25.581 Thread 0x00007f6cef29e800 Exception <a 'java/lang/NoSuchMethodError': java.lang.Object.lambda$comparingInt$7b0bb60$1(Ljava/util/function/ToIntFunction;Ljava/lang/Object;Ljava/lang/Object;)I> (0x00000005bdf0e618) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/ Event: 25.665 Thread 0x00007f6cef29e800 Implicit null exception at 0x00007f6e7efdb410 to 0x00007f6e7efddb85 Event: 25.665 Thread 0x00007f6cef29e800 Implicit null exception at 0x00007f6e7d49a697 to 0x00007f6e7d49a7b5 Event: 25.784 Thread 0x00007f6ceea03800 Exception <a 'sun/nio/fs/UnixException'> (0x00000005c2dc39b8) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.784 Thread 0x00007f6ceea03800 Exception <a 'sun/nio/fs/UnixException'> (0x00000005c2dc3f60) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.794 Thread 0x00007f6a1c001000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005d838d148) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.794 Thread 0x00007f6a1c001000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005d838d6f0) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.832 Thread 0x00007f6a64002800 Implicit null exception at 0x00007f6e7e330862 to 0x00007f6e7e330959 Event: 25.832 Thread 0x00007f6b08005000 Implicit null exception at 0x00007f6e7e330862 to 0x00007f6e7e330959 Event: 25.839 Thread 0x00007f6b08005000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005db870240) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710]   Bootstrap error log :   2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut # An error report file with more information is saved as: 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut # /users/quant_hub/nifi/nifi-1.11.1/hs_err_pid20683.log 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140348715132672 also had an error] 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349382018816 also had an error] 2020-08-10 10:16:51,393 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349099882240 also had an error] 2020-08-10 10:16:51,402 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349347297024 also had an error] 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # If you would like to submit a bug report, please visit: 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # http://bugreport.java.com/bugreport/crash.jsp 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # The crash happened outside the Java Virtual Machine in native code. 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # See problematic frame for where to report the bug. 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # 2020-08-10 10:16:59,250 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-08-10 10:16:59,327 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is running at PID 20683 but is not responding to ping requests 2020-08-10 10:17:10,140 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi     @dfheinz @MattWho  ... View more
Labels:

Re: NiFi instances not running

by Contributor in Support Questions
‎04-05-2020 01:51 PM
‎04-05-2020 01:51 PM
Issue resolved.  Had to re-install NiFi and services are up and running. ... View more

NiFi instances not running

by Contributor in Support Questions
‎04-04-2020 12:43 PM
‎04-04-2020 12:43 PM
Hi, We recently deployed NiFi to production server (04/03) and executed few dataflows. But after few hours, it automatically shut down and when i tried to stop and re-start, it's not running. There are no errors in log file nifi-app.log. And in bootstrap log i just see these details. Could you please tell what is going wrong here.   Bootstrap log file : 2020-04-04 00:28:24,785 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-04-04 00:28:24,791 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-04-04 00:28:24,857 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is not running 2020-04-04 00:28:31,274 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-04-04 00:28:31,328 INFO [main] org.apache.nifi.bootstrap.RunNiFi Runtime Java version: 1.8.0_211 2020-04-04 00:28:31,334 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2020-04-04 00:28:31,335 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /users/quant_hub/nifi/nifi-1.11.1 2020-04-04 00:28:31,335 INFO [main] org.apache.nifi.bootstrap.Command Command: /users/java/jdk1.8.0_211/bin/java -classpath /users/quant_hub/nifi/nifi-1.11.1/./conf:/users/quant_hub/nifi/nifi-1.11.1/./lib/log4j-over-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jcl-over-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-framework-api-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-properties-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jul-to-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/logback-core-1.2.3.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/slf4j-api-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-nar-utils-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/javax.servlet-api-3.1.0.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/logback-classic-1.2.3.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jetty-schemas-3.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-api-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-runtime-1.11.1.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx4096m -Xms2048m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dzookeeper.admin.enableServer=false -Dnifi.properties.file.path=/users/quant_hub/nifi/nifi-1.11.1/./conf/nifi.properties -Dnifi.bootstrap.listen.port=39921 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/users/quant_hub/nifi/nifi-1.11.1/logs org.apache.nifi.NiFi ... View more
Labels:

Re: Nifi process consuming >100%CPU while executi...

by Contributor in Support Questions
‎04-02-2020 09:11 AM
‎04-02-2020 09:11 AM
@stevenmatison  : Changing the Max Thread Count in Admin->Controller Services? Max Timer Driven--16 , Max Event Driven --1  Setting Concurrent Tasks in processor scheduling tab? - Set to 1 as default Increasing Min/Max Ram? - Set to default Below is my Linux configuration : Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 8 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz Stepping: 4 CPU MHz: 2294.609 BogoMIPS: 4589.21 Hypervisor vendor: VMware Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0-7 ... View more

Re: Nifi process consuming >100%CPU while executi...

by Contributor in Support Questions
‎04-01-2020 11:42 PM
‎04-01-2020 11:42 PM
@stevenmatison  @MattWho  : I am still facing issue where NiFi is crashing when CPU >100%. While checking the logs i get only this error. And once CPU % comes down , GUI comes back again. But when i trigger the flow, same issue occurs. Btw, dataflow is to transfer files ( <50MB) from Linux to S3 with no complex logic. Could you please suggest. 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-4] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=0170100e-e28a-1807-249c-3f5fee9fdb0e] with Flow Registry because could not retrieve version 1 of flow with identifier 42614870-ace3-42fa-a719-01148884d252 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-1] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=01701029-e28a-1807-8de5-812b10e812f4] with Flow Registry because could not retrieve version 1 of flow with identifier 043dc40c-7f7a-4ed3-b3b5-dffcc716a332 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,003 ERROR [Framework Task Thread Thread-3] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=0170101b-e28a-1807-55d0-ba1fc7c21ae5] with Flow Registry because could not retrieve version 1 of flow with identifier 555068bb-a5b6-4506-bbe4-25023010c5e5 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-2] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=01701000-e28a-1807-386c-9c1498a492f7] with Flow Registry because could not retrieve version 1 of flow with identifier 13163161-190b-40fb-8dd9-b9adf6c55245 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,345 INFO [main] o.a.nifi.wali.LengthDelimitedJournal 21.25% of the way finished recovering journal ./flowfile_repository/journals/2460243665.journal, having recovered 69338 updates ... View more

How to derive yesterday's date using now expressio...

by Contributor in Support Questions
‎04-01-2020 08:53 PM
‎04-01-2020 08:53 PM
Currently i am using ${now():format("yyyyMMdd")} to extract today's date and passing it to next flow. But i would need Yesterday's and day before's date also in similar way. How can i achieve this. Appreciate quick response on this. ... View more
Labels:

Re: Transfer files based on the YYYYMMDD in their ...

by Contributor in Support Questions
‎04-01-2020 04:47 PM
‎04-01-2020 04:47 PM
@stevenmatison  : On similar thread, can i do match for 2-3 days and push files to respective folder. Like ${filename:endsWith(${today -1})} and ${filename:endsWith(${today -2})}. Basically i want to keep checking 2 days previous files along with current date and push to respective folder based on date condition.   And how will i transfer those files previous day files to target date folder. Currently i am using /users/${now():format("yyyyMMdd")} to push today's file. If i have to do it for previous 2 days. how will it change. ... View more

Re: Transfer files based on the YYYYMMDD in their ...

by Contributor in Support Questions
‎03-27-2020 10:46 AM
‎03-27-2020 10:46 AM
Ignore my previous reply. Solution which you provided is working. Thanks a lot. ... View more
Contact Me
Online
Offline
Last Visited
‎05-07-2021 10:12 AM
Community Statistics
Member Since ‎02-25-2020 09:46 AM
Last Visited ‎05-07-2021 10:12 AM
Posts 35
Kudos received 1