Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Gubbi
Gubbi
Contributor

Re: Convert Parquet file to CSV using NiFi

by Contributor Gubbi in Support Questions
‎09-11-2020 06:58 AM
‎09-11-2020 06:58 AM
@stevenmatison : I tried below flow and getting this error. ConvertRecord[id=7d5d0bb5-0174-1000-1a76-dc9d3a7b1b35] Failed to process StandardFlowFileRecord[uuid=4d10880f-3ce2-475a-ac53-cca01f190600,claim=,offset=0,name=userdata1.parquet,size=0]; will route to failure: org.apache.nifi.parquet.stream.NifiParquetInputFile@17badbbc is not a Parquet file (too small length: 0)   Below is my ConvertRecord settings.   For  reading file from location and i want to convert into CSV and transfer to other location,  i have built flow ListFile-->ConvertRecord-->FetchFile-->PutFile. Can you please check if the above settings is correct and flow is correct for my requirement   ... View more

Convert Parquet file to CSV using NiFi

by Contributor Gubbi in Support Questions
‎09-10-2020 12:56 PM
‎09-10-2020 12:56 PM
I haven't worked on Parquet file but i have requirement to convert Parquet file to CSV using NiFi. Appreciate if you could guide me on this. ... View more
Labels:

NiFi shutting down abruptly

by Contributor Gubbi in Support Questions
‎08-10-2020 10:47 AM
‎08-10-2020 10:47 AM
NiFi services is shutting down abruptly with below exception in production environment. So i tried few things to troubleshoot, when i remove existing flow.xml.gz and restart services i see NiFi UI works as expected. But when i try using flow.xml.gz from archive (latest) and restart i get below error. Not sure what is causing issue here..Appreciate quick response as its impacting production run. Log file nifi-app.log doesnt have any error. Below is error from hs_err_pid*.log Internal exceptions (10 events): Event: 25.581 Thread 0x00007f6cef29e800 Exception <a 'java/lang/NoSuchMethodError': java.lang.Object.lambda$comparingInt$7b0bb60$1(Ljava/util/function/ToIntFunction;Ljava/lang/Object;Ljava/lang/Object;)I> (0x00000005bdf0e618) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/ Event: 25.665 Thread 0x00007f6cef29e800 Implicit null exception at 0x00007f6e7efdb410 to 0x00007f6e7efddb85 Event: 25.665 Thread 0x00007f6cef29e800 Implicit null exception at 0x00007f6e7d49a697 to 0x00007f6e7d49a7b5 Event: 25.784 Thread 0x00007f6ceea03800 Exception <a 'sun/nio/fs/UnixException'> (0x00000005c2dc39b8) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.784 Thread 0x00007f6ceea03800 Exception <a 'sun/nio/fs/UnixException'> (0x00000005c2dc3f60) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.794 Thread 0x00007f6a1c001000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005d838d148) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.794 Thread 0x00007f6a1c001000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005d838d6f0) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710] Event: 25.832 Thread 0x00007f6a64002800 Implicit null exception at 0x00007f6e7e330862 to 0x00007f6e7e330959 Event: 25.832 Thread 0x00007f6b08005000 Implicit null exception at 0x00007f6e7e330862 to 0x00007f6e7e330959 Event: 25.839 Thread 0x00007f6b08005000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005db870240) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710]   Bootstrap error log :   2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut # An error report file with more information is saved as: 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut # /users/quant_hub/nifi/nifi-1.11.1/hs_err_pid20683.log 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140348715132672 also had an error] 2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349382018816 also had an error] 2020-08-10 10:16:51,393 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349099882240 also had an error] 2020-08-10 10:16:51,402 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349347297024 also had an error] 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # If you would like to submit a bug report, please visit: 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # http://bugreport.java.com/bugreport/crash.jsp 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # The crash happened outside the Java Virtual Machine in native code. 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # See problematic frame for where to report the bug. 2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # 2020-08-10 10:16:59,250 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-08-10 10:16:59,327 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is running at PID 20683 but is not responding to ping requests 2020-08-10 10:17:10,140 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi     @dfheinz @MattWho  ... View more
Labels:

How to filter file with multiple pattern

by Contributor Gubbi in Support Questions
‎06-02-2020 05:21 PM
‎06-02-2020 05:21 PM
Currently i am having simple dataflow which has ListFile-->FetchFile-->PutS3. This flow reads data from source and uploads it to S3.  Now, i have source directory /users/data/ which has abc_mmddyyy.pdf, xyz_mmddyyy.pdf and hij_mmddyyyy.pdf which uploads into S3 folder \aws\data\ . We are planning to source new files which would be published in /users/data/demo/www_mmddyyyy.csv and has to be uploaded to \aws\data\demo.   How will i achieve this @stevenmatison  ... View more
Labels:

Re: NiFi instances not running

by Contributor Gubbi in Support Questions
‎04-05-2020 01:51 PM
‎04-05-2020 01:51 PM
Issue resolved.  Had to re-install NiFi and services are up and running. ... View more

NiFi instances not running

by Contributor Gubbi in Support Questions
‎04-04-2020 12:43 PM
‎04-04-2020 12:43 PM
Hi, We recently deployed NiFi to production server (04/03) and executed few dataflows. But after few hours, it automatically shut down and when i tried to stop and re-start, it's not running. There are no errors in log file nifi-app.log. And in bootstrap log i just see these details. Could you please tell what is going wrong here.   Bootstrap log file : 2020-04-04 00:28:24,785 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-04-04 00:28:24,791 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-04-04 00:28:24,792 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-04-04 00:28:24,857 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is not running 2020-04-04 00:28:31,274 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: [] 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED 2020-04-04 00:28:31,278 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED 2020-04-04 00:28:31,328 INFO [main] org.apache.nifi.bootstrap.RunNiFi Runtime Java version: 1.8.0_211 2020-04-04 00:28:31,334 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2020-04-04 00:28:31,335 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /users/quant_hub/nifi/nifi-1.11.1 2020-04-04 00:28:31,335 INFO [main] org.apache.nifi.bootstrap.Command Command: /users/java/jdk1.8.0_211/bin/java -classpath /users/quant_hub/nifi/nifi-1.11.1/./conf:/users/quant_hub/nifi/nifi-1.11.1/./lib/log4j-over-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jcl-over-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-framework-api-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-properties-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jul-to-slf4j-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/logback-core-1.2.3.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/slf4j-api-1.7.26.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-nar-utils-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/javax.servlet-api-3.1.0.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/logback-classic-1.2.3.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/jetty-schemas-3.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-api-1.11.1.jar:/users/quant_hub/nifi/nifi-1.11.1/./lib/nifi-runtime-1.11.1.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx4096m -Xms2048m -Djavax.security.auth.useSubjectCredsOnly=true -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dzookeeper.admin.enableServer=false -Dnifi.properties.file.path=/users/quant_hub/nifi/nifi-1.11.1/./conf/nifi.properties -Dnifi.bootstrap.listen.port=39921 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/users/quant_hub/nifi/nifi-1.11.1/logs org.apache.nifi.NiFi ... View more

Re: Monitor Historical files and transfer to desti...

by Contributor Gubbi in Support Questions
‎04-02-2020 04:49 PM
‎04-02-2020 04:49 PM
@stevenmatison : We dont have control on /users/data folder. Vendor would publish files in folder datewise. So there is no easy way to scan historical files without using minus expression. Minus would restrict for certain days, What if i have file a week back and i need to pick that one too.. There is no definite date when the old files would be pushed to that folder.  ... View more

Transfer files based on date present in filename

by Contributor Gubbi in Support Questions
‎04-02-2020 04:23 PM
‎04-02-2020 04:23 PM
Used case : Transfer files from inbound directory based on date condition in filename to destination folder with date .   Ex : Input directory :/user/mft/inbound/         Filename : etfbfo_20200331.csv , gfsdajaf_20200401.csv, gdfcbakfg_20200402.csv.   Destination : /users/aws/outbound/20200331/ tfbfo_20200331.csv                      /users/aws/outbound/20200401/ gfsdajaf_20200401.csv                      /users/aws/outbound/20200402/gdfcbakfg_20200402.csv.  @stevenmatison  ... View more
Labels:

Monitor Historical files and transfer to destinati...

by Contributor Gubbi in Support Questions
‎04-02-2020 04:14 PM
‎04-02-2020 04:14 PM
I have data flow which is checking folder /users/data/${now():format("yyyyMMdd")} and it would pick all files under this folder and upload to S3 /users/aws/${now():format("yyyyMMdd")}. Suppose i get file today which is 3 days old in the folder /users/data/20200331 i need to scan that folder and pick that file as well and move to S3 user /users/aws/20200331.  Note : File names doesnt have date, Files are just dropped inside date folders.   @stevenmatison        ... View more
Labels:

Re: Nifi process consuming >100%CPU while executi...

by Contributor Gubbi in Support Questions
‎04-02-2020 09:11 AM
‎04-02-2020 09:11 AM
@stevenmatison  : Changing the Max Thread Count in Admin->Controller Services? Max Timer Driven--16 , Max Event Driven --1  Setting Concurrent Tasks in processor scheduling tab? - Set to 1 as default Increasing Min/Max Ram? - Set to default Below is my Linux configuration : Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 8 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz Stepping: 4 CPU MHz: 2294.609 BogoMIPS: 4589.21 Hypervisor vendor: VMware Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0-7 ... View more

Re: Nifi process consuming >100%CPU while executi...

by Contributor Gubbi in Support Questions
‎04-01-2020 11:42 PM
‎04-01-2020 11:42 PM
@stevenmatison  @MattWho  : I am still facing issue where NiFi is crashing when CPU >100%. While checking the logs i get only this error. And once CPU % comes down , GUI comes back again. But when i trigger the flow, same issue occurs. Btw, dataflow is to transfer files ( <50MB) from Linux to S3 with no complex logic. Could you please suggest. 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-4] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=0170100e-e28a-1807-249c-3f5fee9fdb0e] with Flow Registry because could not retrieve version 1 of flow with identifier 42614870-ace3-42fa-a719-01148884d252 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-1] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=01701029-e28a-1807-8de5-812b10e812f4] with Flow Registry because could not retrieve version 1 of flow with identifier 043dc40c-7f7a-4ed3-b3b5-dffcc716a332 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,003 ERROR [Framework Task Thread Thread-3] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=0170101b-e28a-1807-55d0-ba1fc7c21ae5] with Flow Registry because could not retrieve version 1 of flow with identifier 555068bb-a5b6-4506-bbe4-25023010c5e5 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,002 ERROR [Framework Task Thread Thread-2] o.a.nifi.groups.StandardProcessGroup Failed to synchronize StandardProcessGroup[identifier=01701000-e28a-1807-386c-9c1498a492f7] with Flow Registry because could not retrieve version 1 of flow with identifier 13163161-190b-40fb-8dd9-b9adf6c55245 in bucket 0509399b-3d5c-4c8e-8a43-a898b8c58163 due to: Connection refused (Connection refused) 2020-04-01 23:39:02,345 INFO [main] o.a.nifi.wali.LengthDelimitedJournal 21.25% of the way finished recovering journal ./flowfile_repository/journals/2460243665.journal, having recovered 69338 updates ... View more

How to derive yesterday's date using now expressio...

by Contributor Gubbi in Support Questions
‎04-01-2020 08:53 PM
‎04-01-2020 08:53 PM
Currently i am using ${now():format("yyyyMMdd")} to extract today's date and passing it to next flow. But i would need Yesterday's and day before's date also in similar way. How can i achieve this. Appreciate quick response on this. ... View more
Labels:

Re: Transfer files based on the YYYYMMDD in their ...

by Contributor Gubbi in Support Questions
‎04-01-2020 04:47 PM
‎04-01-2020 04:47 PM
@stevenmatison  : On similar thread, can i do match for 2-3 days and push files to respective folder. Like ${filename:endsWith(${today -1})} and ${filename:endsWith(${today -2})}. Basically i want to keep checking 2 days previous files along with current date and push to respective folder based on date condition.   And how will i transfer those files previous day files to target date folder. Currently i am using /users/${now():format("yyyyMMdd")} to push today's file. If i have to do it for previous 2 days. how will it change. ... View more

Re: Transfer files based on the YYYYMMDD in their ...

by Contributor Gubbi in Support Questions
‎03-27-2020 10:46 AM
‎03-27-2020 10:46 AM
Ignore my previous reply. Solution which you provided is working. Thanks a lot. ... View more

Re: Transfer files based on the YYYYMMDD in their ...

by Contributor Gubbi in Support Questions
‎03-27-2020 10:33 AM
‎03-27-2020 10:33 AM
@stevenmatison : I am not using GetFile here. i followed that approach you suggested but issue still persists. Below is dataflow i have been using. i need to route all matching files into target folder based on date. So for today's date 20200327, all files ending with _202003027 must reside in target folder 20200327. So in my fetch file i am passing this /users/FIDS_OZ_DEV/Datastore/Bloomberg/backoffice/${now():format("yyyyMMdd")} ... View more

Transfer files based on the YYYYMMDD in their file...

by Contributor Gubbi in Support Questions
‎03-26-2020 10:40 PM
‎03-26-2020 10:40 PM
@MattWho :  I'm finding a way to get the content of a file based on its filename and it should drop files to target based on date. . All of the target files are in the same directory, but I'd like to select only the files which has "{Today's YYYYMMDD} as their postfix. For example, if today is 20200326,  /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326 -> create folder with 20200326 and drop file. /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200326-> Drop file into above folder /tmp/pfd_exch_namr_bbid.px.hpc.gz.20200325-> not ok because this YYYYMMDD is not today.   Approach i followed : ListFile: Extract only the files whose name ends with "gz.20200326" UpdateAttribute: Substitute YYYYMMDD into an Attribute "today"     RouteOnAttribute: Extract only the file which matches ${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})} (where $filename is the target file name) and pass it as Relationship "target_file"   But when executing, looks like it's not matching my requirement and all files are matching. Could you please assist me on this. ... View more
Labels:

Re: How to ingest files based on the YYYYMMDD in t...

by Contributor Gubbi in Support Questions
‎03-26-2020 10:22 PM
‎03-26-2020 10:22 PM
@shuhei_shogen  : I had similar used case but when i tried to reproduce the same approach its not working.My file name is like equity_asia2.dif.gz.20200324 and i want these files to be in target folder 20200324. In Update Attribute i have used ${filename:matches(${today:prepend('[^\.].*gz.'):append('\d{8}')})}. But it doesnt seem to be working. Could you please check and assist me on this. ... View more

Option to acces AWS3 Secret without exposing keys

by Contributor Gubbi in Support Questions
‎03-20-2020 01:46 PM
‎03-20-2020 01:46 PM
@MattWho  : Currently i am using PutS3 Object to transfer files from source to destination. I am passing secret key in property file and accessing via Credential File configuration. But our security team is not allowing this process in production as it is exposed for others to access. Can you suggest any other  approach to access these keys. Appreciate quick response as we have production deployment in < 5 days.  ... View more
Labels:

Prevent nifi-app logs from filling drive.

by Contributor Gubbi in Support Questions
‎03-16-2020 06:25 PM
‎03-16-2020 06:25 PM
@MattWho I have scheduled my workflow to run for 1 sec everyday. noticed that nifi is creating log files for every second and is cauing my disk space 100%. How would i avoid this.   -rw-r--r-- 1 ozdevaa00002695s001 users 109697330 Mar 16 18:12 nifi-app_2020-03-16_18.231.log -rw-r--r-- 1 ozdevaa00002695s001 users 109326202 Mar 16 18:12 nifi-app_2020-03-16_18.232.log -rw-r--r-- 1 ozdevaa00002695s001 users 109771816 Mar 16 18:12 nifi-app_2020-03-16_18.233.log -rw-r--r-- 1 ozdevaa00002695s001 users 109617535 Mar 16 18:12 nifi-app_2020-03-16_18.234.log -rw-r--r-- 1 ozdevaa00002695s001 users 109453297 Mar 16 18:12 nifi-app_2020-03-16_18.235.log -rw-r--r-- 1 ozdevaa00002695s001 users 109818572 Mar 16 18:12 nifi-app_2020-03-16_18.236.log -rw-r--r-- 1 ozdevaa00002695s001 users 109481576 Mar 16 18:12 nifi-app_2020-03-16_18.237.log -rw-r--r-- 1 ozdevaa00002695s001 users 109715149 Mar 16 18:12 nifi-app_2020-03-16_18.238.log -rw-r--r-- 1 ozdevaa00002695s001 users 109681157 Mar 16 18:12 nifi-app_2020-03-16_18.239.log -rw-r--r-- 1 ozdevaa00002695s001 users 109731956 Mar 16 18:12 nifi-app_2020-03-16_18.240.log -rw-r--r-- 1 ozdevaa00002695s001 users 109645010 Mar 16 18:12 nifi-app_2020-03-16_18.241.log -rw-r--r-- 1 ozdevaa00002695s001 users 102797498 Mar 16 18:12 nifi-app.log ... View more
Labels:

Re: Move entire folder based on date to S3

by Contributor Gubbi in Support Questions
‎03-16-2020 12:31 PM
‎03-16-2020 12:31 PM
@MattWho : Appreciate your inputs as well ... View more

Re: Move entire folder based on date to S3

by Contributor Gubbi in Support Questions
‎03-16-2020 12:27 PM
‎03-16-2020 12:27 PM
@Shu_ashu  : I saw you had suggested similar pattern solution before. Could you please look into this and suggest approach.   https://community.cloudera.com/t5/Support-Questions/NiFi-Creating-the-output-directory-from-the-content-of-the/td-p/231335 ... View more

Re: Fetch File unable to copy files from NAS drive

by Contributor Gubbi in Support Questions
‎03-15-2020 02:04 PM
‎03-15-2020 02:04 PM
Issue has been resolved. ... View more

Fetch File unable to copy files from NAS drive

by Contributor Gubbi in Support Questions
‎03-15-2020 11:22 AM
‎03-15-2020 11:22 AM
I have NAS mounted on Linux server and have to copy files from location to S3, i am using ListFile→FetchFile-→PutS3.  While reading files from NAS mount location, i am getting error saying file not found but i see all required files are present at location. Is there any special configuration which needs to be set while reading files from NAS drive. ... View more
Labels:

Move entire folder based on date to S3

by Contributor Gubbi in Support Questions
‎03-13-2020 01:42 PM
‎03-13-2020 01:42 PM
Used case :  Need to transfer entire folder of files based on date to S3. Ex :  Source Path (Linux) : /users/abc/20200312/gtry/gyyy.csv             S3    : /Users/datastore/20200312/gyyy.csv   Since dates keep changing everyday, and i need to build dataflow which would pick files from date folder ... View more
Labels:

Secured way to store S3 credentials

by Contributor Gubbi in Support Questions
‎03-04-2020 04:32 PM
‎03-04-2020 04:32 PM
i am using PutS3 processor for transferring files to S3 and accessing Access key details via Credentials File property. But i wanted to understand is there any other secured way in handling this. ... View more
Labels:

Re: Transfer files to S3 based on file timestamp

by Contributor Gubbi in Support Questions
‎03-04-2020 12:15 PM
1 Kudo
‎03-04-2020 12:15 PM
1 Kudo
I got solution for this. Had to use expression language in Object Key to fetch date from file and it worked. Below is expression :${filename:substringAfter('.gz.')}/${filename} ... View more

Re: Transfer files to S3 based on file timestamp

by Contributor Gubbi in Support Questions
‎03-03-2020 10:42 PM
‎03-03-2020 10:42 PM
@AustinLiu Just to clarify, i am transferring files from linux box to S3.  ... View more

Re: Transfer files to S3 based on file timestamp

by Contributor Gubbi in Support Questions
‎03-03-2020 10:31 PM
‎03-03-2020 10:31 PM
@AustinLiu : But i need to transfer file abcd_20200303 to S3 folder 20200303 and respectively based on dates. Every day when the files arrive my processor should identify file based on date and push it to respective date folder in S3. ... View more

Transfer files to S3 based on file timestamp

by Contributor Gubbi in Support Questions
‎03-03-2020 08:00 PM
‎03-03-2020 08:00 PM
I have used case where i am reading files with timestamp and these files has to be transferred to S3 and create folder with respective dates. Ex: file names abcd.out.gz.20200303 , abcd.out.gz.20200302 and the file abcd.out.gz.20200303 need to be in S3 under /data/20200303 and file abcd.out.gz.20200302 under /data/20200302.   How can i achieve this in NiFi.  ... View more
Labels:

Re: Nifi process consuming >100%CPU while executi...

by Contributor Gubbi in Support Questions
‎02-28-2020 01:06 PM
‎02-28-2020 01:06 PM
Since you suggested to check on Tasks/Time taken by ListFile processor group i had mentioned 300. If you notice above screenshot Tasks shows 297.  Nifi is still crashing and i am clueless on what's going on here. ... View more
Contact Me
Online
Offline
Last Visited
‎09-11-2020 09:02 PM
Public Statistics
Date Registered ‎02-25-2020 09:46 AM
Last Visited ‎09-11-2020 09:02 PM
Total Messages Posted 35
Total Kudos Received 1