Member since
04-05-2016
139
Posts
143
Kudos Received
16
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
31915 | 02-14-2019 02:53 PM | |
2469 | 01-04-2019 08:39 PM | |
10591 | 11-05-2018 03:38 PM | |
4917 | 09-27-2018 04:21 PM | |
2679 | 07-05-2018 02:56 PM |
05-25-2022
01:41 PM
The ListS3 and FetchS3 processors in Apache NiFi are commonly used to retrieve objects from Amazon S3 buckets, but they can be easily configured to retrieve objects from IBM Cloud buckets. Assume, I have an IBM Cloud bucket that contains three CSV files: First, get the following from your IBM Cloud bucket configuration : Bucket Name Private Endpoint Then, from the Service Credentials of your Cloud Object Storage, get: Access Key ID Secret Access Key Note: If you don't have Service Credentials for the storage instance, create a new one with HMAC set to "true" (https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-uhc-hmac-credentials-main) Create or confirm that your IBM Cloud user has the necessary Bucket Access Policy to view and download objects (https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-iam-bucket-permissions) : With this setup information confirmed, add to and connect ListS3 and FetchS3 processors on your NiFi canvas, similar to the following: In the List S3 configuration, enter the Bucket, Access Key ID, Secret Access Key and Endpoint URL: Note: The Region property is ignored when the Endpoint Override URL property is used. Run the ListS3 processor and you will see a FlowFile generated for each of the bucket objects: Looking at the queue details: Now configure the FetchS3 similarly with the Bucket Name, Access Key ID, Secret Access Key and Endpoint Override URL: Run the FetchS3 processor and the three CSV files are retrieved from the IBM Cloud Bucket:
... View more
Labels:
05-04-2020
11:46 AM
Hi @yuvapraveen_k NiFi Registry is not supported on Windows. You can see the system requirements here: https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#system-requirements
... View more
03-06-2020
01:06 PM
@MarkH Thanks for the comment and especially for the update. Glad to hear the flow is working for you and appreciate the workaround you documented to help others out in the community if they run into the same issue. It had been a while since I had run this flow so I tried it out today and ran a successful test in the latest released NiFi (Version 1.11.3).
... View more
10-16-2019
01:57 PM
The issue seems to be due to the error documented in the Registry Admin Guide (https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#system-requirements) if you don't have a JDK installed: When running Registry with only a JRE you may encounter the following error as Flyway (database migration tool) attempts to utilize a resource from the JDK: java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Validate failed: Detected failed migration to version 1.3 (DropBucketItemNameUniqueness) Try installing the JDK , deleting the `database` directory that was created prior and then start the Registry.
... View more
08-14-2019
02:51 PM
Are you sure the driver path is correct? Can you provide a screenshot of your Finder window where the jar file is located? The only way I can reproduce the "Not a valid URL or file" warning is if the path is wrong. When I point to this path on my machine (Mac OS 10.14.2), I have no errors: /Users/andrewmlim/Downloads/mysql-connector-java-8.0.17/mysql-connector-java-8.0.17.jar Also, the HCC UI is not helpful since the link is very small, but it is better to select "Add comment" as a response to an answer instead of creating a new Answer.
... View more
08-13-2019
01:19 PM
I believe your path to your driver is incorrect. It should be: /Users/kiran/Downloads/mysql/... Here is an example processor setup from my CDC article (https://community.hortonworks.com/content/kbentry/113941/change-data-capture-cdc-with-apache-nifi-version-1-1.html😞
... View more
07-23-2019
02:22 PM
@Thuy Le Click on the value you want to change and you should see an edit dialog as follows:
... View more
07-22-2019
12:18 AM
Hi @Thuy Le Sorry if I am misunderstanding your question, but have you used the Variables Window to manage your variables? https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Variables_Window
... View more
06-25-2019
05:37 PM
I wrote a blog about this setup (https://community.hortonworks.com/content/kbentry/210286/storing-apache-nifi-versioned-flows-in-a-git-repos.html) and it seems like you did all the configuration steps I included. However, I was not using Git VSTS.
... View more
04-09-2019
04:16 PM
convert-xml-to-csv.xml
... View more