Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Putting data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage

avatar
Expert Contributor

Dear community,

I am trying to forward data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage. However I receive following error:

	com.amazonaws.services.s3.model.AmazonS3Exception: com.oracle.pic.casper.integration.auth.dataplane.exceptions.NotAuthenticatedException: Authentication region name'us-east-1' is incorrect. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1545)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1183)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1685)
        at org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:474)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2133)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2103)
        at org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:417)
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Seems cloud provider checks for region. However I had specified S3.properties file as mentioned here https://community.hortonworks.com/articles/86801/working-with-s3-compatible-data-stores-via-apache.h... with all needed properties which should override region, accesskey, secretkey and others.

1 ACCEPTED SOLUTION

avatar
Expert Contributor

The problem occurs because aws processor builds a client for S3 in different manner and supports only S3 regions.

For example, here is how it is recommended to create client for OCI S3(https://docs.us-phoenix-1.oraclecloud.com/Content/Object/Tasks/s3compatibleapi.htm#supportedClients):

// Get S3 credentials from the console and put them here
AWSCredentialsProvider credentials = new AWSStaticCredentialsProvider(new BasicAWSCredentials(
"ocid1.credential.oc1..anEXAMPLE",
"anEXAMPLE="));

// The name of your tenancy
String tenancy = "tenancy";

// The region to connect to
String region = "us-ashburn-1";

// Create an S3 client pointing at the region
String endpoint = String.format("%s.compat.objectstorage.%s.oraclecloud.com",tenancy,region);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
.standard()
.withCredentials(credentials)
.withEndpointConfiguration(endpointConfiguration)
.disableChunkedEncoding()
.enablePathStyleAccess()
.build();

The most interesting part is:

AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
.standard()
.withCredentials(credentials)
.withEndpointConfiguration(endpointConfiguration)
.disableChunkedEncoding()
.enablePathStyleAccess()
.build();

However in nifi-aws-processor client has no support of setting region, so the region is defaulted to 'us-east-1'

To support other storage rather then amazon one should make changes in nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java:132:

<code>         /**
     * Create client using credentials provider. This is the preferred way for creating clients
     */
    @Override
    protected AmazonS3Client createClient(final ProcessContext context, final AWSCredentialsProvider credentialsProvider, final ClientConfiguration config) {
        getLogger().info("Creating client with credentials provider");
        initializeSignerOverride(context, config);

        AmazonS3Client s3 = null;
        if(StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue()).isEmpty() == false && StringUtils.trimToEmpty(context.getProperty(REGION).evaluateAttributeExpressions().getValue()).isEmpty() == false
){
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue(), context.getProperty(REGION).evaluateAttributeExpressions().getValue());
AmazonS3 client = AmazonS3Client.builder()
 .standard()
 .withClientConfiguration(config)
 .withCredentials(credentials)
 .withEndpointConfiguration(endpointConfiguration)
 .disableChunkedEncoding()
 .enablePathStyleAccess()
 .build();
}
else {
	s3 = new AmazonS3Client(credentialsProvider, config);
	initalizeEndpointOverride(context, s3);
}
        return s3;
    }

View solution in original post

2 REPLIES 2

avatar
Expert Contributor

I can confirm that credentials are correct, because I had used the same for s3cmd and everything had been working correctly

avatar
Expert Contributor

The problem occurs because aws processor builds a client for S3 in different manner and supports only S3 regions.

For example, here is how it is recommended to create client for OCI S3(https://docs.us-phoenix-1.oraclecloud.com/Content/Object/Tasks/s3compatibleapi.htm#supportedClients):

// Get S3 credentials from the console and put them here
AWSCredentialsProvider credentials = new AWSStaticCredentialsProvider(new BasicAWSCredentials(
"ocid1.credential.oc1..anEXAMPLE",
"anEXAMPLE="));

// The name of your tenancy
String tenancy = "tenancy";

// The region to connect to
String region = "us-ashburn-1";

// Create an S3 client pointing at the region
String endpoint = String.format("%s.compat.objectstorage.%s.oraclecloud.com",tenancy,region);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
.standard()
.withCredentials(credentials)
.withEndpointConfiguration(endpointConfiguration)
.disableChunkedEncoding()
.enablePathStyleAccess()
.build();

The most interesting part is:

AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
.standard()
.withCredentials(credentials)
.withEndpointConfiguration(endpointConfiguration)
.disableChunkedEncoding()
.enablePathStyleAccess()
.build();

However in nifi-aws-processor client has no support of setting region, so the region is defaulted to 'us-east-1'

To support other storage rather then amazon one should make changes in nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java:132:

<code>         /**
     * Create client using credentials provider. This is the preferred way for creating clients
     */
    @Override
    protected AmazonS3Client createClient(final ProcessContext context, final AWSCredentialsProvider credentialsProvider, final ClientConfiguration config) {
        getLogger().info("Creating client with credentials provider");
        initializeSignerOverride(context, config);

        AmazonS3Client s3 = null;
        if(StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue()).isEmpty() == false && StringUtils.trimToEmpty(context.getProperty(REGION).evaluateAttributeExpressions().getValue()).isEmpty() == false
){
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue(), context.getProperty(REGION).evaluateAttributeExpressions().getValue());
AmazonS3 client = AmazonS3Client.builder()
 .standard()
 .withClientConfiguration(config)
 .withCredentials(credentials)
 .withEndpointConfiguration(endpointConfiguration)
 .disableChunkedEncoding()
 .enablePathStyleAccess()
 .build();
}
else {
	s3 = new AmazonS3Client(credentialsProvider, config);
	initalizeEndpointOverride(context, s3);
}
        return s3;
    }