Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Putting data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage

Solved Go to solution
Highlighted

Putting data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage

Expert Contributor

Dear community,

I am trying to forward data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage. However I receive following error:

	com.amazonaws.services.s3.model.AmazonS3Exception: com.oracle.pic.casper.integration.auth.dataplane.exceptions.NotAuthenticatedException: Authentication region name'us-east-1' is incorrect. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1545)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1183)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1685)
        at org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:474)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2133)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2103)
        at org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:417)
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Seems cloud provider checks for region. However I had specified S3.properties file as mentioned here https://community.hortonworks.com/articles/86801/working-with-s3-compatible-data-stores-via-apache.h... with all needed properties which should override region, accesskey, secretkey and others.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Putting data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage

Expert Contributor

The problem occurs because aws processor builds a client for S3 in different manner and supports only S3 regions.

For example, here is how it is recommended to create client for OCI S3(https://docs.us-phoenix-1.oraclecloud.com/Content/Object/Tasks/s3compatibleapi.htm#supportedClients):

// Get S3 credentials from the console and put them here
AWSCredentialsProvider credentials = new AWSStaticCredentialsProvider(new BasicAWSCredentials(
"ocid1.credential.oc1..anEXAMPLE",
"anEXAMPLE="));

// The name of your tenancy
String tenancy = "tenancy";

// The region to connect to
String region = "us-ashburn-1";

// Create an S3 client pointing at the region
String endpoint = String.format("%s.compat.objectstorage.%s.oraclecloud.com",tenancy,region);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
.standard()
.withCredentials(credentials)
.withEndpointConfiguration(endpointConfiguration)
.disableChunkedEncoding()
.enablePathStyleAccess()
.build();

The most interesting part is:

AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
.standard()
.withCredentials(credentials)
.withEndpointConfiguration(endpointConfiguration)
.disableChunkedEncoding()
.enablePathStyleAccess()
.build();

However in nifi-aws-processor client has no support of setting region, so the region is defaulted to 'us-east-1'

To support other storage rather then amazon one should make changes in nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java:132:

<code>         /**
     * Create client using credentials provider. This is the preferred way for creating clients
     */
    @Override
    protected AmazonS3Client createClient(final ProcessContext context, final AWSCredentialsProvider credentialsProvider, final ClientConfiguration config) {
        getLogger().info("Creating client with credentials provider");
        initializeSignerOverride(context, config);

        AmazonS3Client s3 = null;
        if(StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue()).isEmpty() == false && StringUtils.trimToEmpty(context.getProperty(REGION).evaluateAttributeExpressions().getValue()).isEmpty() == false
){
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue(), context.getProperty(REGION).evaluateAttributeExpressions().getValue());
AmazonS3 client = AmazonS3Client.builder()
 .standard()
 .withClientConfiguration(config)
 .withCredentials(credentials)
 .withEndpointConfiguration(endpointConfiguration)
 .disableChunkedEncoding()
 .enablePathStyleAccess()
 .build();
}
else {
	s3 = new AmazonS3Client(credentialsProvider, config);
	initalizeEndpointOverride(context, s3);
}
        return s3;
    }
2 REPLIES 2

Re: Putting data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage

Expert Contributor

I can confirm that credentials are correct, because I had used the same for s3cmd and everything had been working correctly

Re: Putting data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage

Expert Contributor

The problem occurs because aws processor builds a client for S3 in different manner and supports only S3 regions.

For example, here is how it is recommended to create client for OCI S3(https://docs.us-phoenix-1.oraclecloud.com/Content/Object/Tasks/s3compatibleapi.htm#supportedClients):

// Get S3 credentials from the console and put them here
AWSCredentialsProvider credentials = new AWSStaticCredentialsProvider(new BasicAWSCredentials(
"ocid1.credential.oc1..anEXAMPLE",
"anEXAMPLE="));

// The name of your tenancy
String tenancy = "tenancy";

// The region to connect to
String region = "us-ashburn-1";

// Create an S3 client pointing at the region
String endpoint = String.format("%s.compat.objectstorage.%s.oraclecloud.com",tenancy,region);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
.standard()
.withCredentials(credentials)
.withEndpointConfiguration(endpointConfiguration)
.disableChunkedEncoding()
.enablePathStyleAccess()
.build();

The most interesting part is:

AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
.standard()
.withCredentials(credentials)
.withEndpointConfiguration(endpointConfiguration)
.disableChunkedEncoding()
.enablePathStyleAccess()
.build();

However in nifi-aws-processor client has no support of setting region, so the region is defaulted to 'us-east-1'

To support other storage rather then amazon one should make changes in nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java:132:

<code>         /**
     * Create client using credentials provider. This is the preferred way for creating clients
     */
    @Override
    protected AmazonS3Client createClient(final ProcessContext context, final AWSCredentialsProvider credentialsProvider, final ClientConfiguration config) {
        getLogger().info("Creating client with credentials provider");
        initializeSignerOverride(context, config);

        AmazonS3Client s3 = null;
        if(StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue()).isEmpty() == false && StringUtils.trimToEmpty(context.getProperty(REGION).evaluateAttributeExpressions().getValue()).isEmpty() == false
){
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue(), context.getProperty(REGION).evaluateAttributeExpressions().getValue());
AmazonS3 client = AmazonS3Client.builder()
 .standard()
 .withClientConfiguration(config)
 .withCredentials(credentials)
 .withEndpointConfiguration(endpointConfiguration)
 .disableChunkedEncoding()
 .enablePathStyleAccess()
 .build();
}
else {
	s3 = new AmazonS3Client(credentialsProvider, config);
	initalizeEndpointOverride(context, s3);
}
        return s3;
    }
Don't have an account?
Coming from Hortonworks? Activate your account here