question Putting data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage in Archives of Support Questions (Read Only)

Putting data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage

vladislav_falfu — Sat, 16 Dec 2017 07:24:05 GMT

Dear community,

I am trying to forward data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage. However I receive following error:

	com.amazonaws.services.s3.model.AmazonS3Exception: com.oracle.pic.casper.integration.auth.dataplane.exceptions.NotAuthenticatedException: Authentication region name'us-east-1' is incorrect. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1545)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1183)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1685)
        at org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:474)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2133)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2103)
        at org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:417)
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Seems cloud provider checks for region. However I had specified S3.properties file as mentioned here https://community.hortonworks.com/articles/86801/working-with-s3-compatible-data-stores-via-apache.html with all needed properties which should override region, accesskey, secretkey and others.

Re: Putting data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage

vladislav_falfu — Sat, 16 Dec 2017 16:21:29 GMT

I can confirm that credentials are correct, because I had used the same for s3cmd and everything had been working correctly

Re: Putting data from apache nifi to OCI(Oracle cloud infrastructure) S3 storage

vladislav_falfu — Tue, 19 Dec 2017 03:42:10 GMT

The problem occurs because aws processor builds a client for S3 in different manner and supports only S3 regions.

For example, here is how it is recommended to create client for OCI S3(https://docs.us-phoenix-1.oraclecloud.com/Content/Object/Tasks/s3compatibleapi.htm#supportedClients):

// Get S3 credentials from the console and put them here
AWSCredentialsProvider credentials = new AWSStaticCredentialsProvider(new BasicAWSCredentials(
 "ocid1.credential.oc1..anEXAMPLE",
 "anEXAMPLE="));

// The name of your tenancy
String tenancy = "tenancy";

// The region to connect to
String region = "us-ashburn-1";

// Create an S3 client pointing at the region
String endpoint = String.format("%s.compat.objectstorage.%s.oraclecloud.com",tenancy,region);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
 .standard()
 .withCredentials(credentials)
 .withEndpointConfiguration(endpointConfiguration)
 .disableChunkedEncoding()
 .enablePathStyleAccess()
 .build();

The most interesting part is:

AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
 .standard()
 .withCredentials(credentials)
 .withEndpointConfiguration(endpointConfiguration)
 .disableChunkedEncoding()
 .enablePathStyleAccess()
 .build();

However in nifi-aws-processor client has no support of setting region, so the region is defaulted to 'us-east-1'

To support other storage rather then amazon one should make changes in nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java:132:

<code>         /**
     * Create client using credentials provider. This is the preferred way for creating clients
     */
    @Override
    protected AmazonS3Client createClient(final ProcessContext context, final AWSCredentialsProvider credentialsProvider, final ClientConfiguration config) {
        getLogger().info("Creating client with credentials provider");
        initializeSignerOverride(context, config);

        AmazonS3Client s3 = null;
        if(StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue()).isEmpty() == false && StringUtils.trimToEmpty(context.getProperty(REGION).evaluateAttributeExpressions().getValue()).isEmpty() == false
){
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue(), context.getProperty(REGION).evaluateAttributeExpressions().getValue());
AmazonS3 client = AmazonS3Client.builder()
 .standard()
 .withClientConfiguration(config)
 .withCredentials(credentials)
 .withEndpointConfiguration(endpointConfiguration)
 .disableChunkedEncoding()
 .enablePathStyleAccess()
 .build();
}
else {
	s3 = new AmazonS3Client(credentialsProvider, config);
	initalizeEndpointOverride(context, s3);
}
        return s3;
    }