Ozone S3 with AWS credentials

[#1]

Hi,

We can access/read/write Ozone etc using aws s3api, but would like to see if we can do the same using curl. We may not have the option of using the awscli package in RHEL.

We've tried curl with webhdfs, but that is a bit limited.

This almost seems to work:

$ curl -vvv -H "Authorization:AWS user@FQDN:aws_key" https://host73.fqdn:9879/s3v/bucket/curl_test

But it seems to get stuck here:

> GET /s3v/bucket/curl_test HTTP/1.1
> User-Agent: curl/7.29.0
> Host: host73.fqdn:9879
> Accept: */*
> Authorization:AWS user@FQDN:aws_key
>

It makes the connection to the Ozone S3 gateway, but then just hangs. Does anyone know why it's getting stuck? Log messages seems to be inconsistent, and there's nothing in Ranger. Everything we've tried hasn't worked yet.

[#2]

 Log messages show errors like:

org.apache.hadoop.ozone.s3.signature.MalformedResourceException

But I can;t find details on what the headers or resource should be.

[#3]

@EFasdfSDfaSDFG, Welcome to our community! To help you get the best possible answer, I have tagged our CDP expert @vaishaakb who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#4]

Hello @EFasdfSDfaSDFG Thank you for bringing this to our community. 

I understand you are seeking operability of Ozone with native REST API. 
Per my internal research, I learned that the native REST API for Ozone is completely superseded by the S3 REST API and deprecated in the later versions of ozone.

Additionally, I would like to stress that the native REST API neither has official support nor maintained in the Upstream Server.

To help you efficiently,
May I know the current Ozone and the CDP version that you are making use of?

V

[#5]

Hello @vaishaakb ,

We are running CDP 7.1.8-1.cdh7.1.8.p36.42193080 (and preparing to patch to 7.1.9 soonish), and Ozone 718.2.3-1.ozone718.2.3.p0.46980125.

[#6]

@EFasdfSDfaSDFG This still seems broader scope of requirement.
Can you please help elaborate to specific situation on what we are trying to solve?

My understanding is that since you have restriction on installing awscli on RHEL, you are seeking alternative solutions to access/manage Ozone. Is my understanding correct?

Additionally, Did you get a chance to review the available client interfaces for managing Ozone?
https://ozone.apache.org/docs/1.4.0/interface.html

Let me know your thoughts!

V

[#7]

My understanding is that since you have restriction on installing awscli on RHEL, you are seeking alternative solutions to access/manage Ozone. Is my understanding correct?

---

This is correct. Specifically I am asking how to/if I can use curl to upload files to Ozone:

---

@EFasdfSDfaSDFG wrote:

Hi,

We can access/read/write Ozone etc using aws s3api, but would like to see if we can do the same using curl. We may not have the option of using the awscli package in RHEL.

We've tried curl with webhdfs, but that is a bit limited.

This almost seems to work:

$ curl -vvv -H "Authorization:AWS user@FQDN:aws_key" https://host73.fqdn:9879/s3v/bucket/curl_test

But it seems to get stuck here:

> GET /s3v/bucket/curl_test HTTP/1.1
> User-Agent: curl/7.29.0
> Host: host73.fqdn:9879
> Accept: */*
> Authorization:AWS user@FQDN:aws_key
>

It makes the connection to the Ozone S3 gateway, but then just hangs. Does anyone know why it's getting stuck? Log messages seems to be inconsistent, and there's nothing in Ranger. Everything we've tried hasn't worked yet.

---

As for https://ozone.apache.org/docs/1.4.0/interface.html, Java API might be a possible better option than cURL.

[#8 · Accepted solution]

@EFasdfSDfaSDFG 
Took sometime to do a feasibility study and found a couple of things from our Internal resources.

1. FYI: The native Ozone REST API is completely superseded by the S3 REST API.
Therefore, there is no native REST API to manage Ozone.
2. Since I am unsure of your Use-case, I would suggest you test this either by Java API or HttpFS Gateway Interface.[0a]
[0a]https://ozone.apache.org/docs/1.4.0/interface.html

HTH.

If you find this answering your question, Please mark this as Accept as Solution.
Also, You may thank me by clicking the thumbs-up!

Cheers!