question Re: Ozone S3 with AWS credentials in Support Questions

Ozone S3 with AWS credentials

EFasdfSDfaSDFG — Tue, 21 Apr 2026 06:43:25 GMT

Hi,

We can access/read/write Ozone etc using aws s3api, but would like to see if we can do the same using curl. We may not have the option of using the awscli package in RHEL.

We've tried curl with webhdfs, but that is a bit limited.

This almost seems to work:

$ curl -vvv -H "Authorization:AWS user@FQDN:aws_key" https://host73.fqdn:9879/s3v/bucket/curl_test

But it seems to get stuck here:

> GET /s3v/bucket/curl_test HTTP/1.1
> User-Agent: curl/7.29.0
> Host: host73.fqdn:9879
> Accept: */*
> Authorization:AWS user@FQDN:aws_key
>

It makes the connection to the Ozone S3 gateway, but then just hangs. Does anyone know why it's getting stuck? Log messages seems to be inconsistent, and there's nothing in Ranger. Everything we've tried hasn't worked yet.

Re: Ozone S3 with AWS credentials

EFasdfSDfaSDFG — Tue, 30 Jan 2024 06:24:42 GMT

Log messages show errors like:

org.apache.hadoop.ozone.s3.signature.MalformedResourceException

But I can;t find details on what the headers or resource should be.

Re: Ozone S3 with AWS credentials

VidyaSargur — Tue, 30 Jan 2024 08:40:59 GMT

@EFasdfSDfaSDFG, Welcome to our community! To help you get the best possible answer, I have tagged our CDP expert @vaishaakb who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Ozone S3 with AWS credentials

vaishaakb — Mon, 11 Mar 2024 12:49:19 GMT

Hello @EFasdfSDfaSDFG Thank you for bringing this to our community.

I understand you are seeking operability of Ozone with native REST API.
Per my internal research, I learned that the native REST API for Ozone is completely superseded by the S3 REST API and deprecated in the later versions of ozone.

Additionally, I would like to stress that the native REST API neither has official support nor maintained in the Upstream Server.

To help you efficiently,
May I know the current Ozone and the CDP version that you are making use of?

Re: Ozone S3 with AWS credentials

EFasdfSDfaSDFG — Mon, 11 Mar 2024 23:47:53 GMT

Hello @vaishaakb ,

We are running CDP 7.1.8-1.cdh7.1.8.p36.42193080 (and preparing to patch to 7.1.9 soonish), and Ozone 718.2.3-1.ozone718.2.3.p0.46980125.

Re: Ozone S3 with AWS credentials

vaishaakb — Wed, 13 Mar 2024 07:05:37 GMT

@EFasdfSDfaSDFG This still seems broader scope of requirement.
Can you please help elaborate to specific situation on what we are trying to solve?

My understanding is that since you have restriction on installing awscli on RHEL, you are seeking alternative solutions to access/manage Ozone. Is my understanding correct?

Additionally, Did you get a chance to review the available client interfaces for managing Ozone?
https://ozone.apache.org/docs/1.4.0/interface.html

Let me know your thoughts!

Re: Ozone S3 with AWS credentials

EFasdfSDfaSDFG — Fri, 15 Mar 2024 04:44:47 GMT

My understanding is that since you have restriction on installing awscli on RHEL, you are seeking alternative solutions to access/manage Ozone. Is my understanding correct?

This is correct. Specifically I am asking how to/if I can use curl to upload files to Ozone:

@EFasdfSDfaSDFG wrote:
Hi,
We can access/read/write Ozone etc using aws s3api, but would like to see if we can do the same using curl. We may not have the option of using the awscli package in RHEL.
We've tried curl with webhdfs, but that is a bit limited.
This almost seems to work:
$ curl -vvv -H "Authorization:AWS user@FQDN:aws_key" https://host73.fqdn:9879/s3v/bucket/curl_test
But it seems to get stuck here:
> GET /s3v/bucket/curl_test HTTP/1.1
> User-Agent: curl/7.29.0
> Host: host73.fqdn:9879
> Accept: */*
> Authorization:AWS user@FQDN:aws_key
>
It makes the connection to the Ozone S3 gateway, but then just hangs. Does anyone know why it's getting stuck? Log messages seems to be inconsistent, and there's nothing in Ranger. Everything we've tried hasn't worked yet.

As for https://ozone.apache.org/docs/1.4.0/interface.html, Java API might be a possible better option than cURL.

Re: Ozone S3 with AWS credentials

vaishaakb — Wed, 20 Mar 2024 20:05:34 GMT

@EFasdfSDfaSDFG
Took sometime to do a feasibility study and found a couple of things from our Internal resources.

1. FYI: The native Ozone REST API is completely superseded by the S3 REST API.
Therefore, there is no native REST API to manage Ozone.
2. Since I am unsure of your Use-case, I would suggest you test this either by Java API or HttpFS Gateway Interface.[0a]
[0a]https://ozone.apache.org/docs/1.4.0/interface.html

HTH.

If you find this answering your question, Please mark this as Accept as Solution.
Also, You may thank me by clicking the thumbs-up!

Cheers!