question Setting max S3 connections in Archives of Support Questions (Read Only)

Setting max S3 connections

Pettax — Fri, 16 Sep 2022 10:50:05 GMT

Hi all,

Best Practices for Using Impala with S3 states "Set the safety valve fs.s3a.connection.maximum to 1500 for impalad."

Can annyone clarify which safety valve field should be used and with what syntax? I'm reading somewhere that this setting belongs to core-site.xml but Impala configuration in Cloudera Manger does not seem to have a safety valve for core-site.xml. The instructions mentions safety valve for impalad but that safety valve seems to be for command line arguments to impalad.

The problem we are trying to adress is

hdfsSeek(desiredPos=503890631): FSDataInputStream#seek error:
com.cloudera.com.amazonaws.AmazonClientException: Unable to execute HTTP request: Timeout waiting for connection from pool

that we keep getting when using Impala for querying data stored in S3.

We are using CDH 5.8.3

Thanks,

Petter

Re: Setting max S3 connections

saileshmukil — Wed, 30 Nov 2016 16:32:19 GMT

Hi Pettax,

You should be able to find the safety valve in the Cloudera Manager under the HDFS service. The S3AConnector used by Impala is managed by the HDFS service. It will be under the title: "Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml".

Let me know if you have any other issues.

- Sailesh

Re: Setting max S3 connections

Pettax — Tue, 06 Dec 2016 16:12:08 GMT

Thank you Sailesh!

This solved my problem.

Br,

Petter