Update core-site.xml for hive from Cloudera Manager.

[#1]

Hi,

 I am looking to add the following properties:

<property>
  <name>fs.s3a.access.key</name>
  <value>Amazon S3 Access Key</value>
</property>

<property>
  <name>fs.s3a.secret.key</name>
  <value>Amazon S3 Secret Key</value>
</property>

to the file /etc/hive/conf.cloudera.hive/core-site.xml Now I am adding them manually to the file but if the changes are pushed looks like Cloudera Manager disregards the information already present in the file.

Thanks,

Sumit

[#2]

When the  cluster is managed by Cloudera manager , its bad approach to manually edit the local file .

Cloudera manager by design will not pick up the changes .

You should push the changes using Cloudera manager web ui -> hive ->configuration -> advace config / safety valve . it would should us stale and restart of the service would be needed . 

Hope this helps .

[#3]

Hi @csguna, I am did the same but the updates are not going to the right file and the connections are failing. Hive Service-wide: Advanced: - Hive Service Advanced Configuration Snippets (Safety Valve) for core-site.xml

[#4]

Could you perform a hard stop and start of the Cloudera-scm-agent and let me know if . its reflecting or not .

[#5]

@Sumit

Pls refer the below link, it will explain 

• S3 as storage for Impala tables
• S3 as a source or destination for HDFS and Hive/Impala replication and for cluster storage
• Where to update the credentials, etc

https://www.cloudera.com/documentation/enterprise/5-12-x/topics/cm_auth_aws.html#concept_tmd_nsh_2y

[#6]

Hi @saranvisa,

I did configure the s3 connector. But that is not allowing Hive to Read Write. The blog asks to update the core-site.xml and keep a local copy but I need it for all users.

Thanks,

Sumit.

[#7]

@Sumit

1. What is your Cloudera Version? If you are using Cloudera 5.10 or above then you can follow the instruction from the link that i've given above espectially 'Adding AWS Credentials' . You didn't mentioned whether you have already tried or not

2. Not sure you have restarted your cluster after the configuration change

3. If you follow this option, it will be applicable for all the users

4. I don't know which blog that you are following and how old it is... if you are using cloudera then use cloudera document

[#8]

Hi @saranvisa,

Thank you for your Inputs.

1. What is your Cloudera Version? If you are using Cloudera 5.10 or above then you can follow the instruction from the link that i've given above espectially 'Adding AWS Credentials' . You didn't mentioned whether you have already tried or not
   • Version: Cloudera Express - 5.14.2
   • I have Added "AWS Credentials" in "External Accounts" from "Administration" menu.
2. Not sure you have restarted your cluster after the configuration change
   • Yes, I did restart the cluster couple of times and every time the updates from core-site.xml vanished.
3. If you follow this option, it will be applicable for all the users
4. I don't know which blog that you are following and how old it is... if you are using cloudera then use cloudera document
   • https://www.cloudera.com/documentation/enterprise/5-12-x/topics/cm_s3_clients.html

Thanks,

Sumit

[#9]

@Sumit

The link that you have provided says "If you have selected IAM authentication, no additional steps are needed" - which includes editing the core-site.xml

You have confirmed that added "AWS Credentials" in "External Accounts" from "Administration" menu - which is IAM role based authentication. 

so you don't need to do both