Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Update core-site.xml for hive from Cloudera Manager.

avatar
Explorer

Hi,

 

 I am looking to add the following properties:

<property>
  <name>fs.s3a.access.key</name>
  <value>Amazon S3 Access Key</value>
</property>

<property>
  <name>fs.s3a.secret.key</name>
  <value>Amazon S3 Secret Key</value>
</property>


to the file /etc/hive/conf.cloudera.hive/core-site.xml Now I am adding them manually to the file but if the changes are pushed looks like Cloudera Manager disregards the information already present in the file.

 

Thanks,

Sumit

8 REPLIES 8

avatar
Champion

When the  cluster is managed by Cloudera manager , its bad approach to manually edit the local file .

Cloudera manager by design will not pick up the changes .

You should push the changes using Cloudera manager web ui -> hive ->configuration -> advace config / safety valve . it would should us stale and restart of the service would be needed . 

Hope this helps .

avatar
Explorer
Hi @csguna, I am did the same but the updates are not going to the right file and the connections are failing. Hive Service-wide: Advanced: - Hive Service Advanced Configuration Snippets (Safety Valve) for core-site.xml

avatar
Champion
Could you perform a hard stop and start of the Cloudera-scm-agent and let me know if . its reflecting or not .

avatar
Champion

@Sumit

 

Pls refer the below link, it will explain 

  • S3 as storage for Impala tables
  • S3 as a source or destination for HDFS and Hive/Impala replication and for cluster storage
  • Where to update the credentials, etc

https://www.cloudera.com/documentation/enterprise/5-12-x/topics/cm_auth_aws.html#concept_tmd_nsh_2y

 

 

avatar
Explorer

Hi @saranvisa,

 

I did configure the s3 connector. But that is not allowing Hive to Read Write. The blog asks to update the core-site.xml and keep a local copy but I need it for all users.

 

Thanks,

Sumit.

avatar
Champion

@Sumit

 

1. What is your Cloudera Version? If you are using Cloudera 5.10 or above then you can follow the instruction from the link that i've given above espectially 'Adding AWS Credentials' . You didn't mentioned whether you have already tried or not

2. Not sure you have restarted your cluster after the configuration change

3. If you follow this option, it will be applicable for all the users

4. I don't know which blog that you are following and how old it is... if you are using cloudera then use cloudera document

 

 

 

avatar
Explorer

Hi @saranvisa,

 

Thank you for your Inputs.

  1. What is your Cloudera Version? If you are using Cloudera 5.10 or above then you can follow the instruction from the link that i've given above espectially 'Adding AWS Credentials' . You didn't mentioned whether you have already tried or not
    • Version: Cloudera Express - 5.14.2
    • I have Added "AWS Credentials" in "External Accounts" from "Administration" menu.
  2. Not sure you have restarted your cluster after the configuration change
    • Yes, I did restart the cluster couple of times and every time the updates from core-site.xml vanished.
  3. If you follow this option, it will be applicable for all the users
  4. I don't know which blog that you are following and how old it is... if you are using cloudera then use cloudera document

Thanks,

Sumit

avatar
Champion

@Sumit

 

The link that you have provided says "If you have selected IAM authentication, no additional steps are needed" - which includes editing the core-site.xml

 

You have confirmed that added "AWS Credentials" in "External Accounts" from "Administration" menu - which is IAM role based authentication. 

 

so you don't need to do both