Reply
New Contributor
Posts: 4
Registered: ‎06-08-2018

Update core-site.xml for hive from Cloudera Manager.

Hi,

 

 I am looking to add the following properties:

<property>
  <name>fs.s3a.access.key</name>
  <value>Amazon S3 Access Key</value>
</property>

<property>
  <name>fs.s3a.secret.key</name>
  <value>Amazon S3 Secret Key</value>
</property>


to the file /etc/hive/conf.cloudera.hive/core-site.xml Now I am adding them manually to the file but if the changes are pushed looks like Cloudera Manager disregards the information already present in the file.

 

Thanks,

Sumit

Champion
Posts: 744
Registered: ‎05-16-2016

Re: Update core-site.xml for hive from Cloudera Manager.

When the  cluster is managed by Cloudera manager , its bad approach to manually edit the local file .

Cloudera manager by design will not pick up the changes .

You should push the changes using Cloudera manager web ui -> hive ->configuration -> advace config / safety valve . it would should us stale and restart of the service would be needed . 

Hope this helps .

Posts: 512
Topics: 14
Kudos: 85
Solutions: 45
Registered: ‎09-02-2016

Re: Update core-site.xml for hive from Cloudera Manager.

[ Edited ]

@Sumit

 

Pls refer the below link, it will explain 

  • S3 as storage for Impala tables
  • S3 as a source or destination for HDFS and Hive/Impala replication and for cluster storage
  • Where to update the credentials, etc

https://www.cloudera.com/documentation/enterprise/5-12-x/topics/cm_auth_aws.html#concept_tmd_nsh_2y

 

 

New Contributor
Posts: 4
Registered: ‎06-08-2018

Re: Update core-site.xml for hive from Cloudera Manager.

Hi @csguna, I am did the same but the updates are not going to the right file and the connections are failing. Hive Service-wide: Advanced: - Hive Service Advanced Configuration Snippets (Safety Valve) for core-site.xml
New Contributor
Posts: 4
Registered: ‎06-08-2018

Re: Update core-site.xml for hive from Cloudera Manager.

Hi @saranvisa,

 

I did configure the s3 connector. But that is not allowing Hive to Read Write. The blog asks to update the core-site.xml and keep a local copy but I need it for all users.

 

Thanks,

Sumit.

Posts: 512
Topics: 14
Kudos: 85
Solutions: 45
Registered: ‎09-02-2016

Re: Update core-site.xml for hive from Cloudera Manager.

@Sumit

 

1. What is your Cloudera Version? If you are using Cloudera 5.10 or above then you can follow the instruction from the link that i've given above espectially 'Adding AWS Credentials' . You didn't mentioned whether you have already tried or not

2. Not sure you have restarted your cluster after the configuration change

3. If you follow this option, it will be applicable for all the users

4. I don't know which blog that you are following and how old it is... if you are using cloudera then use cloudera document

 

 

 

New Contributor
Posts: 4
Registered: ‎06-08-2018

Re: Update core-site.xml for hive from Cloudera Manager.

Hi @saranvisa,

 

Thank you for your Inputs.

  1. What is your Cloudera Version? If you are using Cloudera 5.10 or above then you can follow the instruction from the link that i've given above espectially 'Adding AWS Credentials' . You didn't mentioned whether you have already tried or not
    • Version: Cloudera Express - 5.14.2
    • I have Added "AWS Credentials" in "External Accounts" from "Administration" menu.
  2. Not sure you have restarted your cluster after the configuration change
    • Yes, I did restart the cluster couple of times and every time the updates from core-site.xml vanished.
  3. If you follow this option, it will be applicable for all the users
  4. I don't know which blog that you are following and how old it is... if you are using cloudera then use cloudera document

Thanks,

Sumit

Posts: 512
Topics: 14
Kudos: 85
Solutions: 45
Registered: ‎09-02-2016

Re: Update core-site.xml for hive from Cloudera Manager.

@Sumit

 

The link that you have provided says "If you have selected IAM authentication, no additional steps are needed" - which includes editing the core-site.xml

 

You have confirmed that added "AWS Credentials" in "External Accounts" from "Administration" menu - which is IAM role based authentication. 

 

so you don't need to do both

Champion
Posts: 744
Registered: ‎05-16-2016

Re: Update core-site.xml for hive from Cloudera Manager.

Could you perform a hard stop and start of the Cloudera-scm-agent and let me know if . its reflecting or not .
Announcements