Hive with Google Cloud Storage

[#1]

I have installed a hadoop 2.6.5 version cluster in GCP using VM's instances. Used GCP connector and pointed by hdfs to use gs bucket. Added the below 2 entries in coresite.xml:

google.cloud.auth.service.account.json.keyfile=<Path-to-the-JSON-file> 
fs.gs.working.dir=/

When using hadoop gs -ls / works fine , but when I am creating a hive tables

CREATE EXTERNAL TABLE test1256(name string,id  int)   LOCATION   'gs://bucket/';

I get the following error:

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.security.AccessControlException: Permission denied: user=hdpuser1, path="gs://bucket/":hive:hive:drwx------) (state=08S01,code=1)

Apart form changes to coresite.xml are there any changes to be made at hive.xml also?

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_cloud-data-access/content/authentication...

[#2]

Hi @sseth

Issue is resolved after adding following property in core-site.xml

fs.gs.reported.permissions=777

Normal Users can access hive and create external table pointing to GCS location.

[#3]

@sseth

I have downloaded the latest jar

https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar

Tried creating the external table and its failing with following error:

FAILED: HiveAccessControlException Permission denied: user [abcd] does not have [READ] privilege on [gs://hdp-opt1/forhive/languages] (state=42000,code=40000)

I have enabled Hive plugin and set the permission of 777 in coresite.xml

Where there any changes made to jar?? I also see few properties have changed in this link:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_cloud-data-access/content/gcp-cluster-co...

Is it mandatory to use the json key? If my vm instance has required permission to talk to gcs

[#4]

@sudi ts Were you able to resolve this issue?

[#5]

Hi, 

You need to set in hive-site.xml, these three tags to get this working with hive:

<property>
<name>google.cloud.auth.service.account.json.keyfile</name>
<value>/home/hadoop/keyfile.json</value>
</property>

<property>
<name>fs.gs.reported.permissions</name>
<value>777</value>
</property>
<property>
<name>fs.gs.path.encoding</name>
<value>/home/hadoop/</value>
</property>

Same xml tags we can have it on hadoop in core-site.xml to have it working with hdfs,

On beeline, just execute this and it shall work:

INSERT OVERWRITE DIRECTORY 'gs://bucket/table' ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' SELECT * FROM table;

Please upvote if you found helpful!