@sseth I have downloaded the latest jar https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar Tried creating the external table and its failing with following error: FAILED: HiveAccessControlException Permission denied: user [abcd] does not have [READ] privilege on [gs://hdp-opt1/forhive/languages] (state=42000,code=40000) I have enabled Hive plugin and set the permission of 777 in coresite.xml Where there any changes made to jar?? I also see few properties have changed in this link: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_cloud-data-access/content/gcp-cluster-config.html Is it mandatory to use the json key? If my vm instance has required permission to talk to gcs ... View more

@Jonathan Sneep I have added the active directory certificate to ambari-truststore but still cluster creation fails with below error: Ambari operation start failed: [component:'INSTALL_START', requestID: '1', context: 'Logical Request: Provision Cluster 'cbd-master1' FAILED: Failed to create the account for HTTP/cbdmaster1-m-0-20180711174907.c.ibxev-edl2-61a040fc.internal@CLUSTER.COM', status: 'ABORTED'] In Manual Installation we have check if KDC/Kerberos credentials are working fine by checking the connection. How can we do the same in cloudbreak? ... View more

@mmolnar @Jonathan Sneep I did follow your recommendations. Thanks for the input. Without enabling Kerberos authentication cluster setup is successful. But enabling the Kerberos security setup fails with following error: Ambari operation start failed: [component:'INSTALL_START', requestID: '1', context: 'Logical Request: Provision Cluster 'cbd-master' FAILED: unable to find valid certification path to requested target', status: 'ABORTED'] Please find my Kerberos details. Could it be failing because of any of below parameters. Kerberos Type: Existing KDC with active directory Kerberos URL: 35.237.19.153 Kerberos Admin URL: 35.237.19.153 Kerberos Realm: CLUSTER.COM Kerberos AD Ldap URL: ldaps://35.237.19.153:636 Kerberos AD Container DN: ou=hdpusers,dc=cluster,dc=com Use TCP connection: Yes ... View more

@mmolnar I am using GCP platform and trying to configure Datalake service by enabling Kerberos. External Authentication test succeeded. But when creating a cluster by enabling Kerberos and adding required KDC credentials it failing with the error attached. Below are my Kerberos Details: KDC host: hdp-ad.cluster.com ( 35.237.19.153 ). I am not why kerberosization is failing. ldap-error.txt ... View more

@Jonathan Sneep Thanks for the input, tried the cluster setup but it failed with the following error: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later. I was able to fix the errors in ambari, I have 2 questions hope you would be able to answer 1. How do i sync ambari with cloudbreak 2. Post cluster creation failure, cloudbreak UI doesn't load at all. ... View more

@pooja kamle, Were you able to setup single ranger instance for emphermal clusters ... View more

Hi @sseth Issue is resolved after adding following property in core-site.xml fs.gs.reported.permissions=777 Normal Users can access hive and create external table pointing to GCS location. ... View more

@Geoffrey Shelton Okot I was able to create hive external table pointing my storage as GCS. But it only works as hive superuser but doesn't work as a normal hive user meaning, hdpuser1 cannot create hive table it fails with above error, but if execute su - hive it works . I am no sure how to rectify this. ... View more

@Geoffrey Shelton Okot I did try, but still fails. CommandException: hdpuser1:WRITE is not a valid ACL changehdpuser1 is not a valid scope type The GCS bucket has storage admin rights given to service account hadoop fs -ls gs://bucket/ = works fine ... View more

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.security.AccessControlException: Permission denied: user=hdpuser1, path="gs://bucket/":hive:hive:drwx------) (state=08S01,code=1) hdpuser1 is an AD user, using the same user I execute $ hdfs dfs -ls gs://bucket/ but using beeline when I try to create an external table it fails ... View more

Hi Neeraj, We are trying to test GCS connector with HDP 2.6.5 ( https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_cloud-data-access/content/authentication-gcp.html ) by having GCS as my storage. When trying to create hive external table it’s failing with the following error: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.security.AccessControlException: Permission denied: user=hdpuser1, path="gs://hive_metastore/":hive:hive:drwx------) (state=08S01,code=1) Syntax: CREATE EXTERNAL TABLE test1256(name string,id int) LOCATION 'gs://hive_metastore/' ; ... View more

Hi, Thanks a lot for the info, but still facing the same issue. I did create the user in AD and have a valid ticket , hdfs command does work accessing the GCS but cannot create external hive table. ... View more

Hi Vipin, Thanks for the feedback. Can we build a custom plugin https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741207 for GCS. Ranger did have a plugin a WASB in azure in the future do we have one for GCS as well? ... View more

Hi, thanks for the reply, but i don't see snapshot for the below mentioned user id -bash-4.2$ hdfs dfs -du -h /user/alin/ 1.3 T /user/alin/.Trash 40.5 M /user/alin/.hiveJars -bash-4.2$ hdfs dfs -du -h /user/alin/.snapshot du: `/user/alin/.snapshot': No such file or directory -bash-4.2$ hdfs dfs -ls /user/alin/.snapshot ls: `/user/alin/.snapshot': No such file or directory I have extracted fsimage and using ELK to see the storage used by hdfs, the space consumption of certain directories don;t match with the actual size on the cluster. Eg: /user/alin is consuming 1 TB of storage according hdfs dfs du command but in fsimage it shows that same user is consuming 40.5 M of storage.... on performing hdfs ls command on /user/alin i dont see any files of 1 TB As checked with my operations they say because of hdfs snapshot saved on the cluster it gives those values as blocks are still allocated..but i don't see any doc in hortonworks mentioning that... If thats the case how do i calculate the actual storage used by HDFS? does fsimage give accurate data how to make sure fsimage also get snapshot details ? ... View more

Hi Klien, Thanks for the info..can you let me know how is that calculated. I have been reading forums but i couldn't understand. Eg: I have application A with 4576 Gb of data i was able to calculate the space on hdfs which is 1804 Gb and requires 1.12 data nodes as data node we have are of 23 TB.. I want to know how does it impact the existing setup?? as u mentioned above how will DFS, remaining, and non DFS change. ... View more

Hi, Thanks for the reply I know that 300 GB of data will be used by HDFS..but my question is with 300 GB of data how will it impact the existing setup?? In ambari i can see above details how will that change and with more and more application coming in, how to assess the impact ... View more

Hi Can anyone please let me know how to proceed with the request?? I can extract ranger policies via the following command: curl -iv -u username:passwd -H "Content-type:application/json" -X GET http://sandbox.com:6080/service/public/v2/api/policy I need the automate the script such that it there is no need to enter the username and passwd... is it possible ?? is there any API from which i can test it.... ... View more

I have the same command: curl -iv -u admin:admin -H "Content-type:application/json" -X GET http://sandbox.com:6080/service/public/v2/api/policy My question: Is it possible to automate the above command such that it doesn't ask for ranger admin user name and password and it authenticates via kerboros authentication ... View more

Can not automate the ranger admin curl command wherein it doesn't ask for username and passwd for ranger credentials ... View more