Member since
02-10-2018
21
Posts
0
Kudos Received
0
Solutions
09-07-2018
07:52 PM
I am able to create external table on s3 using hive. It is giving above issue only in Hiveserver2/beeline.
... View more
09-07-2018
07:52 PM
Hi, We have hortonwork cluster with kerberos and sqlstdautherization. Both security work fine excepts for create table for other users except hive. hive and hdfs are part of admin users.
Hive user is able to create all kinds of table but hdfs user not. S3 policy is open for all users. I am able to use hadoop command to see s3 bucket in hdfs user. I am able to create local/external table on hdfs user. but I am not able to create external table on S3 using hdfs user. I am getting below error.
Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: Principal [name=hdfs, type=USER] does not have following privileges for operation CREATETABLE [[OBJECT OWNERSHIP] on Object [type=DFS_URI, name=s3a://bucket_name/user/hdfs/TEST]] (state=42000,code=40000)
... View more
Labels:
- Labels:
-
Apache Hive
06-30-2018
03:31 AM
Thanks Vinicius Higa Murakami it is very helpful.
... View more
06-25-2018
09:13 PM
When we run beeline query on big table, the /tmp space is going up to 1 TB and total usage is 3 TB due to replication factor. can we minimize /tmp space usage?
... View more
Labels:
- Labels:
-
Apache Hive
06-14-2018
11:49 PM
its look like some linux update issue. what kind of precaution we can take to make hadoop component installation proper. I run apt-get update on each amazon instance while it create. -First time i am able to create cluster up to date. - I am getting this error second time only. i am using external postgres database. Please help me on this.
... View more
06-14-2018
11:39 PM
I am getting below error only one node while try to create 3 node cluster using ansible hortonworks. - not able to install any hadoop component. - Its look like hadoop component not installing. but amabari-agent is fine and i can see node in ambari. - i am using ubuntu 14.04 i tried below but didn't work and giving same error: apt-get clean apt-get update apt-get install -f Error: dpkg: error processing package hadoop-2-6-2-0-205-hdfs-zkfc (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libhdfs0-2-6-2-0-205:
libhdfs0-2-6-2-0-205 depends on hadoop-2-6-2-0-205 (= 2.7.3.2.6.2.0-205); however:
Package hadoop-2-6-2-0-205 is not configured yet.
dpkg: error processing package libhdfs0-2-6-2-0-205 (--configure):
dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
No apport report written because the error message indicates its a followup error from a previous failure.
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
Errors were encountered while processing:
zookeeper-2-6-2-0-205
hadoop-2-6-2-0-205
hadoop-2-6-2-0-205-hdfs
hadoop-2-6-2-0-205-yarn
hadoop-2-6-2-0-205-mapreduce
hadoop-2-6-2-0-205-client
hadoop-2-6-2-0-205-hdfs-datanode
hadoop-2-6-2-0-205-hdfs-journalnode
hadoop-2-6-2-0-205-hdfs-namenode
hadoop-2-6-2-0-205-hdfs-secondarynamenode
hadoop-2-6-2-0-205-hdfs-zkfc
libhdfs0-2-6-2-0-205
E: Sub-process /usr/bin/dpkg returned an error code (1)
2018-06-14 23:00:41,091 - Failed to install package libhdfs0-2-6-2-0-205. Executing '/usr/bin/apt-get update -qq'
2018-06-14 23:00:45,600 - Retrying to install package libhdfs0-2-6-2-0-205 after 30 seconds
... View more
Labels:
- Labels:
-
Apache Hadoop
06-02-2018
04:48 PM
Hi i am not able to run sql on hive. I have kerberos cluster with postgres external metadata. -- do i need kerberos ticket to run query in zeppline. I think its not required. -- I think it has to do with access but i don't know hwat is gpaadmin Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "gpadmin" %jdbc show databases p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #212121; -webkit-text-stroke: #212121; background-color: #ffffff}
span.s1 {font-kerning: none}
span.Apple-tab-span {white-space:pre} org.apache.zeppelin.interpreter.InterpreterException: Error in doAs at org.apache.zeppelin.jdbc.JDBCInterpreter.getConnection(JDBCInterpreter.java:415) at org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(JDBCInterpreter.java:633) at org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(JDBCInterpreter.java:733) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:502) at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelScheduler.java:162) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1884) at org.apache.zeppelin.jdbc.JDBCInterpreter.getConnection(JDBCInterpreter.java:407) ... 13 more Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "gpadmin" at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:420) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:195) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:127) at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:41) at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) at org.postgresql.Driver.makeConnection(Driver.java:414) at org.postgresql.Driver.connect(Driver.java:282) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:208) at org.apache.commons.dbcp2.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:79) at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:205) at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435) at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) at org.apache.commons.dbcp2.PoolingDriver.connect(PoolingDriver.java:129) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:270) at org.apache.zeppelin.jdbc.JDBCInterpreter.getConnectionFromPool(JDBCInterpreter.java:362) at org.apache.zeppelin.jdbc.JDBCInterpreter.access$000(JDBCInterpreter.java:89) at org.apache.zeppelin.jdbc.JDBCInterpreter$1.run(JDBCInterpreter.java:410) at org.apache.zeppelin.jdbc.JDBCInterpreter$1.run(JDBCInterpreter.java:407) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
... View more
Labels:
- Labels:
-
Apache Zeppelin
05-15-2018
05:55 PM
Can i use knox instead of ranger?
... View more
05-14-2018
08:04 PM
I have to setup user level access to database. I want to setup read write access for specific users on hive database as give below. e.g. i have hive database db1 and users user1 and user2. Want to setup below access for users
db1 -> user1 all access(Read, Write, create, delete, update)
user2 only Read access(Read)
... View more
Labels:
- Labels:
-
Apache Hive
03-14-2018
04:05 PM
Thanks you very much for your prompt reply. I am able to run map reduce with 300 GB usage. I think i need to look some other parameter. your hint may help me. currently i can not run it but will do it soon.
... View more
03-14-2018
03:48 PM
screen-shot-2018-03-14-at-84702-am.png If you see i am able to run 3 TEZ job. but when i run 3 mapred job in hive.execution.engine=mr in hive. it is accepting one at a time despite enough space.
... View more
03-14-2018
03:40 PM
screen-shot-2018-03-14-at-83913-am.png See th attached file it is inheriting all the resource and it is 100%.
... View more
03-13-2018
10:14 PM
default queue accepting one mr job for hive while tez accepting more than one. default execution engine is Tez. in this case default queue running 3 different query parallely in 3 different session. While i set hive.execution.engine=mr in this case i am able to open multiple hive session but default queue run only one query at a time. My cluster is big and already tried multiple small job parallely for Tez. but setting hive.execution.engine=mr run only one job at a time in default queue.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
-
Apache Tez
03-05-2018
06:26 PM
Thanks. I think something to do with metastore. i am using extrenal metastore which is loacted on RDS instance with postgresql. hive> show tables; FAILED: Error in acquiring locks: Error communicating with the metastore hive> SET hive.support.concurrency=true; SET hive.enforce.bucketing=true; SET hive.exec.dynamic.partition.mode=nonstrict; SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; SET hive.compactor.initiator.on=true; SET hive.compactor.worker.threads=1; set hive.optimize.sort.dynamic.partition=false; hive> show locks; OK Lock IDDatabaseTablePartitionStateBlocked ByTypeTransaction IDLast HeartbeatAcquired AtUserHostnameAgent Info 1.1nmp_devNULLNULLWAITING SHARED_READNULL1520037557808NULLanonymousip-10-16-42-185hive_20180303003917_80949492-5c2b-4aa6-a905-1093ff2a4bea 2.1defaultNULLNULLWAITING SHARED_READNULL1520038346314NULLanonymousip-10-16-42-185hive_20180303005226_ee4e77bc-f57b-42eb-b477-5af80896dddf 3.1defaultNULLNULLWAITING SHARED_READNULL1520038760316NULLadminip-10-16-42-185hive_20180303005920_82a05407-9848-4a35-9717-f7f495ddfd96 4.1defaultNULLNULLWAITING SHARED_READNULL1520038767616NULLadminip-10-16-42-185hive_20180303005927_11f18092-df0e-4244-a997-4469e81f4fad 5.1defaultNULLNULLWAITING SHARED_READNULL1520038783927NULLadminip-10-16-42-185hive_20180303005943_8725c5da-4cd3-4ca4-b35a-de727ede2f06 6.1nmp_devNULLNULLWAITING SHARED_READNULL1520038948139NULLanonymousip-10-16-42-185hive_20180303010228_b2b9c13f-c2f0-4b67-830b-7a952ed33e73 8.1nmp_devNULLNULLWAITING SHARED_READNULL1520040369997NULLanonymousip-10-16-42-185hive_20180303012609_c1aaf992-9ae4-47d4-a5dd-6dba6a87c20f 9.1defaultNULLNULLWAITING SHARED_READNULL1520040691362NULLanonymousip-10-16-42-185hive_20180303013130_5c4a2821-1361-4447-98c7-8629376665f4 10.1nmp_devNULLNULLWAITING SHARED_READNULL1520040715385NULLanonymousip-10-16-42-185hive_20180303013155_97d1549c-4588-4512-ada7-1015adec8d29 11.1defaultNULLNULLWAITING SHARED_READNULL1520207255470NULLanonymousip-10-16-42-185hive_20180304234735_9763e36c-d54b-45ae-b1d8-3881f2d9abe9 12.1defaultNULLNULLWAITING SHARED_READNULL1520208512712NULLanonymousip-10-16-42-185hive_20180305000831_fcb7ef1e-1936-4dfd-a878-6e09c32cc045 13.1defaultNULLNULLWAITING SHARED_READNULL1520209061214NULLanonymousip-10-16-42-185hive_20180305001740_9814fd52-f1e6-47cb-98a5-7b91ac336170 14.1defaultNULLNULLWAITING SHARED_READNULL1520223616543NULLhdfsip-10-16-40-241hdfs_20180305042015_3ac75d63-13b2-495e-a49b-51affcf48dcb 15.1defaultNULLNULLWAITING SHARED_READNULL1520223629549NULLhdfsip-10-16-40-241hdfs_20180305042028_ecbac190-eda7-42a2-adf9-ec588ed3418e 16.1defaultNULLNULLWAITING SHARED_READNULL1520233942704NULLanonymousip-10-16-42-185hive_20180305071221_4f8265ff-a29c-44a7-abf2-67d3895443b0 7.1nmp_devemb_ipthr_25feb18_02mar18_bucketNULLWAITING EXCLUSIVENULL1520039081989NULLanonymousip-10-16-42-185hive_20180303010441_07bc9b06-26a8-4a6d-a353-6123d5d25592 Time taken: 0.211 seconds, Fetched: 17 row(s) hive> show tables; FAILED: Error in acquiring locks: Error communicating with the metastore
... View more
03-05-2018
03:10 AM
hi, iam able to reslove above issue by setting set hive.support.concurrency=false. but the table is not accessible as ACID property must need it true. i have doubt why it hanging if i make hive.support.concurrency==true. any help or suggestion?
... View more
03-05-2018
01:24 AM
I am getting Below error while trying to insert data in ACID property table. set hive.support.concurrency=true; set hive.enforce.bucketing=true; set hive.exec.dynamic.partition.mode=nonstrict; set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; set hive.compactor.initiator.on=true; set hive.compactor.worker.threads=1;
CREATE TABLE students (name VARCHAR(64), age INT, gpa DECIMAL(3, 2))
CLUSTERED BY (age) INTO 2 BUCKETS STORED AS ORC
TBLPROPERTIES ('transactional'='true'); 0: jdbc:hive2://localhost185:10000/> INSERT INTO TABLE students VALUES ('AA', 23, 1.28), ('BB', 32, 2.32); Error: Error while processing statement: FAILED: Error in acquiring locks: Error communicating with the metastore (state=42000,code=10)
... View more
03-04-2018
06:02 PM
Thank Toy very Much for support. @Vikas Srivastava I have set all the ACID parameter as said above. @Deepak Sharma Where can i see hiveserver2 logs? Hive shell is hanging after setting below param. SET hive.txn.manager= org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
... View more
03-03-2018
12:51 AM
I am using HDP 2.6.1 While i set ACID property hive getting Hang, and i forced to do ctl+c I have created a table with the following command: hive> create table test (x int, y string) clustered by (x) into 2 buckets stored as ORC tblproperties ("transactional" = "true"); OK Time taken: 0.148 seconds Further, I have set the following Hive properties: SET hive.support.concurrency=true; SET hive.enforce.bucketing=true; SET hive.exec.dynamic.partition.mode=nonstrict; SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; SET hive.compactor.initiator.on=true; SET hive.compactor.worker.threads=1; set hive.optimize.sort.dynamic.partition=false; Then on, I am doing a test insert into the hive table leveraging the following command: INSERT INTO TEST VALUES (1,a); Hive CLI shell is getting hung and I have to run Ctrl + C command to get out of the shell. I am not able to understand why is this happening ? Any help to resolve this problem will be very helpful.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
02-11-2018
05:07 PM
at the end of the retry i got below error. As i mentioned in above post where ambari agrnt is listening. FAILED - RETRYING: Make sure all of the Ambari Agents have registered (2 retries left). FAILED - RETRYING: Make sure all of the Ambari Agents have registered (1 retries left). failed: [master01] (item=slave01) => {"attempts": 100, "cache_control": "no-store", "changed": false, "connection": "close", "content_type": "text/plain", "expires": "Thu, 01 Jan 1970 00:00:00 GMT", "failed": true, "item": "slave01", "msg": "HTTP Error 404: Not Found", "pragma": "no-cache", "redirected": false, "set_cookie": "AMBARISESSIONID=1ccd4t86mvfvo10ijbzzoae1dv;Path=/;HttpOnly", "status": 404, "url": "http://ip-10-16-42-20.us-west-2.compute.internal:8080/api/v1/hosts/ip-10-16-46-142.us-west-2.compute.internal", "user": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "x_content_type_options": "nosniff", "x_frame_options": "DENY", "x_xss_protection": "1; mode=block"} NO MORE HOSTS LEFT ********************************************************************************************************************************************************************* to retry, use: --limit @/root/ansible-hortonworks/playbooks/install_cluster.retry PLAY RECAP ***************************************************************************************************************************************************************************** localhost: ok=9changed=6 unreachable=0failed=0 master01 : ok=68 changed=4 unreachable=0failed=1 slave01: ok=21 changed=1 unreachable=0failed=0 (ansible)root@ip-10-16-42-20:~/ansible-hortonworks# netstat -anp
... View more
02-11-2018
05:07 PM
https://github.com/hortonworks/ansible-hortonworks . I am using ansible-hortonwork to install hadoop on ec2 instance. I also tried ansible-hadoop. in both cases i got error for Ambari Agents registration.
TASK [ambari-config : Modify the HDF base_url repository for HDF 3.0] ******************************************************************************************************************
skipping: [master01]
TASK [ambari-config : Make sure all of the Ambari Agents have registered] **************************************************************************************************************
ok: [master01] => (item=master01)
FAILED - RETRYING: Make sure all of the Ambari Agents have registered (100 retries left).
FAILED - RETRYING: Make sure all of the Ambari Agents have registered (99 retries left).
FAILED - RETRYING: Make sure all of the Ambari Agents have registered (98 retries left).
FAILED - RETRYING: Make sure all of the Ambari Agents have registered (97 retries left).
FAILED - RETRYING: Make sure all of the Ambari Agents have registered (96 retries left).
FAILED - RETRYING: Make sure all of the Ambari Agents have registered (95 retries left).
FAILED - RETRYING: Make sure all of the Ambari Agents have registered (94 retries left).
FAILED - RETRYING: Make sure all of the Ambari Agents have registered (93 retries left).
I am able to do ssh and password less connection all setup looks fine.
I don't unsderstand whats going on as ambari server is up and amabari agent is up. i think something is related to network and port. I see below things while do netstat -anp
It is something to do with ipv4 and ipv6. instance have ubuntu (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)).
Salve: netstat -anp
tcp00 0.0.0.0:86700.0.0.0:* LISTEN521/python
Master: netstat -anp
tcp00 0.0.0.0:86700.0.0.0:* LISTEN4202/python
tcp6 00 :::8440 :::*LISTEN26419/java
tcp6 00 :::5432 :::*LISTEN16983/postgres
tcp6 00 :::8441 :::*LISTEN26419/java
tcp6 00 ::1:25:::*LISTEN2264/master
tcp6 00 :::8080 :::*LISTEN26419/java
... View more
Labels:
- Labels:
-
Apache Ambari