Member since
06-08-2014
33
Posts
6
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2157 | 11-23-2018 11:18 AM |
12-07-2018
01:15 PM
I have enabled ACID transactions in Hive-3 within HDP-3 at cluster level. When i am trying to disable it on session level using below properties and creating new managed table using this (TBLPROPERTIES ('orc.compress'='ZLIB'),('transactional'='true')) table properties. It's giving below error : set hive.optimize.index.filter=false; set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager ; set hive.compactor.initiator.on=false; set hive.compactor.worker.threads=0; set hive.strict.managed.tables=false; INFO : Starting task [Stage-0:DDL] in serial mode
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Table ref_edw4x_qn1useh1.dummy failed strict managed table checks due to the following reason: Table is marked as a managed table but is not transactional.)
INFO : Completed executing command(queryId=hive_20181207104022_6a3478f0-9b8a-44db-a9a4-4d2ab5fe2b11); Time taken: 0.076 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Table ref_edw4x_qn1useh1.dummy failed strict managed table checks due to the following reason: Table is marked as a managed table but is not transactional.) (state=08S01,code=1)
... View more
Labels:
- Labels:
-
Apache Hive
11-23-2018
11:18 AM
@Prabhu M @Mahesh Balakrishnan : Issue has been resolved. It's a bug with Tez version 0.9.0 (HDP-3.0.0.0) which they have fixed in Tez 0.9.1 version(HDP-3.0.1.0)
... View more
11-12-2018
06:07 PM
18/11/12 10:37:25 [main]: INFO jdbc.HiveConnection: Connected to server1.test.com:10000
Connected to: Apache Hive (version 3.0.0.3.0.0.0-1334)
Driver: Hive JDBC (version 3.0.0.3.0.0.0-1334)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.0.0.3.0.0.0-1334 by Apache Hive
0: jdbc:hive2://server1.test.com:> select count(*) from lnd_edw4x_qn1useh1.redemption_fact_t; INFO : Compiling command(queryId=hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216): select count(*) from lnd_edw4x_qn1useh1.redemption_fact_t INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216); Time taken: 1.218 seconds INFO : Executing command(queryId=hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216): select count(*) from lnd_edw4x_qn1useh1.redemption_fact_t INFO : Query ID = hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Subscribed to counters: [] for queryId: hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216 INFO : Tez session hasn't been created yet. Opening session INFO : Dag name: select count(*) from ...h1.redemption_fact_t (Stage-1) ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1542008498326_0022_1_00, diagnostics=[Vertex vertex_1542008498326_0022_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: redemption_fact_t initializer failed, vertex=vertex_1542008498326_0022_1_00 [Map 1], java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.AdlFileSystem not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2595) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3320) I am already having below jars in Hadoop/hive libs still it's not working: hadoop-azure-3.0.0.3.0.0.0-1334.jar, hadoop-azure-datalake-3.0.0.3.0.0.0-1334.jar, azure-data-lake-store-sdk-2.2.7.jar
... View more
Labels:
- Labels:
-
Apache Hive
07-06-2018
01:50 PM
I am having only hiveserver2 service on Server2.
... View more
07-06-2018
12:47 PM
I have configured Hiverserver2 in HA mode and started in INFO mode and it is generating logs in INFO mode on server1 but on server2 it's generating in DEBUG mode. I also checked in configuration there is no different configuration for both hiveservers. Below are running hiveserver2 processes on server1 & server2. Server1:~> ps -ef | grep hiveserver2
hive 6717 1 2 Jul02 ? 02:02:18 /usr/lib/jvm/jdk1.8.0_71/bin/java -Xmx1024m -Dhdp.version=2.5.3.0-37 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.5.3.0-37 -Dhadoop.log.dir=/hadoop/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.5.3.0-37/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.5.3.0-37/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx12065m -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/usr/hdp/2.5.3.0-37/hive/bin/../conf/parquet-logging.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.5.3.0-37/hive/lib/hive-service-1.2.1000.2.5.3.0-37.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/hadoop/var/log/hive ======================================= Server2:~> ps -ef | grep hiveserver2 hive 4459 1 3 Jul02 ? 03:30:22 /usr/lib/jvm/jdk1.8.0_71/bin/java -Xmx1024m -Dhdp.version=2.5.3.0-37 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.5.3.0-37 -Dhadoop.log.dir=/hadoop/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.5.3.0-37/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.5.3.0-37/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx12065m -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/usr/hdp/2.5.3.0-37/hive/bin/../conf/parquet-logging.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.5.3.0-37/hive/lib/hive-service-1.2.1000.2.5.3.0-37.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/hadoop/var/log/hive Please respond.
... View more
Labels:
- Labels:
-
Apache Hive
08-17-2017
06:25 PM
Hey Nitin, Changed the configs like, object class=user Distinguished name attribute=distinguishedName Group member attribute=member And re-synced users. It is working now. Thanks for help.
... View more
08-17-2017
06:22 PM
Yes this helps a lot. Thanks for quick answer.
... View more
08-17-2017
06:08 PM
I have below set of properties, authentication.ldap.groupMembershipAttr=memberOf authentication.ldap.groupNamingAttr=cn authentication.ldap.groupObjectClass=group authentication.ldap.userObjectClass=person authentication.ldap.usernameAttribute=sAMAccountName authentication.ldap.dnAttribute=dn
... View more
08-17-2017
04:52 PM
1 Kudo
Is there a way to find out - List Users/jobs exhausting the YARN queues , running with more than 4 TB of memory and 400 containers in last 2 weeks
... View more
Labels:
- Labels:
-
Apache YARN