I have enabled ACID transactions in Hive-3 within HDP-3 at cluster level. When i am trying to disable it on session level using below properties and creating new managed table using this (TBLPROPERTIES ('orc.compress'='ZLIB'),('transactional'='true')) table properties. It's giving below error : set hive.optimize.index.filter=false; set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager ; set hive.compactor.initiator.on=false; set hive.compactor.worker.threads=0; set hive.strict.managed.tables=false; INFO : Starting task [Stage-0:DDL] in serial mode ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Table ref_edw4x_qn1useh1.dummy failed strict managed table checks due to the following reason: Table is marked as a managed table but is not transactional.) INFO : Completed executing command(queryId=hive_20181207104022_6a3478f0-9b8a-44db-a9a4-4d2ab5fe2b11); Time taken: 0.076 seconds Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Table ref_edw4x_qn1useh1.dummy failed strict managed table checks due to the following reason: Table is marked as a managed table but is not transactional.) (state=08S01,code=1) ... View more

18/11/12 10:37:25 [main]: INFO jdbc.HiveConnection: Connected to server1.test.com:10000 Connected to: Apache Hive (version 3.0.0.3.0.0.0-1334) Driver: Hive JDBC (version 3.0.0.3.0.0.0-1334) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 3.0.0.3.0.0.0-1334 by Apache Hive 0: jdbc:hive2://server1.test.com:> select count(*) from lnd_edw4x_qn1useh1.redemption_fact_t; INFO : Compiling command(queryId=hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216): select count(*) from lnd_edw4x_qn1useh1.redemption_fact_t INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216); Time taken: 1.218 seconds INFO : Executing command(queryId=hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216): select count(*) from lnd_edw4x_qn1useh1.redemption_fact_t INFO : Query ID = hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Subscribed to counters: [] for queryId: hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216 INFO : Tez session hasn't been created yet. Opening session INFO : Dag name: select count(*) from ...h1.redemption_fact_t (Stage-1) ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1542008498326_0022_1_00, diagnostics=[Vertex vertex_1542008498326_0022_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: redemption_fact_t initializer failed, vertex=vertex_1542008498326_0022_1_00 [Map 1], java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.AdlFileSystem not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2595) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3320) I am already having below jars in Hadoop/hive libs still it's not working: hadoop-azure-3.0.0.3.0.0.0-1334.jar, hadoop-azure-datalake-3.0.0.3.0.0.0-1334.jar, azure-data-lake-store-sdk-2.2.7.jar ... View more

I have configured Hiverserver2 in HA mode and started in INFO mode and it is generating logs in INFO mode on server1 but on server2 it's generating in DEBUG mode. I also checked in configuration there is no different configuration for both hiveservers. Below are running hiveserver2 processes on server1 & server2. Server1:~> ps -ef | grep hiveserver2 hive 6717 1 2 Jul02 ? 02:02:18 /usr/lib/jvm/jdk1.8.0_71/bin/java -Xmx1024m -Dhdp.version=2.5.3.0-37 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.5.3.0-37 -Dhadoop.log.dir=/hadoop/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.5.3.0-37/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.5.3.0-37/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx12065m -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/usr/hdp/2.5.3.0-37/hive/bin/../conf/parquet-logging.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.5.3.0-37/hive/lib/hive-service-1.2.1000.2.5.3.0-37.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/hadoop/var/log/hive ======================================= Server2:~> ps -ef | grep hiveserver2 hive 4459 1 3 Jul02 ? 03:30:22 /usr/lib/jvm/jdk1.8.0_71/bin/java -Xmx1024m -Dhdp.version=2.5.3.0-37 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.5.3.0-37 -Dhadoop.log.dir=/hadoop/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.5.3.0-37/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.5.3.0-37/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx12065m -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/usr/hdp/2.5.3.0-37/hive/bin/../conf/parquet-logging.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.5.3.0-37/hive/lib/hive-service-1.2.1000.2.5.3.0-37.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/hadoop/var/log/hive Please respond. ... View more

We have AD/Kerberose integrated HDP cluster in which we have users in AD are in capital letter(e.g. Abc) and same user's also it's principle are in small letter like abc@REALM.COM, when we are trying to use hive view it's showing an error message : "Service 'userhome' check failed: File does not exist: /user/Abc". User abc home directory is present under /user/abc instead of /user/Abc. So please suggest how we can use hive-view? ... View more

Log-File(nifi-user.log) 2017-06-02 13:39:13,641 INFO [NiFi Web Server-11029] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for Arjun-More 2017-06-02 13:39:13,653 INFO [NiFi Web Server-10949] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<user1><UKTEHDPHDF01P.europe.odcorp.net><CN=UKTEHDPHDF03P.europe.odcorp.net, OU=Hadoop>) GET https://test1.example.com:7071/nifi-api/flow/current-user (source ip: 10.128.198.42) 2017-06-02 13:39:13,653 INFO [NiFi Web Server-10949] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user1 2017-06-02 13:39:13,838 INFO [NiFi Web Server-11021] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJBcmp1bi1Nb3JlQEVVUk9QRS5PRENPUlAuTkVUIiwiaXNzIjoiS2VyYmVyb3NTZXJ2aWNlIiwiYXVkIjoiS2VyYmVyb3NTZXJ2aWNlIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiQXJqdW4tTW9yZUBFVVJPUEUuT0RDT1JQLk5FVCIsImtpZCI6NSwiZXhwIjoxNDk2NDUwMzUzLCJpYXQiOjE0OTY0MDcxNTN9.xsEmnpj-muk60JdqKRsyOBn66dw6hJANTG8zgEv-eQ8) GET https://test1.example.com:7071/nifi-api/flow/client-id (source ip: 10.148.171.33) 2017-06-02 13:39:13,838 INFO [NiFi Web Server-10803] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJBcmp1bi1Nb3JlQEVVUk9QRS5PRENPUlAuTkVUIiwiaXNzIjoiS2VyYmVyb3NTZXJ2aWNlIiwiYXVkIjoiS2VyYmVyb3NTZXJ2aWNlIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiQXJqdW4tTW9yZUBFVVJPUEUuT0RDT1JQLk5FVCIsImtpZCI6NSwiZXhwIjoxNDk2NDUwMzUzLCJpYXQiOjE0OTY0MDcxNTN9.xsEmnpj-muk60JdqKRsyOBn66dw6hJANTG8zgEv-eQ8) GET https://test1.example.com:7071/nifi-api/flow/config (source ip: 10.148.171.33) 2017-06-02 13:39:13,838 INFO [NiFi Web Server-11021] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user1 2017-06-02 13:39:13,838 INFO [NiFi Web Server-10803] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user1 2017-06-02 13:39:13,838 INFO [NiFi Web Server-10959] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJBcmp1bi1Nb3JlQEVVUk9QRS5PRENPUlAuTkVUIiwiaXNzIjoiS2VyYmVyb3NTZXJ2aWNlIiwiYXVkIjoiS2VyYmVyb3NTZXJ2aWNlIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiQXJqdW4tTW9yZUBFVVJPUEUuT0RDT1JQLk5FVCIsImtpZCI6NSwiZXhwIjoxNDk2NDUwMzUzLCJpYXQiOjE0OTY0MDcxNTN9.xsEmnpj-muk60JdqKRsyOBn66dw6hJANTG8zgEv-eQ8) GET https://test1.example.com:7071/nifi-api/flow/cluster/summary (source ip: 10.148.171.33) 2017-06-02 13:39:13,839 INFO [NiFi Web Server-10959] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user1 2017-06-02 13:39:13,852 INFO [NiFi Web Server-10551] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<user1><UKTEHDPHDF01P.europe.odcorp.net><CN=UKTEHDPHDF03P.europe.odcorp.net, OU=Hadoop>) GET https://test1.example.com:7071/nifi-api/flow/config (source ip: 10.134.197.41) 2017-06-02 13:39:13,852 INFO [NiFi Web Server-10551] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user1 2017-06-02 13:39:13,868 INFO [NiFi Web Server-11029] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJBcmp1bi1Nb3JlQEVVUk9QRS5PRENPUlAuTkVUIiwiaXNzIjoiS2VyYmVyb3NTZXJ2aWNlIiwiYXVkIjoiS2VyYmVyb3NTZXJ2aWNlIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiQXJqdW4tTW9yZUBFVVJPUEUuT0RDT1JQLk5FVCIsImtpZCI6NSwiZXhwIjoxNDk2NDUwMzUzLCJpYXQiOjE0OTY0MDcxNTN9.xsEmnpj-muk60JdqKRsyOBn66dw6hJANTG8zgEv-eQ8) GET https://test1.example.com:7071/nifi-api/flow/banners (source ip: 10.148.171.33) 2017-06-02 13:39:13,868 INFO [NiFi Web Server-11029] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user1 2017-06-02 13:39:14,067 INFO [NiFi Web Server-10961] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJBcmp1bi1Nb3JlQEVVUk9QRS5PRENPUlAuTkVUIiwiaXNzIjoiS2VyYmVyb3NTZXJ2aWNlIiwiYXVkIjoiS2VyYmVyb3NTZXJ2aWNlIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiQXJqdW4tTW9yZUBFVVJPUEUuT0RDT1JQLk5FVCIsImtpZCI6NSwiZXhwIjoxNDk2NDUwMzUzLCJpYXQiOjE0OTY0MDcxNTN9.xsEmnpj-muk60JdqKRsyOBn66dw6hJANTG8zgEv-eQ8) GET https://test1.example.com:7071/nifi-api/flow/processor-types (source ip: 10.148.171.33) 2017-06-02 13:39:14,067 INFO [NiFi Web Server-10961] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for user1 ... View more