Member since
11-13-2017
50
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2855 | 11-15-2018 12:35 PM |
11-15-2018
12:35 PM
Solution for the desparate souls like me. copied atlas-application.properties from /etc/atlas/conf to /etc/hbase/conf changed permission to 744 and ownership to atlas:hadoop Restart Hbase and Atlas. Don't know if its the right thing but worked!!! Start to laugh again !!!.
... View more
11-15-2018
12:16 PM
Solution for the desparate souls like me. copied atlas-application.properties from /etc/atlas/conf to /etc/hbase/conf changed permission to 744 and ownership to atlas:hadoop Restart Hbase and Atlas. Don't know if its the right thing but worked!!! Start to laugh again !!!.
... View more
11-15-2018
10:07 AM
Hi, After upgrading to 3.0.1 and enabling kerberos, Hbase master is failing to start due to missing 'master' znode. It is leading to Atlas start failure. atlas
TABLE
Took 8.2656 secondsjava exception
ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master
2018-11-14 17:27:34,307 - Retrying after 10 seconds. Reason: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-003@IIM.LOCAL; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_janus
ATLAS_ENTITY_AUDIT_EVENTS
atlas
TABLE
Took 8.2337 secondsjava exception In zookeeper 'master' znode is present is hbase-unsecure but not in secure-hbase [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-unsecure
[replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, master, switch, running, draining, namespace, hbaseid, table]
[zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-secure [replication, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, switch, running, tokenauth, draining, hbaseid, table] How do I fix this? Is this a known issue in 3.0.1? Thanks in advance.
... View more
Labels:
- Labels:
-
Apache HBase
11-14-2018
05:29 PM
Some more logs atlas
TABLE
Took 8.2656 secondsjava exception
ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master
2018-11-14 17:27:34,307 - Retrying after 10 seconds. Reason: Execution of 'kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-003@IIM.LOCAL; cat /var/lib/ambari-agent/tmp/atlas_hbase_setup.rb | hbase shell -n' returned 1. atlas_janus
ATLAS_ENTITY_AUDIT_EVENTS
atlas
TABLE
Took 8.2337 secondsjava exception ERROR Java::OrgApacheZookeeper::KeeperException::NoNodeException: KeeperErrorCode = NoNode for /hbase-secure/master
... View more
11-14-2018
04:31 PM
2018-11-14 16:15:03,009 WARN [master/hdata4:16000] master.ActiveMasterManager: Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null
2018-11-14 16:15:03,009 INFO [master/hdata4:16000] assignment.AssignmentManager: Stopping assignment manager
2018-11-14 16:15:03,032 WARN [master/hdata4:16000] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions.
2018-11-14 16:15:03,033 INFO [master/hdata4:16000] procedure2.RemoteProcedureDispatcher: Stopping procedure remote dispatcher
2018-11-14 16:15:03,033 INFO [master/hdata4:16000] procedure2.ProcedureExecutor: Stopping
2018-11-14 16:15:03,036 INFO [master/hdata4:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store, isAbort=false
2018-11-14 16:15:03,070 ERROR [master/hdata4:16000] wal.WALProcedureStore: Unable to close the stream
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Client (=DFSClient_NONMAPREDUCE_-826562846_1) is not the lease owner (=DFSClient_NONMAPREDUCE_-1195801889_1: /apps/hbase/data/MasterProcWALs/pv2-00000000000000000011.log (inode 10368947) Holder DFSClient_NONMAPREDUCE_-826562846_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2837)
at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:685)
at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:671)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2858)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:928)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:607)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
at org.apache.hadoop.ipc.Client.call(Client.java:1443)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy18.complete(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:550)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy19.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372)
at com.sun.proxy.$Proxy20.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372)
at com.sun.proxy.$Proxy20.complete(Unknown Source)
2018-11-14 16:15:03,071 INFO [master/hdata4:16000] hbase.ChoreService: Chore service for: master/hdata4:16000.splitLogManager. had [] on shutdown
2018-11-14 16:15:03,071 INFO [master/hdata4:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down.
2018-11-14 16:15:03,071 ERROR [master/hdata4:16000] access.TableAuthManager: Something wrong with the TableAuthManager reference counting: org.apache.hadoop.hbase.security.access.TableAuthManager@7e83992 whose count is null
Zookeeper znode listing for unsecure and secure. master znode is missing from hbase-secure [zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-unsecure
[replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, master, switch, running, draining, namespace, hbaseid, table]
[zk: hdata2.local:2181(CONNECTED) 0] ls /hbase-secure [replication, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, switch, running, tokenauth, draining, hbaseid, table]
... View more
11-12-2018
02:20 PM
After enablling kerberos HBase master failing to restart. On investigation found that in zookeeper /hbase-secure/master node is missing. Zookeeper shows following Acl [zk: localhost:2181(CONNECTED) 1] getAcl /hbase-secure 'world,'anyone
: r 'sasl,'hbase
: cdrwa 'sasl,'hbase
: cdrwa Tried all the threads eg. https://community.hortonworks.com/content/supportkb/151088/how-to-force-remove-znode-with-stale-acl.html https://community.hortonworks.com/articles/82405/how-to-remove-acl-protected-zk-node.html Any ideas?
... View more
Labels:
- Labels:
-
Apache HBase
11-08-2018
11:53 AM
Many Thanks Sandeep. Yes it is working now. I would like to add more details for fix: Update hive-interactive-env template and hive-env template Hive config properties from Ambari UI.
... View more
11-07-2018
03:15 PM
Hi Our Hive Server2 Interactive is stopping after around 10 min after restart. I couldn't find hiveserver2Interactive.log in /var/log/hive, hive-server2-interactive.err is showing following: /usr/hdp/current/hive-server2/conf_llap//hive-env.sh: line 43: [: !=: unary operator expected
2018-11-07 14:40:33: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.0.1.0-187/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = c4e8edc5-ed5e-4f96-ba70-00779ac20dc9
Hive Session ID = b5ad883d-4a65-45f7-a6dd-1ea3797724ea Any help will be appreciated. Thanks
... View more
Labels:
- Labels:
-
Apache Hive
11-06-2018
11:19 AM
I read about it (Data Analytics Studio) in some threads and want to know more about it (trial/download etc.), is it open source? Thanks
... View more
- Tags:
- analytics
Labels:
- Labels:
-
Data Analytics Studio
11-05-2018
04:11 PM
Since we did not have ACID tables, it worked without pre-upgrade tool.
... View more
11-05-2018
04:09 PM
After some investigation foend that KDC Administrator Credentials to the Ambari Credential Store was not properly set. Then set it from Ambari and it started to work
... View more
11-05-2018
11:27 AM
When upgrading to 3.0.1 from 2.6.5 from Ambari I am getting following error message An internal system exception occurred: Unable to build keytab regeneration stage Any idea? Where should I look for exception error? Tried restarting KDC but of no help.
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
11-04-2018
02:26 PM
The pre upgrade tool is reporting error requires Hive 2.x. Actual: 1.2.1000 I have Hive Interactive enabled and confirms to this thread https://community.hortonworks.com/questions/55051/no-hive-21-in-hdp-25.html I don't have ACID tables in Hive. Is it OK if I go ahead with version upgrade from Ambari without running pre-upgrade tool? Will it upgrade Hive? Kindly let me know
... View more
Labels:
10-24-2018
11:44 AM
Hi, I am in the process of upgrading hdp from 2.6 to 3.0.1, but got stuck with hive version 1.2. Getting following error from PreUpgradetool 18/10/23 17:55:19 ERROR acid.PreUpgradeTool: PreUpgradeTool failed
java.lang.IllegalStateException: preUpgrade requires Hive 2.x. Actual: 1.2.1000 In ambari both YARN pre-emption and Hive Interactive are greyed so unable to change them. My beeline command is as follows and it works fine and loads hive 1.2.1000 beeline -u "jdbc:hive2://data1.dev.local:10000/default;" How can I upgrade to hive 2.0+? At the moment I don't have ACID tables so do I still need to go through preUpgrade tool? will version upgrade not automatically upgrade from 1.2 to 3.0? Regards MB
... View more
Labels:
- Labels:
-
Apache Hive
08-16-2018
10:31 AM
Thanks for quick reply. I tried but still spark2 service principle is missing. I even checked it with kadmin. Wondering that Spark2 clients are running fine on the respective nodes, but issue is with Spark2 History Server and Spark2 Thrift Server. I can see in Spark2 config that spark.history.kerberos.keytab = /etc/security/keytabs/spark2.headless.keytab Looks like a bug somewhere!
... View more
08-15-2018
04:09 PM
Hi, Recently upgraded to HDP 2.7 and added Spark2 via Ambari on kerberised cluster. It is failing to start Spark2 history server because of missing Spark2 headless keytab file. What is the right way to resolve it.
... View more
Labels:
- Labels:
-
Apache Spark
07-26-2018
03:05 PM
@Sandeep Nemuri Thanks for swift reply. Is it OK if I also upgrade the underlying OS of Ambari host first followed by Ambari upgrade OR do the OS upgrade separately after Ambari is upgraded to 2.6.2.2?
... View more
07-26-2018
02:43 PM
Thanks Felix, almost there, although we have a small cluster but In-Place & Restore options suits us (set up with best practice in mind -:). Also a silly question: Where in Ambari UI is Ambari "Recover Host" functionality? We are using Ambari 2.5.0.3 Thanks MB
... View more
07-26-2018
12:30 PM
Hi, I have a 4 node dev cluster which was recently upgraded to 2.6.0.3. It is running on Ubuntu 14. I want to upgrade the underlying OS to 16.04.4. What are the right steps to achieve this? I am thinking on the following lines: 1. Pick one server at a time 2. Switch on maintenance mode and stop all services 3. run 'do-release-upgrade' command to do release upgrade 4. start all the services on the server Once all the servers are done, perform the version upgrade from Ambari UI for Ubuntu 16. Is this the right approach? Kindly let me know. Thanks in advance as usual. Regards
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
07-26-2018
11:23 AM
Following jar files should be present gmetric4j-1.0.3.jar jmxetric-1.0.4.jar oncrpc-1.0.7.jar Since I did the version upgrade from 2.5 to 2.6 I copied them from /usr/hdp/2.5.3.0-37/storm/contrib/storm-jmxetric/lib folder Hope this helps
... View more
06-07-2018
09:33 AM
Thanks Gaurav, it was helpful.
... View more
06-03-2018
12:56 PM
Hi, We have a development/test cluster with 4 nodes each having 12 cores vcpu, 2 TB SDD and 32GB RAM previously used for cassandra. All running on Ubuntu 14. It is configured as follows 1 NN 1 SNN 4 DN Jobs are taking longer due to limited resources, so we want to upgrade the HDP version from 2.5 to 2.6 and OS to Ubuntu 16.04 and add more resources. We are planning to upgrade as follows, 1. Increase RAM to 128 GB on all nodes 2. Double the storage by adding additional SAS disks 3. Add 2 more additional Data Nodes with 10 cores, 128 GB Questions I got is 1. Is it a good practice to use all of them as DN, or we should separate the NN 2. Since it was borrowed off Cassandra, is it ok if we keep all nodes of same specs including 2 new nodes 3. There is no harm in mixed storage options ie. adding extra storage in SAS wrt existing SSD 4. With the proposed new cluster, upgrading to HDP 3.0 shouldn't be a issue, I guess Thanks in advance MB
... View more
Labels:
- Labels:
-
Apache Hadoop
02-13-2018
12:25 PM
Thanks Eugene for swift reply. We are running on HDP 2.5 and MERGE is available in 2.6. What options do I have? Thanks MB
... View more
02-12-2018
05:32 PM
Hi, I have a clustered bucketted table, with ACID enabled Does hive supports updates from another table? This is something I am trying to achieve, updating a column values from another table UPDATE transaction_bak SET fl_t_order = tt.t_order FROM trans_temp tt WHERE transaction_bak.id = tt.id; I tried with left join, but no success, getting ParseException line 2:0 missing SET at 'left' UPDATE transaction_bak LEFT OUTER JOIN trans_temp tt SET fl_t_order = tt.t_order WHERE transaction_bak.id = tt.id; Any idea
... View more
Labels:
- Labels:
-
Apache Hive
02-07-2018
05:33 PM
Hi, I am getting this error when trying to insert into a hive table, the host in question (hdata2) is accessible and other services are running fine from it. klist shows valid kerberos ticket. I want to merge many tables into one, so was inserting executing external hql script. Any idea where to look for, logs looks fine. INFO : Tez session hasn't been created yet. Opening session
ERROR : Failed to execute tez graph.
java.lang.IllegalArgumentException: java.net.UnknownHostException: hdata2.xxxxxxxxxxxx.local
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:438)
at org.apache.hadoop.yarn.util.timeline.TimelineUtils.buildTimelineTokenService(TimelineUtils.java:120)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:165)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.tez.client.TezYarnClient.init(TezYarnClient.java:46)
at org.apache.tez.client.TezClient.start(TezClient.java:345)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:272)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:152)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1151)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:253)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: hdata2.xxxxxxxxxxxx.local
... 30 more
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1) Thanks MB
... View more
Labels:
- Labels:
-
Apache Hive
12-22-2017
03:58 PM
It is using keytab, is it possible that kdestroy on the host can interfere
... View more
12-22-2017
10:32 AM
Hi, My Spark Thrift Server goes down after the kerberos ticket time of 10 hrs. How to set up auto renewal of ticket. This is happening with Thrift server only. Regards MB
... View more
Labels:
- Labels:
-
Apache Spark
11-29-2017
10:49 AM
ok. found it here https://community.hortonworks.com/questions/100335/kerberos-install-on-hdp-cluster-after-rangerranger.html
... View more
11-29-2017
10:41 AM
Hi, I am about to kerberise the cluster from Ambari after disabling it. Ranger is also running, should I stop Ranger services before kerberising or decommission it and re-install Rangers after kerberos is installed? Thanks MB
... View more
Labels:
- Labels:
-
Apache Ranger
11-28-2017
01:38 PM
Also, I just noticed the PolicyRefresher exception in hadoop-hdfs-namenode-hmaster log Ranger services and plugins are disabled at the moment. Should I switch them on? 2017-11-28 13:27:51,752 ERROR util.PolicyRefresher (PolicyRefresher.java:loadPolicyfromPolicyAdmin(255)) - PolicyRefresher(serviceName=003_hadoop): failed t
o refresh policies. Will continue to use last known version of policies (2)
com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
... View more