Member since
05-09-2018
44
Posts
3
Kudos Received
0
Solutions
06-06-2021
05:24 PM
Hi All, We have Hue (version 4.2.0) in our prod environment and it is coupled with our LDAP. We manually onboard new users by going to "Manage Users" -> "Add/Sync Manage Users". Here, we add the LDAP ID and later when the user gets added. Once the user is added, we again search the newly added LDAP user, click on that and go to "Profiles and Group". Here, we select "temporary-access" so that User can able to access the UI. Without Temporary-access selection, user will not be able to see the UI. We want to get this automated by writing a Py program and we are looking for the right REST API for adding users to Hue and give him/her "temp-access" in profiles and group. The following link wasn't helpful - http://cloudera.github.io/hue/latest/developer/api/ Please assist us. I would highly appreciate any form of assistance or help. Regards, Shesh Kumar
... View more
Labels:
05-27-2019
05:40 PM
@Geoffrey Shelton Okot : Thank you for the detailed explanation.
... View more
05-27-2019
05:24 PM
@Geoffrey Shelton Okot I do not see that option 😞
... View more
05-21-2019
08:19 PM
Hi, As per my knowledge (please correct me if I am wrong), the Datanodes sends the block report to both Active and Standby Namenodes. The job of Active NN is to write to the Journal Nodes and the job of Standby namenode is to read from Journal nodes. Now why does Standby namenode need to read from Journal nodes when the Datanodes (slaves) are already sending the block reports to it?
... View more
Labels:
- Labels:
-
Apache Hadoop
05-08-2019
05:38 AM
My cluster was hung. Was unable to add hosts or perform any basic activities in Ambari like restart of a service. Was constantly seeing the WARN snippet in Ambari Server logs: Unable to lookup the cluster by ID; assuming that there is no cluster and therefore no configs for this execution command: Cluster not found, clusterName=clusterID=-1 Here's a small hack to resolve the issue: 1. Check the cluster id in your backend Ambari DB. Mine is MySQL. select * from clusterstate; 2. The same value found in step 1 should be there in Stage table's "cluster_id" columns select stage_id, request_id, cluster_id from stage; 3. If there are values as -1 please update it to the correct value found in step 1. Example: UPDATE stage SET cluster_id='2' WHERE request_id IN (383,384,388,389); 4. Restart Ambari-Server ambari-server restart 5. Post this check by restarting any service like Grafana or any small service which not does impact the Hadoop service. If it proceeds, the cluster is now stable and you will be able to add nodes. 6. If issue persists, the perform the following in your backend Ambari DB. SELECT * FROM host_role_command WHERE status='PENDING'; 7. If you get any output, you need to update the status to "ABORTED". UPDATE host_role_command SET status='ABORTED' WHERE status='PENDING'; 8. Restart Ambari-Server ambari-server restart Validate the health of Ambari by restarting Grafana or any small service which not does impact the Hadoop service. If everything is good, proceed by adding the nodes.
... View more
Labels:
03-02-2019
12:44 PM
Hi, No need to do any deletion. Just follow the steps here to resolve the issue without any data loss: https://community.hortonworks.com/questions/242343/hbase-table-is-stuck-in-disabling-state-neither-en.html Thanks, Shesh
... View more
03-02-2019
12:35 PM
1 Kudo
This is just a knowledge sharing article. I had faced this issue in production and took me a day to resolve it. The workaround that I'm sharing will help you in getting your table back online in "Enabled" without deleting Zookeeper Hbase table znode or any data as a matter of fact.. Here are steps to resolve it. 1. Run a "get" command against hbase:meta for the affected table hbase(main):003:0> get 'hbase:meta', '<AFFECTED_TABLE_NAME>', 'table:state' COLUMN CELL table:state timestamp=1551456805377, value=\x08\x02 2. Notice the above "value". Its pointing to \x08\x02 which is wrong. The value should either be \x08\x00 (Enabled) or \x08\x01 (Disabled) 3. Edit the value manually. hbase(main):003:0> put 'hbase:meta','<AFFECTED_TABLE_NAME>','table:state',"\b\0" Click here for more information on "Control Characters" 4. Verify the same. hbase(main):003:0> get 'hbase:meta', '<AFFECTED_TABLE_NAME>', 'table:state' The "value" now should be \x08\x00 Post this, again run disable <table_name> and enable <table_name> just for the love of sanity check in hbase shell and you are done with the issue.
... View more
Labels:
12-18-2018
12:53 PM
I'm getting the following error after following "Option #2". My Ranger is not TLS/SSL enabled. Please help 18 Dec 2018 12:17:23 INFO LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Using principal = rangerusersync/stg-agent001-stg-cloud009.XXXXX.nm2@XXXXXX.COM and keytab = /etc/security/keytabs/rangerusersync.service.keytab 18 Dec 2018 12:17:24 ERROR LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to build Group List : com.sun.jersey.api.client.UniformInterfaceException: POST http://stg-agent001-stg-cloud009.xxxxx.nm2:6080/service/xusers/groups/ returned a response status of 404 Not Found and 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - LdapDeltaUserGroupBuilder.getUsers() completed with user count: 0 18 Dec 2018 12:17:24 ERROR LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.sun.jersey.api.client.UniformInterfaceException: POST http://stg-agent001-stg-cloud009.xxxxx.nm2:6080/service/xusers/ugsync/auditinfo/ returned a response status of 404 Not Found
... View more
12-12-2018
01:01 PM
Hi, I'm seeing this error in RS logs. Can someone help me resolve the issue? 2018-12-12 18:28:54,407 INFO [B.defaultRpcServer.handler=12,queue=0,port=16020] shortcircuit.ShortCircuitCache: ShortCircuitCache(0x3d630f8a): could not load 1093156401_BP-853897652-10.84.192.246-1489729943941 due to InvalidToken exception.
org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /apps/hbase/data/data/default/cyclops-edges/865c0549943a300f09f5dfcd63fbaa67/s/1979e37ca9294d24bd16cd580fb663ab
2018-12-12 18:09:57,326 INFO [sync.2] wal.FSHLog: Slow sync cost: 105 ms, current pipeline: [DatanodeInfoWithStorage[10.84.197.254:50010,DS-159866b7-1b97-469f-9a22-4b03b3dbbe56,DISK], DatanodeInfoWithStorage[10.84.192.255:50010,DS-26463076-ed2d-4883-9013-2870ce87f281,DISK], DatanodeInfoWithStorage[10.84.192.76:50010,DS-b1b894fc-d1e2-4ccf-8cc7-7ccd649cd507,DISK]]
2018-12-12 18:09:57,391 WARN [B.defaultRpcServer.handler=21,queue=0,port=16020] hdfs.BlockReaderFactory: I/O error constructing remote block reader.
java.io.IOException: Got error, status message opReadBlock BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530, for OP_READ_BLOCK, self=/10.84.197.254:13978, remote=/10.84.192.246:50010, for file /apps/hbase/data/data/default/cyclops-audits-dedup/e44738e4889089bcecf58b66878a7501/l/457245b1ea724f7f80bab245ea6c0604, for pool BP-853897652-10.84.192.246-1489729943941 block 1093248459_19508530
at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140)
at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:456)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:424)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:816)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:695)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1181)
at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1118)
at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1478)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1441)
at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
at org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1625)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:441)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:642) at java.lang.Thread.run(Thread.java:745)
2018-12-12 18:09:57,391 WARN [B.defaultRpcServer.handler=21,queue=0,port=16020] hdfs.DFSClient: Connection failure: Failed to connect to /10.84.192.246:50010 for file /apps/hbase/data/data/default/cyclops-audits-dedup/e44738e4889089bcecf58b66878a7501/l/457245b1ea724f7f80bab245ea6c0604 for block BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530:java.io.IOException: Got error, status message opReadBlock BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530, for OP_READ_BLOCK, self=/10.84.197.254:13978, remote=/10.84.192.246:50010, for file /apps/hbase/data/data/default/cyclops-audits-dedup/e44738e4889089bcecf58b66878a7501/l/457245b1ea724f7f80bab245ea6c0604, for pool BP-853897652-10.84.192.246-1489729943941 block 1093248459_19508530
java.io.IOException: Got error, status message opReadBlock BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530, for OP_READ_BLOCK, self=/10.84.197.254:13978, remote=/10.84.192.246:50010, for file /apps/hbase/data/data/default/cyclops-audits-dedup/e44738e4889089bcecf58b66878a7501/l/457245b1ea724f7f80bab245ea6c0604, for pool BP-853897652-10.84.192.246-1489729943941 block 1093248459_19508530
at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140) The issue that we are facing is very slow Hbase compaction. HDP Version: HDP-2.4.3.0-227 HBase: 1.1.2.2.4 Thanks, Shesh
... View more
Labels: