Member since
09-29-2015
186
Posts
63
Kudos Received
12
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3183 | 08-11-2017 05:27 PM | |
2144 | 06-27-2017 10:58 PM | |
2256 | 04-09-2017 09:43 PM | |
3214 | 04-01-2017 02:04 AM | |
4411 | 03-13-2017 06:35 PM |
03-19-2021
08:12 AM
@isoardi oh this is a cluster that I can use to test upgrades etc. But I did install HDP 2.6.x onto this cluster which is similar to our production environment.
... View more
03-18-2021
09:51 PM
@balo Please refer to "Generating download credentials": https://docs.cloudera.com/csa/1.2.0/download/topics/csa-download-cred.html
... View more
05-11-2020
05:52 PM
1 Kudo
To set up health_percent of LLAP, do the following:
On the Hiveserver2Interactive server nodes, edit /usr/hdp/<hdp-version>/hive/scripts/llap/yarn/package.py
Example: /usr/hdp/3.1.0.224-3/hive/scripts/llap/yarn/package.py The --health-percent defaults to 80. Change this to desired-number.
Move the following to a temporary backup location: a. /usr/hdp/<hdp-version>/hive/scripts/llap/yarn/package.pyc
b. /usr/hdp/<hdp-version>/hive/scripts/llap/yarn/package.pyo
Restart Hive.
... View more
Labels:
11-19-2018
10:11 PM
1 Kudo
This article just gives an example of how 'grant'/'revoke' works when the Hive plugin is enabled with Ranger in CDP.
A user who is 'admin' in Ranger, can manage access to Hive tables via 'grant'/'revoke' operation.
In Ranger UI > Settings > Users and Groups > Users
Note: User 'hive' is in role 'Admin'
On the beeline, login as user 'hive'. Run the grant command to give select privileges on a table:
0: jdbc:hive2://a.b.c.co> grant select on table mix to user mugdha;
INFO : Compiling command(queryId=hive_20211021024819_c3de84a7-a312-4a1f-9a8d-8b328cced054): grant select on table mix to user mugdha
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling command(queryId=hive_20211021024819_c3de84a7-a312-4a1f-9a8d-8b328cced054); Time taken: 0.022 seconds
INFO : Executing command(queryId=hive_20211021024819_c3de84a7-a312-4a1f-9a8d-8b328cced054): grant select on table mix to user mugdha
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20211021024819_c3de84a7-a312-4a1f-9a8d-8b328cced054); Time taken: 0.548 seconds
INFO : OK
No rows affected (0.634 seconds)
In Ranger, a new policy is created by that command:
Similarly, in a 'revoke' run, user 'mugdha', will be removed from the policy:
0: jdbc:hive2://a.b.c.co> revoke select on table mix from user mugdha;
INFO : Compiling command(queryId=hive_20211021025423_cdf81a8a-df0d-4c40-9509-f4325d3ba112): revoke select on table mix from user mugdha
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling command(queryId=hive_20211021025423_cdf81a8a-df0d-4c40-9509-f4325d3ba112); Time taken: 0.032 seconds
INFO : Executing command(queryId=hive_20211021025423_cdf81a8a-df0d-4c40-9509-f4325d3ba112): revoke select on table mix from user mugdha
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20211021025423_cdf81a8a-df0d-4c40-9509-f4325d3ba112); Time taken: 0.274 seconds
INFO : OK
No rows affected (0.323 seconds)
This also works the same way in HDP, see Provide User Access to Hive Database Tables from the Command Line
... View more
08-17-2017
12:12 AM
@tsharma In my cluster I do see that after enabling Kerberos yarn.acl.enable was set to true. I am not sure though..
... View more
06-30-2017
11:49 PM
PROBLEM: Click Alerts and then Actions > Manage Alert Group -> Custom Alert Group. Then, click the + sign on the right side and pick any alert definition and press OK.
Click Save and you will see the 500(server error) Server Error on the alert group screen: And in the ambari-server.log there is error: WARN [qtp-ambari-client-510524] ServletHandler:563 - /api/v1/clusters/<cluster-name>/alert_groups/155
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
at java.util.HashMap$KeyIterator.next(HashMap.java:1453)
at org.eclipse.persistence.indirection.IndirectSet$1.next(IndirectSet.java:471)
at org.apache.ambari.server.orm.entities.AlertGroupEntity.setAlertTargets(AlertGroupEntity.java:313)
at org.apache.ambari.server.controller.internal.AlertGroupResourceProvider.updateAlertGroups(AlertGroupResourceProvider.java:344)
at org.apache.ambari.server.controller.internal.AlertGroupResourceProvider.access$100(AlertGroupResourceProvider.java:60)
at org.apache.ambari.server.controller.internal.AlertGroupResourceProvider$2.invoke(AlertGroupResourceProvider.java:187)
at org.apache.ambari.server.controller.internal.AlertGroupResourceProvider$2.invoke(AlertGroupResourceProvider.java:184)
at org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:450)
at org.apache.ambari.server.controller.internal.AbstractResourceProvider.modifyResources(AbstractResourceProvider.java:331)
ROOT CAUSE: https://issues.apache.org/jira/browse/AMBARI-19259 RESOLUTION: Upgrade Ambari to 2.5
... View more
Labels:
06-30-2017
11:48 PM
Consider the example: Total input paths = 7
Input size for job = 510K 1) While are using a custom InputFormat which extends ‘org.apache.hadoop.mapred.FileInputFormat’ and having ‘isSplitable’ as false. Expected : 7 splits [As FileInputFormat doesn't split file smaller than blockSize (128 MB) so there should be one split per file]
Actual: 4 splits 2) Default value for 'hive.input.format' is CombineHiveInputFormat.
After setting ‘set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;’, there are 7 splits as expected. From above two points, it looks hive uses ‘CombineHiveInputFormat’ on top of the custom InputFormat to determine number of splits. How splits were calculated: For deciding the number of mappers when using CombineInputFormat, data locality plays a role. Now to find where those files belong we can get it from command:
hadoop fsck /<file-path> -files -blocks -locations 1. On. a.a.a.a /user/user1/hive/split/file1_0000
[/default-rack/a.a.a.a:1019, /default-rack/e.e.e.e:1019]
/user/user1/hive/split/file1_0002
[/default-rack/a.a.a.a:1019, /default-rack/e.e.e.e:1019]
2. On b.b.b.b /user/user1/hive/split/file1_0003
[/default-rack/b.b.b.b:1019, /default-rack/a.a.a.a:1019]
/user/user1/hive/split/file1_0005
[/default-rack/b.b.b.b:1019, /default-rack/a.a.a.a:1019]
/user/user1/hive/split/file1_0006
[/default-rack/b.b.b.b:1019, /default-rack/e.e.e.e:1019]
3. On c.c.c.c /user/user1/hive/split/file1_0001
[/default-rack/c.c.c.c:1019, /default-rack/a.a.a.a:1019]
4. On d.d.d.d /user/user1/hive/split/file1_0004
[/default-rack/d.d.d.d:1019, /default-rack/a.a.a.a:1019]
Hive is picking up blocks from these 4 DNs. Files on 1 DN are combined into 1 task. If a maxSplitSize is specified, then blocks on the same node are combined to form a single split. Blocks that are left over are then combined with other blocks in the same rack. If maxSplitSize is not specified, then blocks from the same rack are combined in a single split; no attempt is made to create node-local splits. If the maxSplitSize is equal to the block size, then this class is similar to the default splitting behavior in Hadoop: each block is a locally processed split.
Ref: https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/lib/CombineFileInputFormat.html The reason it has picked the first block location for each blocks while combining is any Hadoop Client will use the first block location and will consider the next only if reading the first fails. Usually NameNode will return the block locations of a block sorted based upon the distance between the client and location. NameNode will give all block locations but CombineHiveInputFormat / Hadoop Client / MapReduce Program uses the first block location.
... View more
Labels:
06-30-2017
11:46 PM
PROBLEM: After moving the Zookeeper servers and setting correctly in the yarn configs, Resource managers come up but are in standby state. Even after removing the znode - rmstore, none of the nodes transition to active. ROOT CAUSE: Zookeeper data is stored in znode yarn-leader-election. This is used for RM leader election. This znode has stale data about zookeeper leader. RESOLUTION: 1. Login into zkcli 2. rmr /yarn-leader-election 3. Restart Resource managers.
... View more
06-30-2017
11:30 PM
6 Kudos
PROBLEM: When the hostname is mixed like: 172.26.93.148 GRAFANA-hdp253-s1.openstacklocal GRAFANA-hdp253-s1
172.26.93.149 GRAFANA-hdp253-s2.openstacklocal GRAFANA-hdp253-s2
172.26.93.150 GRAFANA-hdp253-s3.openstacklocal GRAFANA-hdp253-s3 Ambari creates the datasource with a lower case hostname: Once you enter the hostname with mixed case, Grafana works: RESOLUTION Grafana 2.6.0 backend uses Go 1.5.
Go's DNS lookup had a bug where the look up is case sensitive: https://github.com/golang/go/issues/12806 We can manual workaround by using the original casing in the Grafana Data Source URL as mentioned in the description. Another workaround is to update the /etc/hosts file to have all lowercase patterns as well.
This bug was fixed in Go 1.6, so we will no longer have this issue once we upgrade Grafana to a later version (say 4.1.x) in a future version of Ambari.
... View more
Labels: