Member since
05-29-2018
5
Posts
2
Kudos Received
0
Solutions
07-11-2018
03:06 PM
https://kylo.io they have a sandbox of Kylo on HDP. https://kylo.io/quickstart.html It was designed to solve the business user use case and the central job monitoring use case. The sandbox makes it pretty easy to try out.
... View more
07-11-2018
02:59 PM
For Kafka, swap space is probably safe to clear (though I wouldn't), but you should avoid Kafka using swap space. If you look at disk IO on a Kafka broker node, it should be almost all writes, read should come from page cache. Kafka was designed to be the only tenant on a node and runs best that way. This is why you will find recommendations that say Kafka should not share nodes with Zookeeper or other Hadoop components. It is not always possible to dedicate machines to Kafka, so take a look at the disk IO when Kafka is running under normal load, if it is all writes, you can probably shrink the page cache a bit so you do less/no swapping. If there are lots of reads, you may need more memory or more nodes (unless you are deliberately and routinely reading topics from the beginning, in which case disk reads are unavoidable). Can't help you with the zookeeper, I've never had reason to dig into zookeeper's internals, it has always just worked.
... View more
07-10-2018
06:24 PM
1 Kudo
I am unable to produce message to kafka. Using the HDP Sandbox 2.6.4 for virtual box (on mac). Imported fresh VM from .ova. Started fresh machine. Logged into Ambari as raj_ops. Started Kafka and took it out of Maintenance Mode. Started web shell http://localhost:4200 Created a topic Started a the consumer [root@sandbox-hdp ~]# cd /usr/hdp/current/kafka-broker
[root@sandbox-hdp kafka-broker]# bin/kafka-topics.sh --list --zookeeper localhost:2181
ATLAS_ENTITIES
ATLAS_HOOK
__consumer_offsets
[root@sandbox-hdp kafka-broker]# bin/kafka-topics.sh --zookeeper localhost:2181 --topic simpletesttopic --partitions 1 --replication-factor 1 --create
Created topic "simpletesttopic".
[root@sandbox-hdp kafka-broker]# bin/kafka-topics.sh --list --zookeeper localhost:2181
ATLAS_ENTITIES
ATLAS_HOOK
__consumer_offsets
simpletesttopic
[root@sandbox-hdp kafka-broker]# bin/kafka-console-consumer.sh --bootstrap-server localhost:6667 --from-beginning --topic simpletesttopic
Then I opened a new web shell and used the console produce to attempt to send a message. [root@sandbox-hdp ~]# cd /usr/hdp/current/kafka-broker
[root@sandbox-hdp kafka-broker]# bin/kafka-console-producer.sh --topic simpletesttopic --broker-list localhost:6667
A Test Message
[2018-07-10 17:31:47,435] ERROR Error when sending message to topic simpletesttopic with key: null, value: 14 bytes with error: (org.apache.kaf
ka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
So, why doesn't this work? I didn't change anything in the sandbox besides starting Kafka. Someone will come on and tell me to check the logs, OK I'll do that... there is nothing after the topic creation. Here is the last 11 lines of the controller.log and last 4 lines of the server.log. [root@sandbox-hdp ~]# cd /usr/hdp/current/kafka-broker/logs
[root@sandbox-hdp logs]# tail -11 controller.log
[2018-07-10 17:29:25,086] DEBUG [TopicChangeListener on Controller 1001]: Topic change listener fired for path /brokers/topics with children ATLAS_HOOK,simpletesttopic,ATLAS_ENTITIES,__consumer_offsets (kafka.controller.PartitionStateMachine$TopicChangeListener)
[2018-07-10 17:29:25,089] INFO [TopicChangeListener on Controller 1001]: New topics: [Set(simpletesttopic)], deleted topics: [Set()], new partition replica assignment [Map([simpletesttopic,0] -> List(1001))] (kafka.controller.PartitionStateMachine$TopicChangeListener)
[2018-07-10 17:29:25,089] INFO [Controller 1001]: New topic creation callback for [simpletesttopic,0] (kafka.controller.KafkaController)
[2018-07-10 17:29:25,092] INFO [Controller 1001]: New partition creation callback for [simpletesttopic,0] (kafka.controller.KafkaController)
[2018-07-10 17:29:25,092] INFO [Partition state machine on Controller 1001]: Invoking state change to NewPartition for partitions [simpletesttopic,0] (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,096] INFO [Replica state machine on controller 1001]: Invoking state change to NewReplica for replicas [Topic=simpletesttopic,Partition=0,Replica=1001] (kafka.controller.ReplicaStateMachine)
[2018-07-10 17:29:25,098] INFO [Partition state machine on Controller 1001]: Invoking state change to OnlinePartition for partitions [simpletesttopic,0] (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,100] DEBUG [Partition state machine on Controller 1001]: Live assigned replicas for partition [simpletesttopic,0] are: [List(1001)] (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,102] DEBUG [Partition state machine on Controller 1001]: Initializing leader and isr for partition [simpletesttopic,0] to(Leader:1001,ISR:1001,LeaderEpoch:0,ControllerEpoch:2) (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,116] INFO [Replica state machine on controller 1001]: Invoking state change to OnlineReplica for replicas [Topic=simpletesttopic,Partition=0,Replica=1001] (kafka.controller.ReplicaStateMachine)
[2018-07-10 17:29:25,118] INFO [Controller-1001-to-broker-1001-send-thread], Controller 1001 connected to sandbox-hdp.hortonworks.com:6667 (id:1001 rack: null) for sending state change requests (kafka.controller.RequestSendThread)
[root@sandbox-hdp logs]# tail -4 server.log
[2018-07-10 17:29:25,121] INFO [ReplicaFetcherManager on broker 1001] Removed fetcher for partitions simpletesttopic-0 (kafka.server.ReplicaFetcherManager)
[2018-07-10 17:29:25,123] INFO Completed load of log simpletesttopic-0 with 1 log segments and log end offset 0 in 0 ms (kafka.log.Log)
[2018-07-10 17:29:25,124] INFO Created log for partition [simpletesttopic,0] in /kafka-logs with properties {compression.type -> producer, message.format.version -> 0.10.1-IV2, file.delete.delay.ms -> 60000, max.message.bytes -> 1000000, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> true, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
[2018-07-10 17:29:25,125] INFO Partition [simpletesttopic,0] on broker 1001: No checkpointed highwatermark is found for partition [simpletesttopic,0] (kafka.cluster.Partition)
... View more
Labels:
05-31-2018
02:19 PM
I think this solution would work from a technical perspective. However, I am not sure my users will accept the naming pattern limitation. I have been unable to find a example of anyone using the {OWNER} variable. @svenkat said how useful it was here https://community.hortonworks.com/questions/118123/hdp-26-allows-user-variable-in-ranger-policies-are.html but didn't provide an example of actually using it. Does it work?
... View more
05-30-2018
08:40 PM
1 Kudo
I have a database where all members of a group are allowed to create tables. When a user creates a table, only the creator should be allowed access to that table. I will create a new policy to allow other members of the group access to the new table after an approval business process. HDP 2.6.5, Hive and Ranger (I cannot use Atlas and tag based policies at this time) This policy allows members of the group to create tables, but the owner has no access after creating. resources(database:testdb, table:*, column:*)
policyItems (accesses:create,groups:testgroup) beeline as user1, a member of testgroup 0: jdbc:hive2://localhost:10000> use testdb;
No rows affected (2.509 seconds)
0: jdbc:hive2://localhost:10000> CREATE TABLE testtable (id STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/hive/data/testtable';
No rows affected (0.669 seconds)
0: jdbc:hive2://localhost:10000> show tables;
+------------+--+
| tab_name |
+------------+--+
| testtable |
+------------+--+
1 row selected (1.029 seconds)
0: jdbc:hive2://localhost:10000> describe testtable;
Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [user1] does not have [SELECT] privilege on[testdb/testtable/*] (state=42000,code=40000) I tried adding a policy item using the {OWNER} variable but had the same result. resources(database:testdb, table:*, column:*)
policyItems [(accesses:[select,update,drop,alter,index,lock,read,write] users:{OWNER}), (accesses:create, groups:testgroup)]
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Ranger
