Member since
08-29-2016
30
Posts
15
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1622 | 03-31-2017 05:31 AM | |
18395 | 03-31-2017 05:14 AM |
08-03-2018
12:19 PM
5 Kudos
In kafka there are three types of communication : 1. Between brokers 2. Between client and broker. 3. Broker to zookeeper In order to communicate in kerberos enabled cluster one needs to authenticate itself. So when broker will try to communicate with other broker in the cluster it will need to authenticate first. Same for the clients communicating to the brokers. In Kafka, JAAS files is used for authentication. Lets first understand what's there in JAAS file. In kafka there are two JAAS files : 1. kafka_jaas.conf
2. kafka_client_jaas.conf.
Let's discuss about kafka_jaas.conf. This file will be used for authentication when a broker in a cluster tries to communicate with other brokers in cluster. Take a look at the content and understand its purpose : KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="kafka/c6401.ambari.apache.org@EXAMPLE.COM";
}; KafkaServer section will be used by broker for authentication when it tries to communicate with other brokers in cluster. It should always be configured to use keytab and principal. Value of `serviceName` should be the principal as which kafka is running. `storeKey=true` : significance of setting the storekey parameter to true in jaas.conf Client { // used for zookeeper connection
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
storeKey=true
useTicketCache=false
serviceName="zookeeper"
principal="kafka/c6401.ambari.apache.org@EXAMPLE.COM";
}; Client section in kafka_jaas.conf will be used for authentication when broker wants to communicate with zookeeper. It should always be configured to use keytab and principal. Value of `serviceName` should be the principal as which zookeeper service is running.
kafka_client_jaas.conf will be used by clients(producer/consumer) to authenticate to kafka broker. kafka_client_jaas.conf has two sections, take a look : KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="kafka";
}; Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="zookeeper";
}; KafkaClient section : As the name suggest it will be used when client wants to communicate to broker.
Value of `serviceName` should be the principal as which kafka is running. You can configure it to use ticket cache or keytab and principal.
Client : This part of JAAS file is only used by clients which are using old consumer api. In old consumer api, consumers need to connect to zookeeper.
In new consumer api clients communicate to brokers instead of zookeeper. Hence while authentication it will use KafkaClient section in kafka_client_jaas.conf.
Producers will always use KafkaClient section in kafka_client_jaas.conf as it will send request to broker node.
For long running kafka clients it recommended to configure JAAS file to use keytab and principal. Please refer below example : KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/storm.service.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="storm@EXAMPLE.COM";
}; When kerberos is integrated with kafka we see lot of issues while trying to produce/consume messages to kafka. There are instances where client throws a generic error and we don't know what's going wrong. To tackle such conditions I will discuss about the checks which need to be done when you face such issue :
1. make sure kerberos client is installed on the node.
2. Check if you can obtain a ticket of the principal :
- If you want to obtain TGT using password : # kinit <principalName> If you are using keytab check if user has permission to read the keytab :
- Using keytab : # kinit -kt /Path/to/Keytab <PrincipalName> 3. Confirm if you a ticket in ticket cache which is not expired : # klist 4. Confirm if user has read permission on JAAS file which is used. 5. Confirm if client can communicate to kafka broker and port on which broker is listening : # ping <broker.hostname>
# telnet broker.hostname:port 6. Check if you are using correct security protocol `--security-protocol` as configured in server.properties in broker. 7. Try exporting JAAS file and run producer/consumer again : # export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/Path/to/Jaas/File" In order to enable debug for kerberos : # export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/Path/to/Jaas/File -Dsun.security.krb5.debug=true" 8. Still if you are facing authentication issue, try enabling debug for console-producer/console-consumer on kafka client node :
As root user : # vim /usr/hdp/current/kafka-broker/config/tools-log4j.properties log4j.rootLogger=DEBUG, stderr In debug logs you should see which principal, security protocol is used and to which broker request is being send. 9. Once you are confirmed that authentication is working fine it time for you to confirm if user has the required permission on the topic.
If kafka is configured to use kafka acl's, please refer below link : Authorization commands If kafka is configured to use ranger, make sure policy is defined for the topic and principal
... View more
- Find more articles tagged with:
- consumekafka
- How-ToTutorial
- Kafka
- Kerberos
- producer
- Security
Labels:
10-11-2017
10:08 AM
@Tech Gig Please refer https://community.hortonworks.com/questions/79093/problem-of-creating-topics-in-kafka-with-kerberos.html
... View more
09-13-2017
02:56 PM
1 Kudo
1.You
can execute below api to get a list of hosts in your cluster in file
hostcluster.txt # curl -s -u admin:admin
http://ambari:8080/api/v1/hosts|grep host_name|sed -n
's/.*"host_name" :
"\([^\"]*\)".*/\1/p'>hostcluster.txt 2. In the loop you can
write the api which need to run on the nodes : ~~~ while read line ; do j=$line mkdir -p $j done < hostcluster.txt ~~~
-admin:admin : username:password -
Above loop will take each entries from file hostcluster.txt and need to
execute the. 3. In order to install the clients you can use below
API’s. Below API’s will install only 5
clients as mentioned below : ####### Installing HDFS_CLIENT, YARN_CLIENT,
ZOOKEEPER_CLIENT and MAPREDUCE2_CLIENT on "HOSTANAME" as following: +++++++++++ # curl -u admin:admin -H "X-Requested-By:ambari"
-i -X POST -d '{"RequestInfo":{"context":"Install HDFS
Client"},"Body":{"host_components":[{"HostRoles":{"component_name":"HDFS_CLIENT"}}]}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts?Hosts/host_name=$j # curl -u admin:admin -H "X-Requested-By:ambari"
-i -X POST -d '{"RequestInfo":{"context":"Install YARN
Client"},"Body":{"host_components":[{"HostRoles":{"component_name":"YARN_CLIENT"}}]}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts?Hosts/host_name=$j # curl -u admin:admin -H "X-Requested-By:ambari"
-i -X POST -d '{"RequestInfo":{"context":"Install
MapReduce2
Client"},"Body":{"host_components":[{"HostRoles":{"component_name":"MAPREDUCE2_CLIENT"}}]}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts?Hosts/host_name=$j # curl -u admin:admin -H "X-Requested-By:ambari"
-i -X POST -d '{"RequestInfo":{"context":"Install
ZooKeeper
Client"},"Body":{"host_components":[{"HostRoles":{"component_name":"ZOOKEEPER_CLIENT"}}]}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts?Hosts/host_name=$j +++++++++++ Where,
-admin:admin : Is username and password for
ambary server. -ambari-hostname : hostname of your ambari server
-$j : is
the variable which will substitute each value from hostcluster.txt NOTE : If you want to add more clients such as spark/oozie etc you
need to change below value from above command : -"context":"Install ZooKeeper
Client" <-- Modify as per the client - component_name":"MAPREDUCE2_CLIENT" <-- Modify as per the client you want to
install 3 Below API’s is to pull the configurations for all the
clients which are installed in step 2 : . ####### Initialize the HDFS_CLIENT, YARN_CLIENT, ZOOKEEPER_CLIENT
and MAPREDUCE2_CLIENT clients on $j +++++++++++ curl -u admin:admin -H "X-Requested-By:ambari" -i
-X PUT -d '{"RequestInfo":{"context":"Install HDFS
Client","operation_level":{"level":"HOST_COMPONENT","cluster_name":"rest","host_name":"$j","service_name":"HDFS"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts/$j/host_components/HDFS_CLIENT?HostRoles/state=INIT curl -u admin:admin -H "X-Requested-By:ambari" -i
-X PUT -d '{"RequestInfo":{"context":"Install YARN
Client","operation_level":{"level":"HOST_COMPONENT","cluster_name":"rest","host_name":"$j","service_name":"YARN"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts/$j/host_components/YARN_CLIENT?HostRoles/state=INIT curl -u admin:admin -H "X-Requested-By:ambari" -i
-X PUT -d '{"RequestInfo":{"context":"Install
MapReduce2
Client","operation_level":{"level":"HOST_COMPONENT","cluster_name":"rest","host_name":"$j","service_name":"MAPREDUCE2"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts/$j/host_components/MAPREDUCE2_CLIENT?HostRoles/state=INIT # curl -u admin:admin -H "X-Requested-By:ambari"
-i -X PUT -d '{"RequestInfo":{"context":"Install
ZooKeeper
Client","operation_level":{"level":"HOST_COMPONENT","cluster_name":"rest","host_name":"$j","service_name":"ZOOKEEPER"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts/$j/host_components/ZOOKEEPER_CLIENT?HostRoles/state=INIT +++++++++++ Where,
-ambari-hostname : hostname of your ambari server
-$j : is
the variable which will take each value from hostcluster.txt If you have added more clients in step 2 then you need to add more
commands in step 3 based on the clients installed in step 2. Below is the scrip to
install : HDFS_CLIENT, YARN_CLIENT, ZOOKEEPER_CLIENT and MAPREDUCE2_CLIENT
1.Create
a .sh file and copy below contents: # vi script.sh ~~~~ curl -s -u admin:admin http://ambari:8080/api/v1/hosts|grep
host_name|sed -n 's/.*"host_name" :
"\([^\"]*\)".*/\1/p'>hostcluster.txt while read line ; do j=$line mkdir -p $j curl -u admin:admin -H "X-Requested-By:ambari" -i
-X POST -d '{"RequestInfo":{"context":"Install HDFS
Client"},"Body":{"host_components":[{"HostRoles":{"component_name":"HDFS_CLIENT"}}]}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts?Hosts/host_name=$j curl -u admin:admin -H "X-Requested-By:ambari" -i
-X POST -d '{"RequestInfo":{"context":"Install YARN
Client"},"Body":{"host_components":[{"HostRoles":{"component_name":"YARN_CLIENT"}}]}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts?Hosts/host_name=$j curl -u admin:admin -H "X-Requested-By:ambari" -i
-X POST -d '{"RequestInfo":{"context":"Install
MapReduce2
Client"},"Body":{"host_components":[{"HostRoles":{"component_name":"MAPREDUCE2_CLIENT"}}]}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts?Hosts/host_name=$j curl -u admin:admin -H "X-Requested-By:ambari" -i
-X POST -d '{"RequestInfo":{"context":"Install
ZooKeeper
Client"},"Body":{"host_components":[{"HostRoles":{"component_name":"ZOOKEEPER_CLIENT"}}]}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts?Hosts/host_name=$j curl -u admin:admin -H "X-Requested-By:ambari" -i
-X PUT -d '{"RequestInfo":{"context":"Install HDFS
Client","operation_level":{"level":"HOST_COMPONENT","cluster_name":"rest","host_name":"$j","service_name":"HDFS"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts/$j/host_components/HDFS_CLIENT?HostRoles/state=INIT curl -u admin:admin -H "X-Requested-By:ambari" -i
-X PUT -d '{"RequestInfo":{"context":"Install YARN
Client","operation_level":{"level":"HOST_COMPONENT","cluster_name":"rest","host_name":"$j","service_name":"YARN"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts/$j/host_components/YARN_CLIENT?HostRoles/state=INIT curl -u admin:admin -H "X-Requested-By:ambari" -i
-X PUT -d '{"RequestInfo":{"context":"Install
MapReduce2
Client","operation_level":{"level":"HOST_COMPONENT","cluster_name":"rest","host_name":"$j","service_name":"MAPREDUCE2"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts/$j/host_components/MAPREDUCE2_CLIENT?HostRoles/state=INIT curl -u admin:admin -H "X-Requested-By:ambari" -i
-X PUT -d '{"RequestInfo":{"context":"Install ZooKeeper
Client","operation_level":{"level":"HOST_COMPONENT","cluster_name":"rest","host_name":"$j","service_name":"ZOOKEEPER"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}'
http://ambari-hostname:8080/api/v1/clusters/rest/hosts/$j/host_components/ZOOKEEPER_CLIENT?HostRoles/state=INIT done < hostcluster.txt
2.Make it execuable : # chmod 755 script.sh 3.Execute it. # ./script.sh NOTE: If any clients are already installed on few
nodes you may see below messages, please don’t panic. ~~ }HTTP/1.1 409 Conflict X-Frame-Options: DENY X-XSS-Protection: 1; mode=block Set-Cookie:
AMBARISESSIONID=vam8jlo7ys401q0r5nm10bm71;Path=/;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT User: admin Content-Type: text/plain Content-Length: 250 Server: Jetty(8.1.19.v20160209) ~~
... View more
- Find more articles tagged with:
- client
- HDFS
- How-ToTutorial
- Mapreduce
- Sandbox & Learning
- YARN
Labels:
08-02-2017
05:25 AM
@Sanjib Behera Is this the same issue as provided in screenshot ? If yes, Could you please check "listeners" section in server.properties. As you are not using kerberos while consuming value of "listeners" should contain PLAINTEXT. If this is a different error then please provide complete traceback message.
... View more
06-30-2017
10:02 AM
How does the AmbariKerberizationWizard generate samaccountnames?
... View more
06-30-2017
10:01 AM
I cannot see the audits for hdfs user being audited in ranger audit. How can I see the audits for the same ?
... View more
Labels:
- Labels:
-
Apache Ranger
06-29-2017
03:13 PM
I am trying to run a word count topology to test storm. But getting following error in supervisor.log 2017-06-27 08:29:55 b.s.config [INFO] SET worker-user 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a xbblwv5
2017-06-27 08:29:55 b.s.d.supervisor [INFO] Running as user:storm command:("/usr/hdp/2.2.6.0-2800/storm/bin/worker-launcher" “storm” "worker" "/disk/hadoop/storm/workers/6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a" "/disk/hadoop/storm/workers/6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a/storm-worker-script.sh")
2017-06-27 08:29:55 b.s.util [WARN] Worker Process 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a:Invalid permissions on worker-launcher binary.
2017-06-27 08:29:55 b.s.util [WARN] Worker Process 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a:The configured nodemanager group 501 is different from the group of the executable 0
2017-06-27 08:29:55 b.s.d.supervisor [INFO] 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a still hasn't started
2017-06-27 08:29:55 b.s.d.supervisor [INFO] Worker Process 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a exited with code: 22
2017-06-27 08:29:55 b.s.d.supervisor [INFO] 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a still hasn't started
2017-06-27 08:29:56 b.s.d.supervisor [INFO] 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a still hasn't started
2017-06-27 08:29:56 b.s.d.supervisor [INFO] 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a still hasn't started
2017-06-27 08:29:57 b.s.d.supervisor [INFO] 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a still hasn't started
2017-06-27 08:29:57 b.s.d.supervisor [INFO] 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a still hasn't started
2017-06-27 08:29:58 b.s.d.supervisor [INFO] 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a still hasn't started
2017-06-27 08:29:58 b.s.d.supervisor [INFO] 6f0c4ad9-a6c2-4c4c-9e48-4f9da485bd2a still hasn't started
... View more
Labels:
- Labels:
-
Apache Storm
05-07-2017
07:24 AM
1 Kudo
@Connor O'Neal In order to login to zookeeper node you can use below command # /usr/hdp/current/zookeeper-server/bin.zkCli.sh -server <hostname>:2181 Although the kafka delete command may seem like it deletes topics and returns successfully, in fact behind the scene it creates - " /admin/delete_topics/<topic> " node in zookeeper and only triggers deletion. We can verify this by checking the same via "zkCli.sh" as below: # cd /usr/hdp/current/zookeeper-server/bin/
# ./zkCli.sh -server <hostname>:2181
[zk: <broker-hostname>:2181 (connected)] ls /
[zk: <broker-hostname>:2181 (connected)] ls /admin [zk: <broker-hostname>:2181 (connected)] ls /admin/delete_topics As soon as broker sees this update, the topic no longer accepts any new produce/consume requests and eventually the topic will be deleted. 1. Topic Command issues topic deletion by creating a new admin path - "/admin/delete_topics/<topic>".
2. The controller listens for child changes on /admin/delete_topic and starts topic deletion for the respective topics
3. The controller has a background thread that handles topic deletion. The purpose of having this background thread is to accommodate the TTL feature, when we have it. This thread is signaled whenever deletion for a topic needs to be started or resumed. Currently, a topic's deletion can be started only by the onPartitionDeletion callback on the controller. In the future, it can be triggered based on the configured TTL for the topic.
A topic will be ineligible for deletion in the following scenarios -
a. broker hosting one of the replicas for that topic goes down
b. partition reassignment for partitions of that topic is in progress
c. preferred replica election for partitions of that topic is in progress (though this is not strictly required since it holds the controller lock for the entire duration from start to end)
4. Topic deletion is resumed when -
a. broker hosting one of the replicas for that topic is started
b. preferred replica election for partitions of that topic completes
c. partition reassignment for partitions of that topic completes
... View more
04-23-2017
03:58 PM
Please refer below link in order to get more information about a specific provenance event for nifi which is running in clustered mode : - How to get information of a specific provenance event when nifi is running in standalone/clustered mode ?
... View more
04-23-2017
03:53 PM
6 Kudos
- Please follow steps mentioned in below link in order to get the list of provenance event id's : Nifi- how to get provenance event id in nifi? - In order to get more information about a specific event : For Nifi running in Standalone mode : # curl -i -X GET http://<hostname>:9090/nifi-api/provenance-events/id For Nifi running in clustered mode: # curl -i -X GET http://<hostname>:9090/nifi-api/provenance-events/id?clusterNodeId=<NODE UUID> There may be multiple events with the same event id(one on each node), so you need to specify from which node you want to return that specific event.
... View more
- Find more articles tagged with:
- How-ToTutorial
- NiFi
- nifi-api
- nifi-reporting
- Sandbox & Learning
Labels:
04-19-2017
09:40 AM
@Sanjib Behera Yes kafka client can be used as a consumer on windows. If your kafka is kerberized then you may face issues related to kerberos if you have long running consumers/producers. Consumer/producer code runs fine until TGT obtained is valid. Once it gets expired you it fails to consume/produce messages. This happens if kinit is not installed and command path is not configured.
... View more
04-19-2017
07:43 AM
@Andy LoPresto I got below ouput after executing GET request to http://<hostname>:9090/nifi-api/provenance/query-id : {"provenanceEvents":[{"id":"a9d352d4-20e9-4fbb-851c-3191204d58ba4","eventId":4,"eventTime":"04/18/2017 09:30:14.775 UTC","lineageDuration":1,"eventType":"CREATE","flowFileUuid":"c87d810d-17b7-4c5f-b313-dd4653e81a8d","fileSize":"0 bytes","fileSizeBytes":0,"clusterNodeId":"a9d352d4-20e9-4fbb-851c-3191204d58ba","clusterNodeAddress":"amnifi1.openstacklocal:9090","groupId":"b90e8d65-015a-1000-22ce-e993c787469e","componentId":"61665eb8-015b-1000-0000-0000653a830f","componentType":"ListFTP","componentName":"AAAAAAAAAAAA","attributes":[{"name":"file.group","value":"0"},{"name":"file.lastModifiedTime","value":"2017-04-18T09:30:00+0000"},{"name":"file.owner","value":"0"},{"name":"file.permissions","value":"rw-r--r--"},{"name":"filename","value":"jj1.txt"},{"name":"ftp.listing.user","value":"akshay"},{"name":"ftp.remote.host","value":"amnifi3.openstacklocal"},{"name":"ftp.remote.port","value":"21"},{"name":"path","value":"ALL_DAY_WORK"},{"name":"uuid","value":"c87d810d-17b7-4c5f-b313-dd4653e81a8d"}],"parentUuids":[],"childUuids":[],"contentEqual":false,"inputContentAvailable":false,"outputContentAvailable":false,"outputContentClaimFileSize":"0 bytes","outputContentClaimFileSizeBytes":0,"replayAvailable":false,"replayExplanation":"Cannot replay data from Provenance Event because the event does not contain the required Content Claim”}, Now when I am trying to submit GET request to get more info about provenance event I am getting below error message : $ curl -i -X GET http://<hostname>:9090/nifi-api/provenance-events/{4} HTTP/1.1 400 Bad Request
Date: Wed, 19 Apr 2017 07:21:13 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Server: Jetty(9.3.9.v20160517)
The cluster node identifier must be specified.
Nifi is clustered. Not sure where I should mentioned cluster node id. Please help.
... View more
04-04-2017
03:40 PM
@Maeve Ryan What is the error you are getting ? How are you checking the size of hdfs ?
... View more
03-31-2017
05:31 AM
@Param NC @Sumit Sharma This is a know issue. What is the version of HDP you are using ? After the cluster is kerberized, the default value of ‘host’ is set to ‘null’ and ‘port’ is set to ‘-1’ which becomes invalid to the legacy Kafka clients. For example: [zk: localhost:2181(CONNECTED) 1] get /brokers/ids/1001 {"jmx_port":-1,"timestamp":"1479492220088","endpoints":["PLAINTEXTSASL://test.support.com:6667"],"host":null,"version":2,"port":-1} The default ‘host’ and ‘port’ here in Zookeeper are used to maintain backward compatibility for older Kafka clients, but only PLAINTEXT protocol is supported as default. If the broker does not listen on PLAINTEXT protocol, for example PLAINTEXTSASL after kerberizing, an empty endpoint (that is, "host":null,"port”:-1) will be registered and older clients will break. Could you please try to manually change the ‘host’ and ‘port’ to actual value. For example: [zk: localhost:2181(CONNECTED) 2] set /brokers/ids/1001 {"jmx_port":-1,"timestamp":"1481503998915","endpoints":["PLAINTEXTSASL://test.support.com:6667"],"host":"test.support.com","version":3,"port":6667}
... View more
03-31-2017
05:14 AM
@Zhao Chaofeng Please refer below links and see if it helps : https://community.hortonworks.com/articles/17059/apache-ranger-and-kafka-1.html https://cwiki.apache.org/confluence/display/RANGER/Kafka+Plugin#KafkaPlugin-CanIauthorizeraccesstoKafkaoveranon-securechannelviaRanger?
... View more
03-31-2017
04:56 AM
2 Kudos
@Sanjib Behera Is your cluster kerberized ? Is yes could you please try to use below command : # sh kafka-console-producer.sh --broker-list XXXX.XXX.com:6667 --topic march31 --security-protocol SASL_PLAINTEXT Please check connectivity between kafka client and kakfka broker: # telnet XXXX.XXX.com:6667
- As which user you are running producer ? - Please post the output of klist command as a user which is executing producer code/script.
... View more
03-31-2017
04:32 AM
@Sumit Sharma - Are you using custom producer or kafka-console-producer script ? Post the exact command used for producing messages to topic. - Are you facing this issue for a specific topic or for all topics ? - As which user you are running producer ? - Please post the output of klist command as a user which is executing producer code/script. - Output of describe command. - What is the version of jdk you are running ?
... View more
03-08-2017
07:08 PM
1 Kudo
@Param NC From command I can see that you are using --new-consumer to describe consumer group. When new consumer is used it tries to fetch consumer group info from consumer offset topics which gets created in kafka log directory. Try using --zookeeper instead of --new-consumer, for eg : $ /usr/bin/kafka-consumer-groups.sh --zookeeper <zookeeper-hostname>:2181 --describe --group <consumer-group>
... View more
03-07-2017
05:05 PM
@yjiang Creating topics with kafka user is a good practice. But If you want to create a topic as a non kafka user in a kerberized environment you need to workaround by following below steps : If you are not using Ranger : 1. Make sure "auto.create.topic.enable = true" 2. Give acl's for the user from which you want to create a topic, for ex : # bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --producer --topic Test-topic 3. Do a kinit as a user from which you want to create topic. 4. Now try to produce messages to topic as that user : # ./kafka-console-producer.sh --broker-list <hostname-broker>:6667 --topic Test-topic --security-protocol PLAINTEXTSASL If you are using Ranger : Instead of point 2 in above steps you will need to add a policy for the topic in ranger. Allow permissions for that user to produce, create, consume. Restart kafka service. Then follow step 3 and 4 as mentioned above.
... View more
03-07-2017
04:59 PM
@yjiang If you try to create a topic as a non kafka user, it creates a topic but with no Leader and ISR. This is a known issue. According to me, the reason behind this could be the zookeeper acl's. Once topic is created in zookeeper, its acl's will not allow kafka to read details about it. If you want to create a topic as a non kafka user you need to workaround by following below steps : If you are not using Ranger : 1. Make sure "auto.create.topic.enable
= true" 2. Give acl's for the user from which you want to create a topic, for ex : # bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --producer --topic Test-topic 3. Do a kinit as a user from which you want to create topic. 4. Now try to produce messages to topic as that user : #
./kafka-console-producer.sh --broker-list <hostname-broker>:6667 --topic Test-topic --security-protocol PLAINTEXTSASL If you are using Ranger : Instead of point 2 in above steps you will need to add a policy for the topic in ranger. Allow permissions for that user to produce, create, consumer. Restart kafka service. Then follow step 3 and 4 as mentioned above. Hope this helps !!
... View more
02-28-2017
11:54 PM
I have added sasl.kerberos.principal.to.local.rules = RULE:[1:$1@$0](test1@OPENSTACKLOCAL)s/@.*/test/,DEFAULT in custom kafka broker in ambari. Now I want to confirm to which user my kerberos principal is resolving to when I try to produce to a topic ? I tried enabling debug in log4j for ranger authorization in kafka but cannot see any of logs in ranger-kafka.log. I cannot enable ranger audit logs. Can some one please help ?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Ranger
02-24-2017
10:54 PM
Consumer will consume messages if you are producing messages in the same topic at the same time. If you want to see the messages which were produced earlier you will need to pass --from-beginning flag to consumer command
... View more
02-23-2017
08:34 PM
@Jeeva Jeeva Could you please check in zookeeper if they are registered : Execute below command to login to zkcli : /usr/hdp/current/zookeeper-server/bin/zkCli.sh ls /brokers/ids/<id's> <-- here you should see two ids as there are two broker nodes get /brokers/ids/<id1> and <id2> <--- Please share the output Do you see any error is server.log which is located at /var/log/kafka/
... View more
02-23-2017
08:26 PM
@Jeeva Jeeva Could you please confirm if all brokers/broker in your cluster is up and running ? If not please start your broker node and try to execute create topic command.
... View more
12-02-2016
06:44 PM
@Sriharsha Chintalapani Yes ! It worked. Thanks a lot. /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1.openstacklocal:2181/kafka --create --topic my-topic --partitions 2 --replication-factor 2
Created topic "my-topic".
... View more
12-02-2016
06:35 PM
@Sriharsha Chintalapani @Rajkumar Singh
... View more
12-02-2016
05:37 PM
HDP version : 2.5 Here are the steps which I have followed : 1. Installed kafka brokers on 2 nodes in my cluster. 2. Tried to create a topic , Created successfully 3. I tried to change the zookeeper root directory for kafka by following below steps : > In ambari > Kafka > Configs > changed value of zookeeper.connect as follows : node1.example.com:2181,node2.example.com:2181,node1.example.com:2181/kafka 4. Saved the changes and manually created /kafka in zookeeper using zkCli.sh 5. Restarted kafka brokers. After restart znodes were created in /kafka in zookeeper. 6. Now when I try to create a topic it fails with below stack trace : Error while executing topic command : replication factor: 2 larger than available brokers: 0
[2016-12-02 16:13:12,851] ERROR kafka.admin.AdminOperationException: replication factor: 2 larger than available brokers: 0
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:117)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:403)
at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:110)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:61)
at kafka.admin.TopicCommand.main(TopicCommand.scala) I can confirm that all znodes are created correctly under /kafka, brokers are up and running fine. I suspect that for some reason kafka admin or kafka controller(not sure who looks at the brokers znode value) is not able to identify live brokers because it tries to find the broker ids in previous zookeeper root dir i.e /brokers/ids/ which is empty. In order to confirm if this is the scenario, I reverted the changes which were made. > Changed the zookeeper root directory to default i.e <node1.example.com:2181,node2.example.com:2181,node1.example.com:2181> > Restarted zookeeper broker > Confirmed that all znodes are with correct value (/controller_epoch) (/brokers/ids/1004 /brokers/ids/1006) Now I tried to create a topic which got created without any issues. Then I deleted both the broker IDS in /brokers/ids and tried to create a topic. It failed with same error i,e : Error while executing topic command : replication factor: 2 larger than available brokers: 0 [2016-12-02 16:13:12,851] ERROR Can some one please help me to understand this behaviour ?
... View more
Labels:
- Labels:
-
Apache Kafka