@saur  Previously  I had written a comprehensive note on this issue but unfortunately, I can't locate it. I have just completed fresh documentation but since attachments were disabled, please  download the document from my adobe share https://documentcloud.adobe.com/link/track?uri=urn:aaid:scds:US:ee72188c-cfb5-48f8-b1cc-e5eae799910b I am sure it will help you keep me posted. Happy hadooping ... View more

@Petch  I don't know whether you unintentionally accept your answer  but it seems you are still questions is your problem resolved  if not then reject the answer and update the thread ... View more

@SKL  Ambari explicitly configures a series of Kafka settings and creates a JAAS configuration file for the Kafka server. It is not necessary to modify these settings but check the below values in Server.properties Listeners   listeners=SASL_PLAINTEXT://kafka01.example.com:6667 listeners=PLAINTEXT://your_host:9092, TRACE://:9091, SASL_PLAINTEXT://0.0.0.0:9093   Advertised.listeners A list of listeners to publish to ZooKeeper for clients to use If advertised.listeners is not set, the value for listeners will be used   advertised.listeners=SASL_PLAINTEXT://kafka01.example.com:6667   Security.inter.broker.protocol In a Kerberized cluster, brokers are required to communicate over SASL   security.inter.broker.protocol=SASL_PLAINTEXT   Principal.to.local.class Transforms the Kerberos principals to their local Unix usernames.   principal.to.local.class=kafka.security.auth.KerberosPrincipalToLocal   super.users Specifies user accounts that will acquire all cluster permissions these super users have all permissions that would otherwise need to be added through the kafka-acls.sh script   super.users=user:developer1;user:analyst1   JAAS Configuration File for the Kafka Server Enabling Kerberos sets up a JAAS login configuration file for the Kafka server to authenticate the Kafka broker against Kerberos. Usually in /usr/hdp/current/kafka-broker/config/kafka_server_jaas.conf     KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/home/ec2-user/kafka.service.keytab" storeKey=true useTicketCache=false serviceName="kafka" principal="kafka/<public_DNS@EXAMPLE.COM"; }; Client { // used for zookeeper connection com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/home/ec2-user/kafka.service.keytab" storeKey=true useTicketCache=false serviceName="zookeeper" principal="kafka/<public_DNS@EXAMPLE.COM"; };   Setting for the Kafka Producer   Ambari usually sets the below key-value pair in the server.properties file if nonexistent please add it:   security.protocol=SASL_PLAINTEXT   JAAS Configuration File for the Kafka Client This file will be used for any client (consumer, producer) that connects to a Kerberos-enabled Kafka cluster. The file is stored at: /usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf Kafka client configuration with keytab, for producers:   KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/home/ec2-user/kafka.service.keytab" storeKey=true useTicketCache=false serviceName="kafka" principal=""kafka/<public DNS>@EXAMPLE.COM"; };   Kafka client configuration without keytab, for producers:   KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="kafka"; };   Kafka client configuration for consumers:   KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="kafka"; };   Check  and set the Ranger policy permissions for kafka and ensure that all the Kafka keytab  is executable by Kafka   Hope that helps       ... View more

@Hadoopservice    Sorry, I was away for a while is this still an issue? ... View more

@Petch  There are a couple of things I would like you to test and share the results What is the version of your OpenJDK? Is it higher or lower than 1.8.0_181? If its lower then you will need to upgrade your JDK Can you test the TLS /SSL negotiation is should return exit code 0 # openssl s_client -connect util.localdomain:7183   SELinux or iptables must be disabled. From datanode1 can you telnet successfully? # telnet 192.168.56.106 7182 Can you check the hostname is properly configured on util # hostname -f   Please revert  ... View more

@Petch    Firstly can you ensure all the agents are running on all the hosts?  I would think it a good idea to truncate the old logs Stop Agent(s) $ sudo service cloudera-scm-agent stop Truncate the agent logs $ sudo truncate --size 0  cloudera-scm-agent.log Restart Agent(s) $ sudo service cloudera-scm-agent restart Checking Agent(s) Status $ sudo service cloudera-scm-agent status Then retry  and let me know  ... View more

@SKL    Please have a look at this response by Vipin Rathor   Ranger Policy download with HTTP response 401   Hope that helps ... View more

@BrandonH1987  Can you elaborate what you wanted to implement?  ... View more

@he-admin  Kerberos is a MUST have for any production environment so I am wondering how you have been running your production with such a security loophole 🙂 How is your cluster managed if you are using CM or Ambari then when you enable Kerberos the principals & keytabs are generated automatically for the components. If your hadoop is an off the shelf deployment then you will have to generate the keytabs using the kadmin CLI. Applications usually authenticate by providing Java Authentication and Authorization Service (JAAS) configuration at the application start. A common way of passing the JAAS configuration to be used is setting the -Djava.security.auth.login.config Java option with the configuration file path. https://andriymz.github.io/kerberos/authentication-using-kerberos/#authenticate-with-jaas-configuration-and-a-keytab     kafkaJaas { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=false useKeyTab=true doNotPrompt=true keyTab="/path/to/spark.keytab" principal="spark@YOUR_REALM"; };     Enabling Kerberos doesn't not change your software code because the Kerberos config files don't inject any code to your application software but rather it becomes a gatekeeper trying to protect unauthenticated execution of your software. The executor should be able to be identified (keytab) against the KDC database  Kerberos uses  symmetric key cryptography  and requires trusted third-party authorization to verify user identities. There are a couple of example of applications authenticating transparently with Kerberos ticket renewal Hope that helps ... View more

@vstadnyk    I think that was a temporary issue because the repo is online now. ... View more

@Petch  The problem seems to be with port 7182 on the cluster but I can ss that its OK on host util.localdomain can you validate on these hosts too 192.168.56.xx1 namenode.localdomain namenode 192.168.56.xx2 datanode1.localdomain datanode1 192.168.56.xx3 datanode2.localdomain datanode2 192.168.56.xx4 datanode3.localdomain datanode3 There could be a firewall running on one of those hosts. Please do check that and let me know       ... View more

@ARVINDR  The " safemode get | grep 'Safe mode is OFF'' returned 1 " means  the Namenode is not started at all or in safe mode or   in safe mode   Could you do the following while logged on as hdfs user $ hdfs dfsadmin -safemode get If you see something like below  safemode: Call From  inqchdpmn1.XXX.com/10.10.31.71 to  inqchdpmn1.XXX.com :8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused Then the Namenode is not running the check and upload the hadoop-hdfs-secondarynamenode-xx and hadoop-hdfs-datanode-xxxlogs in /var/log/hadoop/hdfs/ Solution 2 Try starting the HDP namenode service manually  Share your findings ... View more

@kvinod    Sorry I have been away for quite a while. What's the relation between hostname001 and hostname003 or simply what services are running on these 2 servers? Have you tried regenerating the keytabs? ... View more

@janvi    Could you also share details of the version of your HDP  Cluster? And the path to the  log4j.appender.file you shared? ie Ambari UI, browse to HDFS > Configs > Advanced. ... View more

@MorkSawz  Can you explain the method or how you downloaded the big-data-2.zip ? If it was downléaoded in the Downloads directory then first try to run the below command to verify.   [cloudera@quickstart Downloads] ls   I think there is something you did wrongly. I fired up a CDH sandbox and tested the download and it was successful here is what I did I then navigated to the Downloads directory in the  CDH sandbox and used wget. I couldn't use your Coursera link as it needs a password so I tried downloading something free like skype this should also work in your case. I right-clicked to get the link to the zip executable See  screenshot 1 Then I used wget and later verified using  $ ls  as above and I could see the file in the Downloads directory That should work also for your zip file. happy hadooping  ... View more

@ARVINDR  There is something you got to investigate. The correct output should look like below. I remember answering such a question something I need to locate the solution     [zk: localhost:2181(CONNECTED) o] ls /hiveserver2 [serverUri=hdp2.test.com:10000;version=1.2.1000.2.6.2.0-205;sequence=0000000061] [zk: localhost:2181(CONNECTED) 1] Can you get the ACL for that znode [zk: localhost:2181(CONNECTED) 0] getAcl /hiveserver2 'world,'anyone : cdrwa [zk: localhost:2181(CONNECTED) 1]   The above is for a non-kerberized cluster . Please revert ... View more

@SwasBigData  I think you missed the outputyou  but you should still be able to locate the log. There is a step during the configure Ranger and DAS Databases that should have give you and output. After running the gen_embedded_ranger_db.sh located in /opt/cloudera/cm/bin on the Cloudera Manager host I will do a trial install tomorrow. ... View more

@npdell  Before starting the upgrade did you by any change validate the upgrade path with Cloudera supportmatrix  this should have been your first reference source. ... View more

@desind  I can see the error  Authentication is not valid  but it seems you didn't use the format super:password->super:DyNYQEQvajljsxlhf5uS4PJ9R28=   instead, your input was as below according to the steps you shared. addauth digest super:password And then  delete  the znode  that should work [zk: xxx.unx.sas.com(CONNECTED) 2] deleteall /kafka-acl/Topic Please do that and revert         ... View more

@ARVINDR  Yes, its possible first questions first Don't attempt this if this is a production cluster  !!! Only when it's your scratch/dev cluster. What is the HDP version? The username mapping is managed by the Isilon admin  I don't know how the authorization works maybe the HDP admin user has been delegated to Isilon where changing it at the cluster level won't synchronize with the OS. After the above response, we can come up with a procedure        ... View more

@desind  By default, Zookeeper runs without the option of becoming a superuser to administrate znodes in the ZK ensemble, for example, to fix ACLs, remove znodes that are not required anymore, or create new ones in specific locations. Zookeeper grants permissions through ACLs through different schemas or authentication methods, such as 'world', 'digest', or 'sasl' if we use Kerberos. We can potentially we locked out if we were to grant everyone just read permissions to a znode, as we would not be able to delete it or modify it anymore.     ... View more

@desind  I tweaked it a little bit it should work in Cloudera Go to Cloudera zookeeper server home   # cd $CDH_HOME/zookeeper-server   Run below command   java -cp "./zookeeper.jar:lib/slf4j-api-1.6.1.jar" org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:password   The output should look like below   SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See <a href="<a href="http://www.slf4j.org/codes.html#StaticLoggerBinder" target="_blank">http://www.slf4j.org/codes.html#StaticLoggerBinder</a>" target="_blank"><a href="http://www.slf4j.org/codes.html#StaticLoggerBinder</a" target="_blank">http://www.slf4j.org/codes.html#StaticLoggerBinder</a</a>> for further details. super:password->super:DyNYQEQvajljsxlhf5uS4PJ9R28=   Copy the super: DyNYQEQvajljsxlhf5uS4PJ9R28=  text and login to Cloudera Manager and goto zookeeper config. Add below to zookeeper-env template config   export SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Dzookeeper.DigestAuthenticationProvider.superDigest=super:DyNYQEQvajljsxlhf5uS4PJ9R28="   Save and Restart Zookeeper and l aunch zookeeper shell on CDH cli   # . /bin/zkCli.sh -server your_server.com   addauth as below Now to removing  the ACL should work Now try to delete an ACL in zookeeper this should work.   addauth digest super:password   Unfortunately, I don't have a CDH sandbox  so you might have to adjust some cmds     ... View more

@ARVINDR  In a usual setup, the hive service should run on a master node and you run hive client on the client nodes ie. edge nodes and data nodes because during deployment Ambari copies the master configuration to all clients host e.g copy hive config to all host where hive client has been installed.   ... View more

@ARVINDR  Why did you have to re-install the service? The issue is with Reason: Error mapping uname 'hive' to uid remember the hive uid is mapped to root. I am not an Isilon expert so I am really handicapped here Please let me know  ... View more

@SwasBigData  I have never deployed on GCP and unfortunately, I am not a partner or running the newer CDP the environments I have and HDP 3.x Goodbye open source everything now behind a paywall catch me up on LinkedIn Geoffrey Shelton  ... View more