@EricL  Sorry about the confusion, the location of /etc/security/keytabs/* is specific to HDP the equivalent of  /var/run/cloudera-scm-agent/process   so try to map  so the command    $ klist -kt  /var/run/cloudera-scm-agent/process/*.keytab   shoule be valid  ... View more

@kal  I have written an article in response to a similar question in HCC before, you have a couple of things to do before this can work! My 2 cents advice you should have first configured the 3 clusters using single KDC, then added a second KDC's on each cluster then proceed to configure Cross_Realm Trust. In the process, you would have gained some knowledge on the implementation having said that you will need KDCs in a Master/Slave configuration to be able to propagate principal and Keytabs between the 2 KDC using krb5_prop this is another chapter on its own. I have already noticed also an error in your krb5.conf in the below part, first you have 3 KDC's which are not replicating.there are specific steps to enable replication between KDC's see the one highlighted in ORANGE the values on the left in [lower case ] should mirror on the right in [upper case] see valid example in BLUE   [domain_realm] .xxxxxxx.net = XXXXXXX.NET xxxxxxx.net = XXXXXXX.NET .xxxxxxx.com = XXXXXXX.NET xxxxxxx.com = XXXXXXX.NET .sys.xxxxxxx.net = SPECTRA.XXXXXXX.NET sys.xxxxxx.net = SPECTRA.xxxxxx.NET --Valid-- [domain_realm] .xxxxxxx.net = XXXXXXX.NET xxxxxxx.net = XXXXXXX.NET .xxxxxxx.com = XXXXXXX.COM xxxxxxx.com = XXXXXXX.COM .spectra.xxxxxxx.net = SPECTRA.XXXXXXX.NET spectra.xxxxxx.net = SPECTRA.xxxxxx.NET You should also configure /etc/hosts file on all the 3 clusters to have IP--HOSTNAME--ALIAS and the files should be copied to all hosts in the cluster if DNS is not resolving. The example below depicts hosts in the 3 different network segments # Cluster 1 192.168.0.1 node1.SPECTRA.XXXXXXX.NET node1 192.168.0.2 node2.SPECTRA.XXXXXXX.NET node2 .... 192.168.0.3 node3.SPECTRA.XXXXXXX.NET node3 # Cluster 2 192.168.1.10 node01.XXXXXXX.NET node01 192.168.1.20 node02.XXXXXXX.NET node02 ......... 192.168.1.30 node03.XXXXXXX.NET node01 # Cluster 3 192.168.2.30 nodex.XXXXXXX.COM nodex 192.168.2.40 nodey.XXXXXXX.COM nodey ......... 192.168.2.50 nodez.XXXXXXX.COM nodez if you could breakdown your steps it would be easier to achieve as I reiterated in the beginning but its doable task. Please let me know ... View more

@kal    whats the output of the below snippet? # klist -kt /etc/security/keytabs/hive.service.keytab Could you also share your krb5.conf? Please garble in the important info but not the format 🙂 ... View more

@irfangk1  If its an HDP cluster then I assume you are using Ambari for managing the HDF cluster, you will need to first prepare the 2 new hosts see the ​Prepare the EnvironmentCloudera document   Then add the hosts to the cluster see Add host to cluster Thereafter add HDF to these 2 new nodes it follows the same procedure as adding HDF Services on an existing HDP Cluster hTH ... View more

@irfangk1  It would be good to clarify whether its a CDH or HDP cluster managed by Ambari or not and any relevant information.     ... View more

@Manoj690  Since the hbase meta  is corrupt, I think this is the procedure to resolve that issue How to fix corrupted files for an HBase table   ... View more

@Manoj690    I think you are missing the MySQL database port  3306 !  the DB listener process runs on that port.  without the port, sqoop won't know how to connect. Can you try  sqoop import --connect jdbc:mysql://hostname:3306/ambari1 --username ambari1 --password xxxxx --table emp -m 1   From the Ambari UI you should validate the default port 3306 is configured  and the connection is working or  CLI HTH       ... View more

@pritam_konar    Can you change to the following, maintaining the authentication to Kerberos won't compromise your security    hadoop.http.authentication.simple.anonymous.allowed=true hadoop.security.authentication=kerberos   Please revert ... View more

@vivek_b2  The " on connection exception: java.net.ConnectionException: Connection refused "  is a network issue and the  GSS Exception is a Kerberos one.  Those are 2 different things if you are running the job from the edge node,  which user is executing the job?  Assuming is a user Dev1 on the edge node can validate this user has a valid Kerberos ticket Dev1@localhost $ klist   Share the output of the above snippet !! As you are executing the code from the edge node can you verify that the krb5.conf  file on the edge node is identical to the one on the KDC server, this file should be exactly the same on both the edge node and KDC server. Was the Kerberos client installed on the edge node see below command yum install krb5-workstation The user running the job should kinit with his keytab to be able to grab a valid ticket. I recently created a document to answer a similar Kerberos issue on the edge node. Please check my procedure for creating a user keytab on the edge node to enable him/her excute jobs in a kerberized cluster https://community.cloudera.com/t5/Support-Questions/HDFS-is-not-accessible-from-an-user-after-kerberos/m-p/268878#M206457 Hope that help        ... View more

@subhash_parise3    Can you replace these lines with the below blue line with the orange   export SPARK_HISTORY_OPTS='-Dspark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter -Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=kerberos,kerberos.principal=HTTP/ip-0-0-0-0@HDPCLUSTER.LOCAL,kerberos.keytab=/etc/security/keytabs/spnego.service.keytab"   With the below text block   export SPARK_HISTORY_OPTS='-Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter. params="type=kerberos,kerberos.principal ={ {spnego_principal}},kerberos.keytab={ {spnego_keytab}}"'   Restart and let me know      ... View more

@vivek_b2  Is the cluster running fine? If so has the /etc/hosts on the edge node have entries for you namenodes? Can it resolve the IP's of name node 1 and name node 2. Your issue looks a connectivity issue. I would usually start with the usual culprits FW, DNS and host entry etc HTH ... View more

@Manoj690    Successful Smartsense startup needs an  ID to uniquely identifies a customer account. You can obtain it from Cloudera/Hortonworks support. This is a mandatory field during SmartSense installation see below link   https://docs.hortonworks.com/HDPDocuments/SS1/SmartSense-1.5.0/reference/content/ss_hst_server.html   customer.smartsense.id  The error in your logs tells you exactly why the server isn't staring <Failed to execute command: /usr/sbin/hst start ; Exit code: 1; stdout: Invalid SmartSense ID: unspecified ERROR: Validation failed. Upgrading database... SmartSense Server is not running Done upgrading the database.   Once you resolve that your smart sense should fire up without any issue.       ... View more

@VeljkoC  To be able to apply a Ranger policy the user MUST exist in the /etc/passwd basically that's the list that Ranger pulls. For now I don't see the workaround out there should be some Linux freaks that can help on the HDP side I have hit a dead end! ... View more

@pritam_konar    In reality a user shouldn't be able to execute or kinit with the hdfs keytab but have a keytab created for the specific user and when need be deleted when the user is disabled on the cluster typically this user setup happens on the edge node where the hadoop client software are installed and is the recommended setup for giving users access to the cluster. Below is a demo of the user konar when he attempts to access services in a kerberized cluster # su - konar [konar@simba ~]$ id uid=1024(konar) gid=1024(konar) groups=1024(konar) Now try to list the directories in HDFS [konar@simba ~]$ hdfs dfs -ls / Error  19/08/24 23:59:25 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "simba.kenya.ke/192.168.0.87"; destination host is: "simba.kenya.ke":8020;   Below is the desired output when the user konar attempts to use the hdfs headless keytab, [konar@simba ~]$ kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-jair@KENYA.KE kinit: Permission denied while getting initial credentials To enable a user to access the cluster, on the Kerberos server as the root (Keberos admin)  do the following steps, Assumption  Realm is KENYA.KE  and KDC  host is simba and you have root access on the KDC. Create the admin principle for user konar [root@simba ~]# kadmin.local Authenticating as principal root/admin@KENYA.KE with password. kadmin.local: addprinc konar@KENYA.KE WARNING: no policy specified for konar@KENYA.KE; defaulting to no policy Enter password for principal "konar@KENYA.KE": Re-enter password for principal "konar@KENYA.KE": Principal "konar@KENYA.KE" created. kadmin.local: q Validate the principal was created using the subcommand listprincs [List principals] and limiting the output by restricting to  konar classic Unix stuff [root@simba ~]# kadmin.local Authenticating as principal root/admin@KENYA.KE with password. kadmin.local: listprincs *konar konar@KENYA.KE Type q [quit] to exit the kadmin utility   Generate the keytab Generate keytab for user konar using the ktutil, it's good to change to /tmp or whatever you choose so you know the location of the generated keytab your encryption Algorithm could be different but this should work [root@simba tmp]# ktutil ktutil: addent -password -p konar@KENYA.KE -k 1 -e RC4-HMAC Password for konar@KENYA.KE: ktutil: wkt konar.keytab ktutil: q   Validate the keytab creation Check the keytab was generated in the current directory, notice the file permissions!! [root@simba tmp]# ls -lrt -rw------- 1 root root 58 Aug 25 18:22 konar.keytab   As root copy the generate keytab to the home directory of user konar typically on the edge node [root@simba tmp]# cp konar.keytab /home/konar/   Change to konar's home dir and vaildate the copy was successful [root@simba tmp]# cd /home/konar/ [root@simba konar]# ll total 4 -rw------- 1 root root 58 Aug 25 18:28 konar.keytab   Change file ownership Change the file permission on the konar.keytab so that user konar has the appropriate permissions. [root@simba konar]# chown konar:konar konar.keytab [root@simba konar]# ll total 4 -rw------- 1 konar konar 58 Aug 25 18:28 konar.keytab   Switch to user konar and validate that the user has can't  still access to hdfs $ hdfs dfs -ls / [konar@simba ~]$ hdfs dfs -ls / Output 19/08/25 18:36:44 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "simba.kenya.ke/192.168.0.87"; destination host is: "simba.kenya.ke":8020;   The kerberos klist also confirms that [konar@simba ~]$ klist klist: No credentials cache found (filename: /tmp/krb5cc_1024)   As user Konar now try to kinit with the correct principal, the first step is to identify the correct principal [konar@simba ~]$ klist -kt konar.keytab Keytab name: FILE:konar.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 1 08/25/2019 18:22:34 konar@KENYA.KE The above shows the konar user keytab is valid with the principal in the output   Now user konar can grab a valid ticket ûsing the below snippet concatenating the keytab + principal   [konar@simba ~]$ kinit -kt konar.keytab konar@KENYA.KE The above should throw any error   Now validate the user has a valid ticket [konar@simba ~]$ klist Ticket cache: FILE:/tmp/krb5cc_1024 Default principal: konar@KENYA.KE Valid starting Expires Service principal 08/25/2019 18:53:40 08/26/2019 18:53:40 krbtgt/KENYA.KE@KENYA.KE Bravo you have a valid ticket and hence access to the cluster let's validate that the below  HDFS list  directory should succeed   [konar@simba ~]$ hdfs dfs -ls / Found 10 items drwxrwxrwx - yarn hadoop 0 2018-12-17 21:53 /app-logs drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:22 /apps drwxr-xr-x - yarn hadoop 0 2018-09-24 00:12 /ats drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:12 /hdp drwxr-xr-x - mapred hdfs 0 2018-09-24 00:12 /mapred drwxrwxrwx - mapred hadoop 0 2018-09-24 00:12 /mr-history drwxr-xr-x - hdfs hdfs 0 2018-12-17 19:16 /ranger drwxrwxrwx - spark hadoop 0 2019-08-25 18:59 /spark2-history drwxrwxrwx - hdfs hdfs 0 2018-10-11 11:16 /tmp drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:23 /user   User konar can now list and execute jobs on the cluster !!!! as reiterated the konar user in a recommended architecture should be on the edge node.       ... View more

Can you execute these 2 snippets and revert   Edit the  /etc/ambari-agent/conf/ambari-agent.ini   sed -i /\[security\]/a\/force_https_protocol=PROTOCOL_TLSv1_2 /etc/ambari-agent/conf/ambari-agent.ini   Edit the  /etc/python/cert-verification.cfg sed -i 's/verify=.*/verify=disable/g' /etc/python/cert-verification.cfg   Restart the ambari-server and ambari-agent # ambari-server restart    [ One the ambari server ] # ambari-agent restart   [ One the clients with  ambari  agent ] Please let me know the outcome   ... View more

Make sure you have changed these properties in your hbase-site.xml: <property> <name>hbase.thrift.support.proxyuser</name> <value>true</value> </property> <property> <name>hbase.regionserver.thrift.http</name> <value>true</value> </property> Then check in HDFS-->Configs-->Advanced-Custome core-site.xml add these 2 properties to enable HBase to impersonates a user    <property> <name>hadoop.proxyuser.hbase.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hbase.groups</name> <value>*</value> </property>                        Please do these modifications and start any stale configs and revert ... View more

@jkibler  Your syntax looks okay save for a small change by fully qualifying the table name as the table doesn't belong to your schema?   SQL Server uses [dbo.tablename] so in the table parameter try --table dbo.table_name Hope that helps ... View more

@Manoj690    Please follow the below steps remember to skip steps you've already executed on the ambari database host.   # Install the mysql connector [optional if already executed] yum install -y mysql-connector-java ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar   # Enable auto start on boot [optional if already executed] sudo systemctl start mysqld [if not started] sudo systemctl enable mysqld I think you didn't create the Ambari backend database, please follow the steps below as root Assumption root password=w3lc0m31 ambari username = ambari ambari user password=ambari   mysql -u root -pw3lc0m31 CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'gaian-lap386.com'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'gaian-lap386.com' IDENTIFIED BY 'ambari'; FLUSH PRIVILEGES; # Log on as Ambari user and create the Ambari DB mysql -u ambari -pambari CREATE DATABASE ambari; USE ambari; SOURCE /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql; exit; After successfully running the below switch to the  ambari host CLI, the default mysql database port is 3306 anything else is not acceptable. The below command should pull parameters macting the below # grep 'jdbc' /etc/ambari-server/conf/ambari.properties server.jdbc.database=mysql server.jdbc.database_name=ambari server.jdbc.driver=com.mysql.jdbc.Driver server.jdbc.hostname=gaian-lap386.com server.jdbc.port=3306 server.jdbc.rca.driver=com.mysql.jdbc.Driver server.jdbc.rca.url=jdbc:mysql://gaian-lap386.com:3306/ambari server.jdbc.rca.user.name=ambari server.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.dat server.jdbc.url=jdbc:mysql://gaian-lap386.com:3306/ambari server.jdbc.user.name=ambari # Start Ambari as root user # ambari-server start This time it should succeed. # To configure hive, oozie, ranger and rangerkms use this template mysql -uroot -pw3lc0m31 create database hive; grant all privileges on hive.* to 'hive'@'localhost' identified by 'hive'; grant all privileges on hive.* to 'hive'@'%gaian-lap386.com' identified by 'hive';   Substitute the values and names for the subsequent components after the Ambari startup ensure these values are matched in the subsequent component database setup Hope that helps ... View more

@Malthe Borch Nifi cluster coordinator relays on the zookeeper for the election. NiFi employs a Zero-Master Clustering paradigm. Each node in the cluster performs the same tasks on the data, but each operates on a different set of data. One of the nodes is automatically elected (via Apache ZooKeeper) as the Cluster Coordinator. All nodes in the cluster will then send heartbeat/status information to this node, and this node is responsible for disconnecting nodes that do not report any heartbeat status for some amount of time. Additionally, when a new node elects to join the cluster, the new node must first connect to the currently-elected Cluster Coordinator in order to obtain the most up-to-date flow. If the Cluster Coordinator determines that the node is allowed to join (based on its configured Firewall file), the current flow is provided to that node, and that node is able to join the cluster, assuming that the node’s copy of the flow matches the copy provided by the Cluster Coordinator. If the node’s version of the flow configuration differs from that of the Cluster Coordinator’s, the node will not join the cluster. Checklist Ensure you have 3 zookeepers (ensemble) to manage your Nifi cluster, and all should be up and running. Walkthrough Stop all the Nifi instances Ensure the 3 zookeepers are up and running Start the Nifi one at a time Validate that one nifi has been elected coordinator as is PRIMARY See screenshots Coordinator elected HTH ... View more

@Koffi Can you share your HDP setup? HDP Version, number of nodes, HA or not, Kerberized or not and share the exact error being thrown preferably logs ! Typically a node manager should run on the same node as the data node. HTH ... View more

@Sugumar Srinivasan My question is do you have something of value that you want to retrieve for that lost log? if so try the method below I had it documented but it has never occurred to me to use it Else the file will be recreated when that service restarts Recovery If a running program still has the deleted file open, you can recover the file through the open file descriptor in /proc/[pid]/fd/[num] . To determine if this is the case, you can attempt the following: $ lsof | grep "/var/log/Hadoop-yarn/yarn/yarn-yarn-timelineserver-hn0-myprha.log" If the above gives an output of the form: progname 5383 user 22r REG 8,1 16791251 265368 /path/to/file Take note of the PID in the second column, and the file descriptor number in the fourth column. Using this information you can recover the file by issuing the command: $ cp /proc/5383/fd/22 /path/to/restored/file HTH ... View more

@Swapnil Sonawane Your problem seems to be vast! The /etc/security/keytabs should be mounted as persistent volumes PVC stateful or use secrets to mount the files. You will need to share your deployment.yaml or whatever config HELM or other you are using, ... View more