About Shelton

Shelton · ‎03-26-2020

@ARVINDR There is something you got to investigate. The correct output should look like below. I remember answering such a question something I need to locate the solution [zk: localhost:2181(CONNECTED) o] ls /hiveserver2 [serverUri=hdp2.test.com:10000;version=1.2.1000.2.6.2.0-205;sequence=0000000061] [zk: localhost:2181(CONNECTED) 1] Can you get the ACL for that znode [zk: localhost:2181(CONNECTED) 0] getAcl /hiveserver2 'world,'anyone : cdrwa [zk: localhost:2181(CONNECTED) 1] The above is for a non-kerberized cluster . Please revert

Shelton · ‎03-25-2020

@SwasBigData I think you missed the outputyou but you should still be able to locate the log. There is a step during the configure Ranger and DAS Databases that should have give you and output. After running the gen_embedded_ranger_db.sh located in /opt/cloudera/cm/bin on the Cloudera Manager host I will do a trial install tomorrow.

Shelton · ‎03-25-2020

@npdell Before starting the upgrade did you by any change validate the upgrade path with Cloudera supportmatrix this should have been your first reference source.

Shelton · ‎03-25-2020

@desind I can see the error Authentication is not valid but it seems you didn't use the format super:password->super:DyNYQEQvajljsxlhf5uS4PJ9R28= instead, your input was as below according to the steps you shared. addauth digest super:password And then delete the znode that should work [zk: xxx.unx.sas.com(CONNECTED) 2] deleteall /kafka-acl/Topic Please do that and revert

Shelton · ‎03-24-2020

@ARVINDR Yes, its possible first questions first Don't attempt this if this is a production cluster !!! Only when it's your scratch/dev cluster. What is the HDP version? The username mapping is managed by the Isilon admin I don't know how the authorization works maybe the HDP admin user has been delegated to Isilon where changing it at the cluster level won't synchronize with the OS. After the above response, we can come up with a procedure

Shelton · ‎03-24-2020

@desind By default, Zookeeper runs without the option of becoming a superuser to administrate znodes in the ZK ensemble, for example, to fix ACLs, remove znodes that are not required anymore, or create new ones in specific locations. Zookeeper grants permissions through ACLs through different schemas or authentication methods, such as 'world', 'digest', or 'sasl' if we use Kerberos. We can potentially we locked out if we were to grant everyone just read permissions to a znode, as we would not be able to delete it or modify it anymore.

Shelton · ‎03-24-2020

@desind I tweaked it a little bit it should work in Cloudera Go to Cloudera zookeeper server home # cd $CDH_HOME/zookeeper-server Run below command java -cp "./zookeeper.jar:lib/slf4j-api-1.6.1.jar" org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:password The output should look like below SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See <a href="<a href="http://www.slf4j.org/codes.html#StaticLoggerBinder" target="_blank">http://www.slf4j.org/codes.html#StaticLoggerBinder</a>" target="_blank"><a href="http://www.slf4j.org/codes.html#StaticLoggerBinder</a" target="_blank">http://www.slf4j.org/codes.html#StaticLoggerBinder</a</a>> for further details. super:password->super:DyNYQEQvajljsxlhf5uS4PJ9R28= Copy the super: DyNYQEQvajljsxlhf5uS4PJ9R28= text and login to Cloudera Manager and goto zookeeper config. Add below to zookeeper-env template config export SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Dzookeeper.DigestAuthenticationProvider.superDigest=super:DyNYQEQvajljsxlhf5uS4PJ9R28=" Save and Restart Zookeeper and l aunch zookeeper shell on CDH cli # . /bin/zkCli.sh -server your_server.com addauth as below Now to removing the ACL should work Now try to delete an ACL in zookeeper this should work. addauth digest super:password Unfortunately, I don't have a CDH sandbox so you might have to adjust some cmds

Shelton · ‎03-24-2020

@ARVINDR In a usual setup, the hive service should run on a master node and you run hive client on the client nodes ie. edge nodes and data nodes because during deployment Ambari copies the master configuration to all clients host e.g copy hive config to all host where hive client has been installed.

Shelton · ‎03-24-2020

@ARVINDR Why did you have to re-install the service? The issue is with Reason: Error mapping uname 'hive' to uid remember the hive uid is mapped to root. I am not an Isilon expert so I am really handicapped here Please let me know

Shelton · ‎03-24-2020

@SwasBigData I have never deployed on GCP and unfortunately, I am not a partner or running the newer CDP the environments I have and HDP 3.x Goodbye open source everything now behind a paywall catch me up on LinkedIn Geoffrey Shelton

Shelton · ‎03-24-2020

@desind You are not running HDP so are you on MapR or Cloudera?

Shelton · ‎03-24-2020

@kvinod I can see the setuid bit (drwxr-s---) was set which alters the standard behavior so that the group of the files created inside said directory, will not be that of the user who created them, but that of the parent directory itself $ ls -lrt /disk1/yarn/nm/usercache total 4 drwxr-s--- 4 mcaf yarn 4096 Feb 24 01:26 mcaf Can you remove the setuid bit as the root user # chmod -s /disk1/yarn/nm/usercache/mcaf Then rerun Question1. You don't need to explicitly change file permission when you enable Kerberos, it should work out of the box Question2. You don't need to regenerate a new mcafmerged.keytab just copy it to you other edge nodes it should work as that edge node is also part of the cluster Please revert

Shelton · ‎03-24-2020

@ARVINDR In addition to @ stevenmatison switch command which people usually ignore Add a Hive proxy To prevent network connection or notification problems, you must add a hive user proxy for HiveServer Interactive service to access the Hive Metastore Steps: In Ambari, select Services > HDFS > Configs > Advanced. In Custom core-site, add the FQDNs of the HiveServer Interactive host or hosts to the value of hadoop.proxyuser.hive.hosts. Save the changes. Can you also set the hive.server2.enable.doAs hive.server2.enable.doAs=true --> Run hive scripts as end user instead of Hive user. hive.server2.enable.doAs=false --> All the jobs will run as hive user.

Shelton · ‎03-24-2020

@ARVINDR The user hdpuser1 should exist locally, to check whether you have a similar output. Did you run the below. # useradd hdpuser1 To check run the below # cat /etc/passwd | grep hdpuser1 hdpuser1:x:1000:1000:hdpuser1:/home/hdpuser1:/bin/bash I am not an expert on Isilon so user mapping is something I need to educate myself o. Do you have some documentation that I could read?

Shelton · ‎03-24-2020

@SwasBigData In the previous step [Setup Database] you must have entered values need at this stage data_analytics_studio_database_host [host where you DAS database is running] data_analytics_studio_database_password [password for the above DAS databases default] Where is your CDP running AWS or Azure?

Shelton · ‎03-24-2020

@npdell Curious have you checked the firewalls are off and most important that these first 2 lines are present in your/etc/hosts files on each host. Please uncomment them if they are commented out! 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 # Your host entry below here # 192.168.225.68 [FQDN] [ALIAS] Maybe also try to ping and sshing from one host to another Please revert

Shelton · ‎03-24-2020

@npdell " No route to host " Signals that an error occurred while attempting to connect a socket to a remote address and port. Typically, the remote host cannot be reached because of an intervening firewall, or if an intermediate router is down. If you are not using static IP's can you check on hosts 192.168.225.165,192.168.225.68 and 192.168.225.171 that their IP's haven't changed by just running $ ifconfig The output should match the IP's in the /etc/hosts table Please do that and revert

Shelton · ‎03-24-2020

@kasa Can you share a scrambled version of your krb5.conf from both clusters and the auth-to_local of both clusters When copying data from a secure cluster to a secure cluster, the following configuration setting is required in the core-site.xml file: <property> <name>hadoop.security.auth_to_local</name> <value></value> <description>Maps kerberos principals to local user names</description> </property> Secure-to-Secure: Kerberos Principal Name Assign the same principle name to applicable NameNodes in the source and destination clusters. distcp hdfs://hdp-2.0-secure hdfs://hdp-2.0-secure The SASL RPC client requires that the remote server’s Kerberos principal must match the server principal in its own configuration. Therefore, the same principal name must be assigned to the applicable NameNodes in the source and the destination cluster. For example, if the Kerberos principal name of the NameNode in the source cluster is nn/host1@realm, the Kerberos principal name of the NameNode in destination cluster must be nn/host2@realm, rather than nn2/host2@realm. Secure-to-Secure: ResourceManager mapping rules When copying between two HDP2 secure clusters, further ResourceManager (RM) configuration is required if the two clusters have different realms. Can you share your hadoop.security.auth_to_local on both clusters, in order for DistCP to succeed, the same RM mapping rule must be used in both clusters. I am assuming the REALMS are TEST.COM and DEV.COM for cluster 1 and 2 respectively <property> <name>hadoop.security.auth_to_local</name> <value> RULE:[2:$1@$0](rm@.*CLUSTER1.TEST.COM)s/.*/yarn/ RULE:[2:$1@$0](rm@.*CLUSTER2.DEV.COM)s/.*/yarn/ DEFAULT </value> </property> Can you try that and revert

Shelton · ‎03-24-2020

@kvinod That's great that the initial issue was resolved with the keytab merge, but if I could ask why did you merge all the key tabs to mcafmerged.keytab it could have been proper to merge only the Hbase and your mcaf keytab , anyway that said your subsequent is a permission issue on the directory /disk1/yarn/nm/usercache/mcaf. Can you share the output of $ ls /disk1/yarn/nm/usercache and $ ls /disk1/yarn/nm/usercache/mcaf Can you try changing the permission with the correct group for user mcaf i.e as the root user # chown -R mcaf:{group} /disk1/yarn/nm/usercache/mcaf Then rerun the Terragen command that should work. Keep me posted

Shelton · ‎03-23-2020

@kasa Disctcp is used for inter/Intracluster copy but the command you are running is not wrong because you need the source and destination NameNodes. $ hadoop distcp /user/home/test.txt /tmp/ The most common use of DistCp is an inter-cluster copy, where you copy from NameNode1[nn1] to Namenode2[nn2] on 2 different clusters and both clusters should be up and running during the process $ hadoop distcp hdfs://nn1:8020/source hdfs://nn2:8020/destination Where hdfs://nn1:8020/source is the data source, and hdfs://nn2:8020/destination is the destination. This will expand the namespace under /source on NameNode "nn1" into a temporary file, partition its contents among a set of map tasks, and start copying from "nn1" to "nn2". Note that DistCp requires absolute paths. Personally I think you should use CopyToLocal instead as according to my understanding you are trying to copy a file from hdfs to you local tmp directory Assuming your directory /user/home/ is in hdfs and you are running the command as HDFS user! This will copy the test.txt from hdfs to local /tmp directory $ hdfs dfs -copyToLocal /user/home/test.txt /tmp/ And to successfully copy between 2 kerberized cluster you should perform the Kerberos cross-realm trust for distcp it's simple to setup just follow the guide and you will be fine Please let me know if my assumption is correct

Shelton · ‎03-23-2020

@stalsams If you have the root access just switch to user sqoop as root user # su - sqoop $ id uid=1020(sqoop) gid=1007(hadoop) groups=1007(hadoop) Now you should be able to execute all sqoop commands with no need for a password

Shelton · ‎03-23-2020

@Arun66 unfortunately with such a vague and incomplete log, we can't help much. Questions? CDH or HWX Shar e the logs? Share the command being executed? Kerberized or not And any indo you deem important Happy hadooping

Shelton · ‎03-23-2020

@chinni Please elaborate your question else the simplest way to get the latest file in Linux $ ls -lrt /automation_test/oozie/output/Bigdata_Counts/ But I guess you want to do that programmatically?

Shelton · ‎03-23-2020

@ARVINDR This is a permission issue in hdfs, so what you should do is as follow to resolve the issue Assuming you are logged in as root change recursively the directory owner to [user:group ] hive:hdfs # su - hdfs $ hdfs dfs -chown -R hive:hdfs /tmp/hive Check the ownership now $ hdfs dfs -ls /tmp Desired output The output should match the below snippet drwxrwxrwx - hive hdfs 0 2018-12-12 23:43 /tmp/hive Now if you re-run HQL it should succeed. Please let me know

Shelton · ‎03-23-2020

@kvinod Your issue can be resolved by merging the keytabs in question. Merge keytab files If you have multiple keytab files that need to be in one place, you can merge the keys with the ktutil command. Depending on whether you are using MIT or Heimdal Kerberos the process is different but to merge keytab files using MIT Kerberos, use: In the below example I am merging [mcaf.keytab],[hbase.keytab] and [zk.keytab] into mcafmerged.keytab you can merge n number of keytabs but you must ensure the user executing has the correct permissions, it could be a good idea to copy the keytabs and merge them from the users' home directory $ ktutil ktutil: read_kt mcaf.keytab ktutil: read_kt hbase.keytab ktutil: read_kt zk.keytab ktutil: write_kt ktutil: quit To verify the merge Use $ klist -k mcafmerged.keytab Now to access hbase $ sudo kinit -kt mcafmerged.keytab mcaf@Domain.ORG The keytab file is independent of the computer it's created on, its filename, and its location in the file system. Once it's created, you can rename it, move it to another location on the same compute.

Shelton · ‎03-23-2020

@eistre91 Have you thought of using the HDP Sandbox which is still available for download? This is the ONLY environment whose component versions are a near match to the HDP 3.2.5. It's really unfortunate that everything now is behind a paywall. As a contributor, I have also been hit hard and cannot anymore help on newer versions of the software this is the only occasion I regret the merger.

Shelton · ‎03-23-2020

@berti Good to know it's resolved usually upgrading a cluster doesn't reset the hostnames in /etc/hosts? AWS internal IPs are for AWS inter datacenter communication to expose the hosts you definitely need the public (external) IP's Happy hadooping

Shelton · ‎03-20-2020

@berti The first reflex is to regenerate the principals following the below steps Review and regenerate the Kerberos principals for your cluster: Select Administration > Kerberos . The currently configured Kerberos principals are displayed. If you are running HDFS, the hdfs/hostname and host/hostname principals are listed. If you are running MapReduce, the mapred/hostname and host/hostname principals are listed. The principals for other running services are also listed. Only if necessary, select the principals you want to regenerate. Click Regenerate. I have also noticed a discrepancy in the 2 principals I hope it was not a human error while you are trying to mask your real principals user hdfs/hdfs/datanode-fqdn@REALM (auth:KERBEROS user hdfs/datanode-fqdn@REALM (auth:KERBEROS) This also appears in the logs Connection from 172.17.98.94:27869 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user hdfs/hdfs/datanode-fqdn@REALM (auth:KERBEROS) Connection from 172.17.98.96:39391 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user hdfs/datanode-fqdn@REALM (auth:KERBEROS) Could you check that and revert.

Shelton · ‎03-12-2020

@Joe_Jim By default, the broker binds to localhost. Can you share your server.properties and logs? I am of the opinion you need to update the server.properties listeners=PLAINTEXT://<IP:_TO_CHANGE>:9092

Shelton · ‎03-12-2020

@jking I have a suspicion can you share a tokenized version of your cluster /etc/hosts ? How many hosts do you have in your cluster and is the DNS working correctly?

Online	Offline
Last Visited	‎03-30-2020 02:51 PM

Date Registered	‎01-19-2017 04:35 AM
Last Visited	‎03-30-2020 02:51 PM
Total Messages Posted	3,316
Total Kudos Received	519

Re: Zookeeper not starting

Re: HDFS folder permission denied, but the user is...

Re: Configuring Ambari Metrics service in distribu...

Re: Changing IP and Hostname for Ambari Eco System

Re: Adding a second user on hadoop cluster

Re: Unable to run Hive query through putty

Re: Stuck on Cloudera Manager Cluster Configuratio...

Re: No route to host

Re: How to delete an acl in zookeeper

Re: Unable to run Hive query through putty

Re: How to delete an acl in zookeeper

Re: How to delete an acl in zookeeper

Re: Unable to run Hive query through putty

Re: Unable to run Hive query through putty

Re: Stuck on Cloudera Manager Cluster Configuratio...

Re: How to delete an acl in zookeeper

Re: Yarn jobs are failing after enabling MIT-Kerbe...

Re: Unable to run Hive query through putty

Re: Unable to run Hive query through putty

Re: Stuck on Cloudera Manager Cluster Configuratio...

Re: No route to host

Re: No route to host

Re: Distcp got stuck with the below and doesn’t do...

Re: Yarn jobs are failing after enabling MIT-Kerbe...

Re: Distcp got stuck with the below and doesn’t do...

Re: recover username and password from services in...

Re: Distcp got stuck with the below and doesn’t do...

Re: Get latest file from directory

Re: Unable to run Hive query through putty

Re: Yarn jobs are failing after enabling MIT-Kerbe...

Re: Are There Alternatives to HDP?

Re: Datanodes can't authenticate to Namenode - k...

Re: Datanodes can't authenticate to Namenode - k...

Re: Kafka Producer from remote server

Re: Cloudera Management Service not starting