Re: What are the steps to enable HTTPS for YARN an...

lraheja · ‎07-12-2017

@dhieru singh There is no problem in making Ambari host as your master node. I would like to see the error that you are getting. Can you please upload ambari-server logs to have better idea?

lraheja · ‎07-12-2017

@dhieru singh What is the error you are getting? You can upload ambari-server.log located at /var/log/ambari-server/ambari-server.log to have a look. Also do you have only one hosts in the cluster on multiple? If multiple, have you setup passwordless ssh between hosts or are you doing manual ambari-agent installation?

lraheja · ‎07-11-2017

You need to add this to core-site.xml through Ambari in HDFS configurations

lraheja · ‎07-11-2017

@Rui Ornellas Junior You can configure proxy user using properties hadoop.proxyuser.$superuser.hosts along with either or both of hadoop.proxyuser.$superuser.groups and hadoop.proxyuser.$superuser.users. By specifying as below in core-site.xml, the superuser named root can connect only from host1 and host2 <property> <name>hadoop.proxyuser.root.hosts</name> <value>host1,host2</value> </property>

lraheja · ‎07-11-2017

@Farhad Heybati You can use below command to check the authorization on ATLAS_HOOK topic: Note- Below command is run by kafka user. /usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=<ZK_HOSTNAME>:2181 --list --topic ATLAS_HOOK Below is a useful doc link: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_data-governance/content/ch_hdp_data_governance_install_atlas_ambari.html In Ranger enabled environments, need to create Kafka policies: topic=ATLAS_HOOK permission=publish, create; group=public permission=consume, create; user=atlas (for non-kerberized environments, set group=public) topic=ATLAS_ENTITIES permission=publish, create; user=atlas (for non-kerberized environments, set group=public) permission=consume, create; group=public You can use below command to provide access: command can only run by the kafka user. /usr/hdp/current/kafka-broker/bin/kafka-acls.sh --topic ATLAS_HOOK --allow-principals * --operations All --authorizer-properties "zookeeper.connect=hostname:2181" /usr/hdp/current/kafka-broker/bin/kafka-acls.sh --topic ATLAS_ENTITIES --allow-principals * --operations All --authorizer-properties "zookeeper.connect=hostname:2181"

lraheja · ‎07-11-2017

@Ekantheshwara Basappa Groups and roles mapping using ldapRealm in shiro is not supported as per Zeppelin 0.6.0 version. What is your HDP version? Below is the Apache Jira: https://issues.apache.org/jira/browse/ZEPPELIN-1472 The ldap realm will be changed to 'ldapRealm = org.apache.zeppelin.realm.LdapGroupRealm' and you can use 'ldapRealm.rolesByGroup = hdpeng: admin' for group role mapping.

lraheja · ‎07-11-2017

@A. Karray The unicode string for \xe9 is an accented e - é Your error "UnicodeEncodeError:'ascii' codec can't encode character u'\xe9' in position 155: ordinal not in range(128)" indicates you have special char in your username.

lraheja · ‎07-11-2017

@Abhishek Kumar Yes you would need to enable HTTPS for HDFS as well. Below is the link to follow: https://community.hortonworks.com/articles/52875/enable-https-for-hdfs.html Configuring load balancer is not required.

lraheja · ‎07-11-2017

@Abhishek Kumar Below is a good HCC article link you can follow: https://community.hortonworks.com/articles/52876/enable-https-for-yarn-and-mapreduce2.html

lraheja · ‎07-11-2017

@Krishna Kumar Do you have multiple python version installed in your environment? What python version are you using ? Does this happen with only Spark interpreter or any other interpreter has the same behavior?

lraheja · ‎07-11-2017

@Rahul P you can try below steps: Log into your mysql db: # mysql -u root -p -h localhost #use hive; #drop INDEX PCS_STATS_IDX ON PART_COL_STATS; Restart the metastore.

lraheja · ‎05-04-2017

ISSUE: Spark Job fails with "java.lang.LinkageError: ClassCastException: attempting to castjar:file" because of a conflict between RuntimeDelegate from Jersey in yarn client libs and the copy in spark's assembly jar. ERROR: 17/05/02 17:44:25 ERROR ApplicationMaster: User class threw exception: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.class java.lang.LinkageError: ClassCastException: attempting to castjar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.class at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:116) at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91) at javax.ws.rs.core.MediaType.<clinit>(MediaType.java:44) at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:64) at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182) at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175) at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162) at com.sun.jersey.api.client.Client.init(Client.java:342) at com.sun.jersey.api.client.Client.access$000(Client.java:118) at com.sun.jersey.api.client.Client$1.f(Client.java:191) at com.sun.jersey.api.client.Client$1.f(Client.java:187) at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193) at com.sun.jersey.api.client.Client.<init>(Client.java:187) at com.sun.jersey.api.client.Client.<init>(Client.java:170) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(TimelineClientImpl.java:282) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.hive.ql.hooks.ATSHook.<init>(ATSHook.java:67) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:379) at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:60) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1309) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1293) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1347) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:495) at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:484) at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$withHiveState$1.apply(ClientWrapper.scala:290) at org.apache.spark.sql.hive.client.ClientWrapper.liftedTree1$1(ClientWrapper.scala:237) at org.apache.spark.sql.hive.client.ClientWrapper.retryLocked(ClientWrapper.scala:236) at org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:279) at org.apache.spark.sql.hive.client.ClientWrapper.runHive(ClientWrapper.scala:484) at org.apache.spark.sql.hive.client.ClientWrapper.runSqlHive(ClientWrapper.scala:474) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:624) at org.apache.spark.sql.hive.execution.DropTable.run(commands.scala:89) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56) at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55) at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145) at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130) at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52) at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817) at com.ao.multiLevelLoyalty$.main(multiLevelLoyalty.scala:846) at com.ao.multiLevelLoyalty.main(multiLevelLoyalty.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:559) 17/05/02 17:44:25 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.class) 17/05/02 17:44:25 INFO SparkContext: Invoking stop() from shutdown hook 17/05/02 17:44:25 INFO SparkUI: Stopped Spark web UI at http://10.225.135.102:35023 17/05/02 17:44:25 INFO YarnAllocator: Driver requested a total number of 0 executor(s). 17/05/02 17:44:25 INFO YarnClusterSchedulerBackend: Shutting down all executors 17/05/02 17:44:25 INFO YarnClusterSchedulerBackend: Asking each executor to shut down ROOT CAUSE: This happens because of the conflict between RuntimeDelegate from Jersey in yarn client libs and the copy in spark's assembly jar. At runtime, YARN call into ATS code which needs a different version of a class and cannot find it because the version in Spark and the version in YARN have a conflict. RESOLUTION: Set below property using HiveContext: hc = new org.apache.spark.sql.hive.HiveContext(sc) hc.setConf("yarn.timeline-service.enabled","false")

lraheja · ‎02-03-2017

@Colin Cunningham You can follow below steps: Go to shiro.ini file and edit following section: 1) Under [users] section, you can put username and password you want to use for login : [users] # List of users with their password allowed to access Zeppelin. # To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections admin = password1 maria_dev = maria_dev 2) Under [Url] section make below change: [urls] # anon means the access is anonymous. # authcBasic means Basic Auth Security # To enfore security, comment the line below and uncomment the next one /api/version = anon #/** = anon /** = authc 3) Restart the service.

lraheja · ‎01-26-2017

Livy: Livy is an open source REST interface for interacting with Spark. Authorized users can launch a Spark session and submit code. Two different users can access their own private data and session, and they can collaborate on a notebook. Only the Livy server can submit a job securely to a Spark session. Steps to follow to configure livy interpreter to work with secure HDP cluster: Setup proxy for livy interpreter in core-site.xml Go to Ambari->HDFS->config->customer-core-site and add below properties: hadoop.proxyuser.livy.groups=* hadoop.proxyuser.livy.hosts=* 2. Configure livy interpreter in Zeppelin and add below configurations: livy.superusers=zeppelin-spark Note - The value for livy.superusers should be your zeppelin principal. That would be zeppelin-{$Cluster_name} For example, in this case you can find it by running below command: klist -kt /etc/security/keytabs/zeppelin.server.kerberos.keytab Keytab name: FILE:/etc/security/keytabs/zeppelin.server.kerberos.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 11/15/16 17:33:16 zeppelin-spark@HWX.COM 1 11/15/16 17:33:16 zeppelin-spark@HWX.COM 1 11/15/16 17:33:16 zeppelin-spark@HWX.COM 1 11/15/16 17:33:16 zeppelin-spark@HWX.COM 1 11/15/16 17:33:16 zeppelin-spark@HWX.COM zeppelin-spark will be your superuser for livy interpreter. *Make sure this will match with livy.superusers in livy-conf file. livy.impersonation.enabled=true //this configuration should also be present in livy-conf. livy.server.access_control.enabled=true livy.server.access_control.users=livy,zeppelin livy.server.auth.type=kerberos livy.server.auth.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab livy.server.auth.kerberos.principal=HTTP/spark-1.hwx.com@HWX.COM livy.server.launch.kerberos.keytab=/etc/security/keytabs/livy.service.keytab livy.server.launch.kerberos.principal=livy/spark-1.hwx.com@HWX.COM Note - To configure Zeppelin with authentication for Livy you need to set the following in the interpreter settings: zeppelin.livy.principal=zeppelin-spark@HWX.COM zeppelin.livy.keytab=/etc/security/keytabs/zeppelin.service.keytab 3. Make sure zeppelin.livy.url is pointing to hostname not IP address : zeppelin.livy.url=http://spark-3.hwx.com:8998 4. After saving configuration changes in livy interpreter, Please restart interpreter to see the affect.

lraheja · ‎01-17-2017

@Christian Guegi You can go with manual upgrade of cluster and can upgrade kafka brokers one by one: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-upgrade/content/upgrade-kafka-23.html

lraheja · ‎01-11-2017

@Dezka Dex Can you upload new stack trace?

lraheja · ‎01-11-2017

@Dezka Dex The error you are getting is : Caused by: java.net.SocketException: Connection reset Failed to connect to KDC - Failed to communicate with the Active Directory at LDAP://hq.domain.com/OU=Production,OU=domain,DC=hq,DC=domain,DC=com: simple bind failed: hq.domain.com:389 Above error indicates communication failure with AD, but you mentioned KDC test passes? Can you make sure you are using correct communication string? Can you run ldapsearch with it? Also, Have you followed below doc: https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_Ambari_Security_Guide/content/_configure_ambari_to_use_ldap_server.html Can you upload your krb5.conf?

lraheja · ‎01-11-2017

@elkan li Can you provide the output of below commands from the host you are logging into Hive: cat /etc/group|grep hive cat /etc/group|grep nobody cat /etc/passwd|grep hive cat /etc/passwd|grep nobody Also, please confirm above commands output is same on NodeManager hosts as well.

lraheja · ‎01-11-2017

@Teddy Brewski Ranger does not work with wildcard characters. Unfortunately, you would need to specify each table name separately if you would like to create policy for it.

lraheja · ‎09-07-2016

@Jon Zeolla Can you run 'yum clean all' and then again try '/usr/bin/yum -d 0 -e 0 -y install hdp-select' manually?

lraheja · ‎09-07-2016

@da li Can you please check the following: 1. cd to /etc/yum.repos.d/ and please paste the output of ls -ltr. - the issue could be because of conflicting repo. 2. Also please pate the output of HDP.repo 3. Run 'hdp-select' and paste the output here. 4. Run 'yum clean all' and then 'sudo yum install -y ambari-metrics-collector'

lraheja · ‎09-06-2016

@Savanna Endicott This time you are getting same error but for different property. Previously you got for 'dfs.namenode.https-address' and now you are getting for 'dfs.namenode.http-address'. Please repeat the same step again and this time use http property: /var/lib/ambari-server/resources/scripts/configs.sh -u AMBARI_USER -p AMBARI_PASS set AMBARI_HOST_NAME CLUSTER_NAME hdfs-site dfs.namenode.http-address "abc.xyz.com:50070" Remember the port is 50070 this time for http address.

lraheja · ‎09-05-2016

@Savanna Endicott You can use below command to push the property to the cluster and then try to restart NN /var/lib/ambari-server/resources/scripts/configs.sh -u AMBARI_USER -p AMBARI_PASS set AMBARI_HOST_NAME CLUSTER_NAME PROPERTY_FILE PROPERTY_NAME "VALUE" In your case this would be : /var/lib/ambari-server/resources/scripts/configs.sh -u AMBARI_USER -p AMBARI_PASS set AMBARI_HOST_NAME CLUSTER_NAME hdfs-site dfs.namenode.https-address "abc.xyz.com:50470" Replace value according to your cluster specification.Where: AMBARI_USER - Your Ambari UI login user (Default admin) AMBARI_PASSWORD - Login user's password (Default admin) AMBARI_HOST_NAME - Your Amabri server host CLSUTER_NAME - Your cluster name (Case Sensitive) abc.xyz.com - Namenode hostname.

lraheja · ‎09-05-2016

@Nicolas Steinmetz It would depend on your need. If dfs.replication is 3(default) - which means each block would be replicated to 3 Data Nodes then you would atleast need 3 machines and all should have Data Node on it. You can configure this value of in HDFS and you would need to have atleast those many machine. Usually one go for 5 node cluster - 1 Master Node, 3 Data Nodes and 1 Edge Node (All clients on it). If your replication factor is 2 then you can build up a cluster with 2 Node too.

lraheja · ‎09-02-2016

@Nicolas Steinmetz I just tested your usecase in my environment and below are the components that would be needed before you move forward: 1. HDFS 2. YARN 3 Zookeeper 4. MR 5. Hive 6. Pig Client - You can remove this after the installation is done 7. Slider client - You can remove this after the installation is done 8. Tez Client 9. It will give you a Warning for SmartSense and Ambari Metrics but you can by pass that. 10 . Spark Note - I tested this with HDP 2.5 and Ambari 2.4.0.1 Please find the attached screenshot for reference. untitled.pnguntitled-1.pnguntitled-2.pnguntitled-3.pnguntitled-4.png

lraheja · ‎09-01-2016

@Anuraj Pandey can you please modify the value and change it to below and restart the required services : from etl,prod,reporting,dev,default to default,etl,prod,reporting,dev This should fix your issue. Please let me know how it goes. Also do not forget to restart Hive after you make the changes.

lraheja · ‎09-01-2016

@Anuraj Pandey Can you please upload the value of "hive.server2.tez.default.queues" .

lraheja · ‎09-01-2016

@Nicolas Steinmetz I believe you would need HDFS, MR and Zookeeper in addition to Yarn and Spark. Ambari will not let you move forward without these components

lraheja · ‎08-30-2016

This is a known issue with /usr/bin/hdp-select script. The script does not honor any dir except from "CURRENT" and other HDP stack versions. There is an exception in few HDP version where it can tolerate "Share" and "lost+found" folders. You can remove anything except these folder to another location and try again.

lraheja · ‎08-18-2016

@Kumar Veerappan You can run below command to check if Ambari server is running on the machine or not : ps -ef|grep ambari-server and you will ambari server process there if the server is started. Something like below: root 17395 1 1 Jul27 ? 05:49:42 /usr/jdk64/jdk1.8.0_60/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer

Online	Offline
Last Visited	‎08-15-2019 06:35 AM

Date Registered	‎09-14-2015 02:33 AM
Last Visited	‎08-15-2019 06:35 AM
Total Messages Posted	41
Total Kudos Received	16

Re: What are the steps to enable HTTPS for YARN an...

Re: HBase policy with Ranger

Re: public-repo-1.hortonworks.com 404 for packages

Re: ambari error

Re: NameNode failing after HA NameNode Rollback

Re: I have a ambari server on machine 1 and I am g...

Re: I have a ambari server on machine 1 and I am g...

Re: Unauthorized connection for super-user: root f...

Re: Unauthorized connection for super-user: root f...

Re: Can't import from Hive to Atlas: import-hive.s...

Re: How to configure groups and roles mapping usin...

Re: Zeppelin strange start problem

Re: What are the steps to enable HTTPS for YARN an...

Re: What are the steps to enable HTTPS for YARN an...

Re: Need to restart the interpreter every time bef...

Re: Hive Metastore Start

Spark job failure with "java.lang.LinkageError: Cl...

Re: Sanbox Zeppelin: How to login as maria_dev (no...

How to configure zeppelin livy interpreter for sec...

Re: Kafka Rolling Upgrade

Re: Kerberos setup Ambari 2.4 and HDP 2.5 fails (s...

Re: Kerberos setup Ambari 2.4 and HDP 2.5 fails (s...

Re: Can't not login Hive Service with CLI?

Re: HBase policy with Ranger

Re: public-repo-1.hortonworks.com 404 for packages

Re: ambari error

Re: NameNode failing after HA NameNode Rollback

Re: NameNode failing after HA NameNode Rollback

Re: Is it possible to install only ambari yarn and...

Re: Is it possible to install only ambari yarn and...

Re: hive setting tab disabled in ambari

Re: hive setting tab disabled in ambari

Re: Is it possible to install only ambari yarn and...

Re: Install of HDP fails with "ValueError: invalid...

Re: How to find the node where ambari server runs