About lraheja

lraheja · ‎07-12-2017

@dhieru singh There is no problem in making Ambari host as your master node. I would like to see the error that you are getting. Can you please upload ambari-server logs to have better idea?

lraheja · ‎07-12-2017

@Sia As Rajkumar mentioned, You can install 2 separate clusters with different OS . For example Cluster 'test' with hosts 'test1', 'test2','test3' and another cluster 'dev' with hosts 'dev1','dev2','dev3'. However, the OS should be same in 'test1', 'test2','test3' for example Centos and same in 'dev1','dev2','dev3' for example SUSE.

lraheja · ‎07-12-2017

@dhieru singh What is the error you are getting? You can upload ambari-server.log located at /var/log/ambari-server/ambari-server.log to have a look. Also do you have only one hosts in the cluster on multiple? If multiple, have you setup passwordless ssh between hosts or are you doing manual ambari-agent installation?

lraheja · ‎07-11-2017

You need to add this to core-site.xml through Ambari in HDFS configurations

lraheja · ‎07-11-2017

@Rui Ornellas Junior You can configure proxy user using properties hadoop.proxyuser.$superuser.hosts along with either or both of hadoop.proxyuser.$superuser.groups and hadoop.proxyuser.$superuser.users. By specifying as below in core-site.xml, the superuser named root can connect only from host1 and host2 <property> <name>hadoop.proxyuser.root.hosts</name> <value>host1,host2</value> </property>

lraheja · ‎07-11-2017

@Farhad Heybati You can use below command to check the authorization on ATLAS_HOOK topic: Note- Below command is run by kafka user. /usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=<ZK_HOSTNAME>:2181 --list --topic ATLAS_HOOK Below is a useful doc link: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_data-governance/content/ch_hdp_data_governance_install_atlas_ambari.html In Ranger enabled environments, need to create Kafka policies: topic=ATLAS_HOOK permission=publish, create; group=public permission=consume, create; user=atlas (for non-kerberized environments, set group=public) topic=ATLAS_ENTITIES permission=publish, create; user=atlas (for non-kerberized environments, set group=public) permission=consume, create; group=public You can use below command to provide access: command can only run by the kafka user. /usr/hdp/current/kafka-broker/bin/kafka-acls.sh --topic ATLAS_HOOK --allow-principals * --operations All --authorizer-properties "zookeeper.connect=hostname:2181" /usr/hdp/current/kafka-broker/bin/kafka-acls.sh --topic ATLAS_ENTITIES --allow-principals * --operations All --authorizer-properties "zookeeper.connect=hostname:2181"

lraheja · ‎07-11-2017

@Ekantheshwara Basappa Groups and roles mapping using ldapRealm in shiro is not supported as per Zeppelin 0.6.0 version. What is your HDP version? Below is the Apache Jira: https://issues.apache.org/jira/browse/ZEPPELIN-1472 The ldap realm will be changed to 'ldapRealm = org.apache.zeppelin.realm.LdapGroupRealm' and you can use 'ldapRealm.rolesByGroup = hdpeng: admin' for group role mapping.

lraheja · ‎07-11-2017

@A. Karray The unicode string for \xe9 is an accented e - é Your error "UnicodeEncodeError:'ascii' codec can't encode character u'\xe9' in position 155: ordinal not in range(128)" indicates you have special char in your username.

lraheja · ‎07-11-2017

@Abhishek Kumar Yes you would need to enable HTTPS for HDFS as well. Below is the link to follow: https://community.hortonworks.com/articles/52875/enable-https-for-hdfs.html Configuring load balancer is not required.

lraheja · ‎07-11-2017

@Abhishek Kumar Below is a good HCC article link you can follow: https://community.hortonworks.com/articles/52876/enable-https-for-yarn-and-mapreduce2.html

lraheja · ‎07-11-2017

@Krishna Kumar Do you have multiple python version installed in your environment? What python version are you using ? Does this happen with only Spark interpreter or any other interpreter has the same behavior?

lraheja · ‎07-11-2017

@Rahul P you can try below steps: Log into your mysql db: # mysql -u root -p -h localhost #use hive; #drop INDEX PCS_STATS_IDX ON PART_COL_STATS; Restart the metastore.

lraheja · ‎05-04-2017

ISSUE: Spark Job fails with "java.lang.LinkageError: ClassCastException: attempting to castjar:file" because of a conflict between RuntimeDelegate from Jersey in yarn client libs and the copy in spark's assembly jar. ERROR: 17/05/02 17:44:25 ERROR ApplicationMaster: User class threw exception: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.class java.lang.LinkageError: ClassCastException: attempting to castjar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.class at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:116) at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91) at javax.ws.rs.core.MediaType.<clinit>(MediaType.java:44) at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:64) at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182) at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175) at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162) at com.sun.jersey.api.client.Client.init(Client.java:342) at com.sun.jersey.api.client.Client.access$000(Client.java:118) at com.sun.jersey.api.client.Client$1.f(Client.java:191) at com.sun.jersey.api.client.Client$1.f(Client.java:187) at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193) at com.sun.jersey.api.client.Client.<init>(Client.java:187) at com.sun.jersey.api.client.Client.<init>(Client.java:170) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(TimelineClientImpl.java:282) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.hive.ql.hooks.ATSHook.<init>(ATSHook.java:67) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:379) at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:60) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1309) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1293) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1347) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:495) at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:484) at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$withHiveState$1.apply(ClientWrapper.scala:290) at org.apache.spark.sql.hive.client.ClientWrapper.liftedTree1$1(ClientWrapper.scala:237) at org.apache.spark.sql.hive.client.ClientWrapper.retryLocked(ClientWrapper.scala:236) at org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:279) at org.apache.spark.sql.hive.client.ClientWrapper.runHive(ClientWrapper.scala:484) at org.apache.spark.sql.hive.client.ClientWrapper.runSqlHive(ClientWrapper.scala:474) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:624) at org.apache.spark.sql.hive.execution.DropTable.run(commands.scala:89) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56) at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55) at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145) at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130) at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52) at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817) at com.ao.multiLevelLoyalty$.main(multiLevelLoyalty.scala:846) at com.ao.multiLevelLoyalty.main(multiLevelLoyalty.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:559) 17/05/02 17:44:25 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.class) 17/05/02 17:44:25 INFO SparkContext: Invoking stop() from shutdown hook 17/05/02 17:44:25 INFO SparkUI: Stopped Spark web UI at http://10.225.135.102:35023 17/05/02 17:44:25 INFO YarnAllocator: Driver requested a total number of 0 executor(s). 17/05/02 17:44:25 INFO YarnClusterSchedulerBackend: Shutting down all executors 17/05/02 17:44:25 INFO YarnClusterSchedulerBackend: Asking each executor to shut down ROOT CAUSE: This happens because of the conflict between RuntimeDelegate from Jersey in yarn client libs and the copy in spark's assembly jar. At runtime, YARN call into ATS code which needs a different version of a class and cannot find it because the version in Spark and the version in YARN have a conflict. RESOLUTION: Set below property using HiveContext: hc = new org.apache.spark.sql.hive.HiveContext(sc) hc.setConf("yarn.timeline-service.enabled","false")

lraheja · ‎02-03-2017

@Colin Cunningham You can follow below steps: Go to shiro.ini file and edit following section: 1) Under [users] section, you can put username and password you want to use for login : [users] # List of users with their password allowed to access Zeppelin. # To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections admin = password1 maria_dev = maria_dev 2) Under [Url] section make below change: [urls] # anon means the access is anonymous. # authcBasic means Basic Auth Security # To enfore security, comment the line below and uncomment the next one /api/version = anon #/** = anon /** = authc 3) Restart the service.

lraheja · ‎01-26-2017

Livy: Livy is an open source REST interface for interacting with Spark. Authorized users can launch a Spark session and submit code. Two different users can access their own private data and session, and they can collaborate on a notebook. Only the Livy server can submit a job securely to a Spark session. Steps to follow to configure livy interpreter to work with secure HDP cluster: Setup proxy for livy interpreter in core-site.xml Go to Ambari->HDFS->config->customer-core-site and add below properties: hadoop.proxyuser.livy.groups=* hadoop.proxyuser.livy.hosts=* 2. Configure livy interpreter in Zeppelin and add below configurations: livy.superusers=zeppelin-spark Note - The value for livy.superusers should be your zeppelin principal. That would be zeppelin-{$Cluster_name} For example, in this case you can find it by running below command: klist -kt /etc/security/keytabs/zeppelin.server.kerberos.keytab Keytab name: FILE:/etc/security/keytabs/zeppelin.server.kerberos.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 11/15/16 17:33:16 zeppelin-spark@HWX.COM 1 11/15/16 17:33:16 zeppelin-spark@HWX.COM 1 11/15/16 17:33:16 zeppelin-spark@HWX.COM 1 11/15/16 17:33:16 zeppelin-spark@HWX.COM 1 11/15/16 17:33:16 zeppelin-spark@HWX.COM zeppelin-spark will be your superuser for livy interpreter. *Make sure this will match with livy.superusers in livy-conf file. livy.impersonation.enabled=true //this configuration should also be present in livy-conf. livy.server.access_control.enabled=true livy.server.access_control.users=livy,zeppelin livy.server.auth.type=kerberos livy.server.auth.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab livy.server.auth.kerberos.principal=HTTP/spark-1.hwx.com@HWX.COM livy.server.launch.kerberos.keytab=/etc/security/keytabs/livy.service.keytab livy.server.launch.kerberos.principal=livy/spark-1.hwx.com@HWX.COM Note - To configure Zeppelin with authentication for Livy you need to set the following in the interpreter settings: zeppelin.livy.principal=zeppelin-spark@HWX.COM zeppelin.livy.keytab=/etc/security/keytabs/zeppelin.service.keytab 3. Make sure zeppelin.livy.url is pointing to hostname not IP address : zeppelin.livy.url=http://spark-3.hwx.com:8998 4. After saving configuration changes in livy interpreter, Please restart interpreter to see the affect.

lraheja · ‎01-17-2017

@Christian Guegi You can go with manual upgrade of cluster and can upgrade kafka brokers one by one: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-upgrade/content/upgrade-kafka-23.html

lraheja · ‎01-16-2017

@Sachin Ambardekar Can you run "select User,Host,Passwordfrom user where user ='root';" when you log in after using "mysql -u root -p" ?

lraheja · ‎01-16-2017

@Sachin Ambardekar 1. Login to MySQL as root or admin account. From the command line run: #mysql -u root 2. Check user table to see if user exists in MySQL. While in MySQL shell run: mysql> use mysql; mysql> select User,Host,Password from user where user ='root'; 3. make sure 'root'@'datanode3.localdomain' combination exists in your DB. - To create user run following commands: create user 'root'@'datanode3.localdomain' identified by 'hadoop'; grant all privileges on *.* to 'root'@'datanode3.localdomain'; flush privileges; - If user exist and you need to add grants from the command line run: grant all privileges on *.* to 'root'@'datanode3.localdomain'; flush privileges;

lraheja · ‎01-12-2017

@shyam gurram You can try below example: 1.Inside spark-shell, Paste in :paste mode val df = spark .read .option("url", "jdbc:hive2://localhost:10000") .option("dbtable", "people") .format("jdbc") .load 2. Connect to Spark Thrift Server at localhost on port 10000 3. Use people table. It assumes that people table is available.

lraheja · ‎01-11-2017

@Dezka Dex Can you upload new stack trace?

lraheja · ‎01-11-2017

@Dezka Dex The error you are getting is : Caused by: java.net.SocketException: Connection reset Failed to connect to KDC - Failed to communicate with the Active Directory at LDAP://hq.domain.com/OU=Production,OU=domain,DC=hq,DC=domain,DC=com: simple bind failed: hq.domain.com:389 Above error indicates communication failure with AD, but you mentioned KDC test passes? Can you make sure you are using correct communication string? Can you run ldapsearch with it? Also, Have you followed below doc: https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_Ambari_Security_Guide/content/_configure_ambari_to_use_ldap_server.html Can you upload your krb5.conf?

lraheja · ‎01-11-2017

@elkan li Can you provide the output of below commands from the host you are logging into Hive: cat /etc/group|grep hive cat /etc/group|grep nobody cat /etc/passwd|grep hive cat /etc/passwd|grep nobody Also, please confirm above commands output is same on NodeManager hosts as well.

lraheja · ‎01-11-2017

@Teddy Brewski Ranger does not work with wildcard characters. Unfortunately, you would need to specify each table name separately if you would like to create policy for it.

lraheja · ‎01-11-2017

@daniel gerald orbegoso barrantes What is the value of "zookeeper.znode.parent" set to ?

lraheja · ‎01-10-2017

@pradeep arumalla SqlContext and HiveContext are not going away. Spark is big on backward compatibility. So these API’s are still available. Though from documentation it’s clear that they want developers to use SparkSession going forward.

lraheja · ‎09-15-2016

@Sami Ahmad You can use below documentation for kafka install on HDP 2.4: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_installing_manually_book/content/ch_installing_kafka_chapter.html

lraheja · ‎09-14-2016

@Sami Ahmad May I know what doc are you following?

lraheja · ‎09-14-2016

@Jaspreet Singh Can you please provide below info: 1. ls -ltr /usr/hdp/ 2. ls -ltr /usr/hdp/current/ 3. yum info spark-client 4. Run below command manually on host and upload the output: 4.1 yum clean all 4.2 /usr/bin/yum -d 0 -e 0 -y install 'spark_2_2_*' Please let me know how it goes.

lraheja · ‎09-07-2016

@Jon Zeolla Can you run 'yum clean all' and then again try '/usr/bin/yum -d 0 -e 0 -y install hdp-select' manually?

lraheja · ‎09-07-2016

@da li Can you please check the following: 1. cd to /etc/yum.repos.d/ and please paste the output of ls -ltr. - the issue could be because of conflicting repo. 2. Also please pate the output of HDP.repo 3. Run 'hdp-select' and paste the output here. 4. Run 'yum clean all' and then 'sudo yum install -y ambari-metrics-collector'

Online	Offline
Last Visited	‎08-15-2019 06:35 AM

Date Registered	‎09-14-2015 02:33 AM
Last Visited	‎08-15-2019 06:35 AM
Total Messages Posted	41
Total Kudos Received	16

Re: What are the steps to enable HTTPS for YARN an...

Re: HBase policy with Ranger

Re: public-repo-1.hortonworks.com 404 for packages

Re: ambari error

Re: NameNode failing after HA NameNode Rollback

Re: I have a ambari server on machine 1 and I am g...

Re: Hello, I would like to ask if possible to impl...

Re: I have a ambari server on machine 1 and I am g...

Re: Unauthorized connection for super-user: root f...

Re: Unauthorized connection for super-user: root f...

Re: Can't import from Hive to Atlas: import-hive.s...

Re: How to configure groups and roles mapping usin...

Re: Zeppelin strange start problem

Re: What are the steps to enable HTTPS for YARN an...

Re: What are the steps to enable HTTPS for YARN an...

Re: Need to restart the interpreter every time bef...

Re: Hive Metastore Start

Spark job failure with "java.lang.LinkageError: Cl...

Re: Sanbox Zeppelin: How to login as maria_dev (no...

How to configure zeppelin livy interpreter for sec...

Re: Kafka Rolling Upgrade

Re: Ranger configuration: ERROR: Unable to connect...

Re: Ranger configuration: ERROR: Unable to connect...

Re: How to run Spark Sql on Spark Thrift Server

Re: Kerberos setup Ambari 2.4 and HDP 2.5 fails (s...

Re: Kerberos setup Ambari 2.4 and HDP 2.5 fails (s...

Re: Can't not login Hive Service with CLI?

Re: HBase policy with Ranger

Re: Not able to reach HDP 2.5 from host

Re: spark backward compatibility 1.6 vs 2.0

Re: adding Kafka broker issues

Re: adding Kafka broker issues

Re: Need help with Ambari

Re: public-repo-1.hortonworks.com 404 for packages

Re: ambari error