About subhash_parise3

subhash_parise3 · ‎07-12-2019

We have a kerberized 4 node cluster. We are not able to call Livy Api from a remote windows machine. Command to make the API call curl --negotiate -u : -X GET -H "Content-Type: application/json" "http://******:8999/sessions?from=1&size=1" Response <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 403 </title> </head> <body> <h2>HTTP ERROR: 403</h2> <p>Problem accessing /sessions. Reason: <pre> java.lang.IllegalArgumentException</pre></p> <hr /><i><small>Powered by Jetty://</small></i> </body> </html> -Same curl command works successfully from any of the cluster machines after getting a kerberos ticket. -Windows machine is not part of the realm. -I am using the same keytab file to get the kerberso ticket on windows machine which I use on cluster machine. -I suspect that the curl command is not sending the correct user to authenticate. Please let us know if why more information is needed. Appreciate any help.

subhash_parise3 · ‎07-08-2019

Dear community members, we were unable to connect spark-history server ui even we have enabled spengo authentication in Advanced spark-env config as following: export SPARK_HISTORY_OPTS='-Dspark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter -Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=kerberos,kerberos.principal=HTTP/ip-0-0-0-0@HDPCLUSTER.LOCAL,kerberos.keytab=/etc/security/keytabs/spnego.service.keytab"' Any help much appreciated to resolve the above issue ? Thanks in advance

subhash_parise3 · ‎06-19-2019

HI @Jay Kumar SenSharma I am able to connect by using below connection properties: custom.postgres.jdbc.name=postgresql-jdbc.jar previous.custom.postgres.jdbc.name=postgresql-jdbc.jar server.jdbc.connection-pool=internal server.jdbc.database=postgres server.jdbc.database_name=ambari server.jdbc.driver=org.postgresql.Driver server.jdbc.hostname=localhost server.jdbc.port=5432 server.jdbc.postgres.schema=ambari server.jdbc.rca.driver=org.postgresql.Driver server.jdbc.rca.url=jdbc:postgresql://*.*.*.*:5432/ambari?ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory server.jdbc.rca.user.name=ambari server.jdbc.rca.user.passwd=${alias=ambari.db.password} server.jdbc.url=jdbc:postgresql://*.*.*.*:5432/ambari?ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory server.jdbc.user.passwd=${alias=ambari.db.password} if i removed the sslfactory=org.postgresql.ssl.NonValidatingFactory parameter form jdbc url, connection is not working?

subhash_parise3 · ‎06-18-2019

Dear Community Members, I am not able to connect external postgres db with ssl enabled from ambari-server . I have changed jdbc url in ambari.properties file to make the ssl connection( Followed https://community.hortonworks.com/questions/209938/how-to-setup-ambari-with-an-external-postgresql-db.html ), but no luck. Any help much appreciated! Thank you in advance:)

subhash_parise3 · ‎05-27-2019

Hi @Predrag Minovic, Thank you for the sharing your input. No issues with the zookeeper name space, have double checked i,e jdbc:hive2://hadoop-zknode01:2181,hadoop-zknode02:2181,hadoop-zknode03:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2

subhash_parise3 · ‎05-23-2019

Team, i want to setup HDP 3.1.0 cluster in Azure. Does HDP support azure postgres-sql database for ambari,hivemetastore,ranger service bakends ?

subhash_parise3 · ‎05-20-2019

Hello All, I have installed HDP3.1 recently and enabled hive llap. i am able to connect hiveserver2 by using hive cli. when i am trying to connect hive from beeline it's throwing NULL Pointer exception. i haven't found errors in hiveserver logs: 2019-05-20 13:28:10 INFO HiveConnection:203 - Will try to open client transport with JDBC Uri: jdbc:hive2://hive.server2.instance.uri=hadoop-master-nn1:10000;hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hadoop-master-nn1;hive.server2.thrift.port=10000;hive.server2.use.SSL=false/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 java.lang.NullPointerException Any suggestion to resolve the above issue ?

subhash_parise3 · ‎05-16-2019

Dear community members, i am not able see the hive table data while running select query , count query showing the count number. no errors were observed in the logs : create table syntax: CREATE TABLE `demandpriority`( `demandtypekey` int, `itemkey` int, `locationkey` int, `demandcostraw` float, `scpdemandbuildlatelimit` float, `demandvcuftraw` float, `spdemandqty` float, `demandvcasesraw` float, `demanddebugflag` float, `demandquantityraw` float, `demandwcasesraw` float, `spdemandpriority` float, `scpdemandbuildaheadlimit` float, `demandretailraw` float, `scpdemandincrementalflag` boolean, `demandwcuftraw` float) PARTITIONED BY ( `versionkey` int, `timekey` date) CLUSTERED BY ( itemkey) INTO 100 BUCKETS ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://88888/warehouse/tablespace/managed/hive/schema_wmt1dept.db/demandpriority' | TBLPROPERTIES ( 'bucketing_version'='2', 'transactional'='true', 'transactional_properties'='default', 'transient_lastDdlTime'='1557938173') Note : i have tried with MSCK REPAIR and ANALYZE commands but still no luck

subhash_parise3 · ‎05-07-2019

please check below article: https://antnix07.blogspot.com/2017/05/normal-0-false-false-false-en-us-x-none.html

subhash_parise3 · ‎04-08-2019

LOAD DATA LOCAL INPATH '/tmp/sample.txt' OVERWRITE INTO TABLE employee; Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: Principal [name=hive, type=USER] does not have following privileges for operation LOAD [ADMIN] (state=42000,code=40000) 0: jdbc:hive2://starbucks-spark-job-test:2181> dfs -ls /user/hive; Error: Error while processing statement: Permission denied: Principal [name=hive, type=USER] does not have following privileges for operation DFS [ADMIN] (state=,code=1) 0: jdbc:hive2://starbucks-spark-job-test:2181> dfs -ls /tmp; Error: Error while processing statement: Permission denied: Principal [name=hive, type=USER] does not have following privileges for operation DFS [ADMIN] (state=,code=1) 0: jdbc:hive2://starbucks-spark-job-test:2181> In hive config i have found below property in Advanced HiveServer2-site hive.conf.restricted.list = hive.security.authenticator.manager,hive.security.authorization.manager,hive.security.metastore.authorization.manager,hive.security.metastore.authenticator.manager,hive.users.in.admin.role,hive.server2.xsrf.filter.enabled,hive.security.authorization.enabled do something we need to change above property ?? Please Advice ?

subhash_parise3 · ‎03-14-2019

Dear members, i am connecting Hive by using Hive_warehouse_connector from spark-shell with spark user. getting below error while creating the database java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=anonymous, access=EXECUTE, inode="/warehouse/tablespace/managed/hive/hive_test.db/":hive:hadoop:drwxdrwx--- Here my question is why spark-shell using anonymous user to contact hive database instead of spark user?

subhash_parise3 · ‎03-05-2019

login zookeeper cli by using /usr/hdp/current/zookeeper-server/bin/zkCli.sh and run ls / The above command will display all znodes in the cluster

subhash_parise3 · ‎02-19-2019

Hi @Dan Hops, It seems you are working with single node cluster and configured LLAP in top of hive with yarn 100% utilization Please configure the LLAP to use 25% of your Yarn capacity

subhash_parise3 · ‎02-13-2019

Hi @Kashif Amir, The other way is automate this task by using shell or python: you can get the table creation by using "desc formatted db.tablename"

subhash_parise3 · ‎02-13-2019

Hi @Kashif Amir, what is the HDP version you are using ? if the version is HDP 2.X please see https://falcon.apache.org/FalconDocumentation.html#Retention

subhash_parise3 · ‎02-13-2019

Hi @Giridharan C, For Syntax please follow : https://github.com/hortonworks-spark/spark-llap/tree/master

subhash_parise3 · ‎02-12-2019

Please go through https://hortonworks.com/wp-content/uploads/2013/04/Hortonworks-Hive-ODBC-Driver-User-Guide.pdf

subhash_parise3 · ‎02-12-2019

setup guide : https://community.hortonworks.com/articles/223626/integrating-apache-hive-with-apache-spark-hive-war.html

subhash_parise3 · ‎02-12-2019

Hi Giridharan, you need to use hive warehouse connector to connect hive databases from HDP3 on words. please see https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/integrating-hive/content/hive_hivewarehouseconnector_for_handling_apache_spark_data.html

subhash_parise3 · ‎02-12-2019

Seems hostname resolution issue. can you try to connect with ip ?

subhash_parise3 · ‎02-08-2019

Hi @Madhura Mhatre, This may not the perfect answer but you can try this way: First check the below process in one data node, if it work's perfectly please replicate it others 1. Create the new config group under one data node for example you have configured /data1(1.5 T.B) datanode 2. overwrite the dfs.datanode.dir config parameter in new config group Remove /data1 and add /data2(1 T.B) /data3 (1 T.B) 3. save the changes and restart required services. 4. blocks are automatically copied from other data nodes since old drive is missing configuration Note: Cluster may get slowness due to heavy data lifting from other datanodes.

subhash_parise3 · ‎01-21-2019

Hi @Dinesh Singh, Check this article https://community.hortonworks.com/articles/9148/troubleshooting-an-oozie-flow.html

subhash_parise3 · ‎01-11-2019

Try the following, lets assume your hive.tez.container.size=2048. set hive.tez.java.opts=-Xmx1640m (0.8 times hive.tez.container.size) set tez.runtime.io.sort.mb=820 (0.4 times hive.tez.container.size) set tez.runtime.unordered.output.buffer.size-mb=205 (0.1 times hive.tez.container.size)

subhash_parise3 · ‎01-10-2019

Hi @Shobhna Dhami, Can you please check any OutOfMemoryError errors in the log file. please check /var/log/messages log file also. If it is OOM you need to upgrade the memory in that server

subhash_parise3 · ‎01-10-2019

Hi Vikash, This https://community.hortonworks.com/questions/91265/oozie-hive-action-class-not-found-exception.html help you.

subhash_parise3 · ‎01-10-2019

Hi @Naeem Ullah Khan, it's looks like an issue with proxy-server. if you are using proxy-server please add in /etc/yum.conf

subhash_parise3 · ‎01-08-2019

Hi @chaouki trabelsi, Please find the below example: Test-NetConnection 10.0.10.133 -Port 8080 ComputerName : 10.0.10.133 RemoteAddress : 10.0.10.133 RemotePort : 8080 InterfaceAlias : Ethernet 2 SourceAddress : 10.242.2.27 TcpTestSucceeded : True TcpTestSucceeded should be true. if it false you need to open the ports in firewalls.

subhash_parise3 · ‎01-07-2019

Hi @chaouki trabelsi, please check the firewall in windows server to accept the connections from hive you can simple check by using below syntax from powershell. Test-NetConnection hiveserver-ip -Port 10000 if the connection status false then you need to check the firewall in bothsides (Linux+windows)

subhash_parise3 · ‎01-07-2019

Hi @Rajesh Sampath, please follow the below article : https://github.com/hortonworks-spark/spark-llap/tree/master

subhash_parise3 · ‎01-07-2019

Hi Vinay, use the below code to connect hive and list the databases : spark-shell --conf spark.sql.hive.hiveserver2.jdbc.url="jdbc:hive2://hiveserverip:10000/" spark.datasource.hive.warehouse.load.staging.dir="/tmp" spark.hadoop.hive.zookeeper.quorum="zookeeperquoremip:2181" --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar val hive = com.hortonworks.spark.sql.hive.llap.HiveWarehouseBuilder.session(spark).build() hive.showDatabases().show(100, false) Reference article https://github.com/hortonworks-spark/spark-llap/tree/master

Online	Online
Last Visited	‎08-19-2019 11:08 PM

Date Registered	‎12-30-2015 09:04 AM
Last Visited	‎08-19-2019 11:08 PM
Total Messages Posted	161
Total Kudos Received	28

Re: Spark unable to connect Hive database in HDP 3...

Re: ambari cluster + zookeeper snapshot

Re: hdfs dfs -ls command failing with below error ...

Re: install hadoop services are failing due to loc...

Re: LLAP - Status Unknown ( HiveInteractive server...

Issue in making Livy REST Api call in a kerberized...

unable to connect spark history server ui after ke...

Re: ambari is unable to make the connection extern...

ambari is unable to make the connection external p...

Re: unable to login hive by using beeline with zoo...

does HDP3.1.0 supports Azure postgres-sql database...

unable to login hive by using beeline with zookeep...

Hive table showing the count and not displaying th...

Re: This DataNode is not connected to one or more ...

HIve load data local inpath query failing with the...

Creating table by using spark-shell failing from ...

Re: where kafka store the meta data on zookeeper ?...

Re: Hive/Druid (llap) query failing - Queue root.d...

Re: HIve Table - Setup Retention

Re: HIve Table - Setup Retention

Re: In HDP3, Not able to create Hive DB/Table usin...

Re: How to connect Hortonworks Hive ODBC drver

Re: In HDP3, Not able to create Hive DB/Table usin...

Re: In HDP3, Not able to create Hive DB/Table usin...

Re: Hive connection with ODBC Driver

Re: EBS volume resize to smaller size, is there an...

Re: Oozie Workflow got hung while executing Hive "...

Re: Vertex did not succeed due to OWN_TASK_FAILURE

Re: Hbase master stops running after few seconds.

Re: Oozie hive action error - hiveComatibilityMain...

Re: Unable to install HDP 3.1.0.3 with Ambari on o...

Re: how to connect from local application to dista...

Re: how to connect from local application to dista...

Re: hive.executeQuery("select * from rh_demand_pla...

Re: Spark unable to connect Hive database in HDP 3...