About aliyesami

aliyesami · ‎05-28-2019

so there will be two metastores? how can I "connect it to my cluster" using hive-site.xml ?

aliyesami · ‎05-28-2019

with HDP 3.0 it install hive 3.1 but i want hive 2.3 , is it possible to replace hive3.1 with hive2.3 ? if yes how ?

aliyesami · ‎05-22-2019

In installing hive if I use the driver that came with HDP the DB connection test fails , but if I use the older driver I have(don't know the version) the check succeeds . [root@hadoop1 resources]# pwd /var/lib/ambari-server/resources [root@hadoop1 resources]# ls -al mysql-connector-java.jar lrwxrwxrwx. 1 root root 64 May 22 14:56 mysql-connector-java.jar -> /var/lib/ambari-server/resources/mysql-connector-java-8.0.15.jar [root@hadoop1 resources]#

aliyesami · ‎05-22-2019

Geoffrey how can I find you on LinkedIn?

aliyesami · ‎05-22-2019

can someone please help? I reinstalled both hive and spark2 but still its complaining

aliyesami · ‎05-22-2019

hiveserver2 is started and following properties are set hadoop.proxyuser.hdfs.groups * hadoop.proxyuser.hdfs.hosts * hadoop.proxyuser.hive.groups * hadoop.proxyuser.hive.hosts * hadoop.proxyuser.livy.groups * hadoop.proxyuser.livy.hosts * hadoop.proxyuser.root.groups * hadoop.proxyuser.root.hosts *

aliyesami · ‎05-22-2019

I just installed HDP 3.1 but hive server is showing the following alert Connection failed on host hadoop2.tolls.dot.state.fl.us:10000 (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/alerts/alert_hive_thrift_port.py", line 204, in execute ldap_password=ldap_password) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/hive_check.py", line 84, in check_thrift_port_sasl timeout_kill_strategy=TerminateStrategy.KILL_PROCESS_TREE, File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run returns=self.resource.returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call raise ExecutionFailed(err_msg, code, out, err) ExecutionFailed: Execution of 'beeline -n hive -u 'jdbc:hive2://hadoop2.tolls.dot.state.fl.us:10000/;transportMode=binary' -e ';' 2>&1 | awk '{print}' | grep -i -e 'Connected to:' -e 'Transaction isolation:'' returned 1. )

aliyesami · ‎05-10-2019

: FINISHED 2019-05-10 14:23:13,482 [232a3e6e-06be-9256-92ee-bf27f13b643c:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id 232a3e6e-06be-9256-92ee-bf27f13b643c issued by anonymous: show tables 2019-05-10 14:23:13,842 [232a3e6e-06be-9256-92ee-bf27f13b643c:frag:0:0] WARN o.a.d.e.s.h.c.TableNameCacheLoader - Failure while attempting to get hive tables. Retries once. org.apache.hadoop.hive.metastore.api.MetaException: Got exception: org.apache.thrift.TApplicationException Invalid method name: 'get_tables_by_type' at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1382) ~[drill-hive-exec-shaded-1.16.0.jar:1.16.0] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:1405) ~[drill-hive-exec-shaded-1.16.0.jar:1.16.0] at org.apache.drill.exec.store.hive.client.TableNameCacheLoader.load(TableNameCacheLoader.java:59) [drill-storage-hive-core-1.16.0.jar:1.16.0] at org.apache.drill.exec.store.hive.client.TableNameCacheLoader.load(TableNameCacheLoader.java:41) [drill-storage-hive-core-1.16.0.jar:1.16.0] at org.apache.drill.shaded.guava.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) [drill-shaded-guava-23.0.ja

aliyesami · ‎05-10-2019

. org.apache.hadoop.hive.metastore.api.MetaException: Got exception: org.apache.thrift.TApplicationException Invalid method name: 'get_tables_by_type' at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1382) ~[drill-hive-exec-shaded-1.16.0.jar:1.16.0] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:1405) ~[drill-hive-exec-shaded-1.16.0.jar:1.16.0]

aliyesami · ‎05-08-2019

can someone please reply ?

aliyesami · ‎05-01-2019

I am trying to install apache drill 1.6 with HDP2.6 using the following GitHub rep but its not giving me any hosts to choose in "Assign slave" window, and when I click next it fails. is there a latest GitHub repository then the one below ? https://github.com/dvergari/ambari-drill-service

aliyesami · ‎02-28-2019

I am loading a phoenix table but in hbase this table row key is shown as hex values. how can I correct this to see the ASCII values ? that is the integer value sqlline version 1.1.8 0: jdbc:phoenix:localhost:2181:/hbase-unsecur> SELECT * FROM EXAMPLE; +--------+-------------+------------+ | MY_PK | FIRST_NAME | LAST_NAME | +--------+-------------+------------+ | 12345 | John | Doe | | 67890 | Mary | Jane | +--------+-------------+------------+ 2 rows selected (0.053 seconds) Version 1.1.2.2.6.5.0-292, r897822d4dd5956ca186974c10382e9094683fa29, Fri May 11 08:01:42 UTC 2018 hbase(main):001:0> scan 'EXAMPLE'ROW COLUMN+CELL \x80\x00\x00\x00\x00\x0009 column=M:FIRST_NAME, timestamp=1551381906272, value=John \x80\x00\x00\x00\x00\x0009 column=M:LAST_NAME, timestamp=1551381906272, value=Doe \x80\x00\x00\x00\x00\x0009 column=M:_0, timestamp=1551381906272, value=x \x80\x00\x00\x00\x00\x01\x092 column=M:FIRST_NAME, timestamp=1551381906272, value=Mary \x80\x00\x00\x00\x00\x01\x092 column=M:LAST_NAME, timestamp=1551381906272, value=Jane \x80\x00\x00\x00\x00\x01\x092 column=M:_0, timestamp=1551381906272, value=x2 row(s) in 0.2030 seconds

aliyesami · ‎02-26-2019

so if the hbase-site.xml directory is in the classpath then no connection information needs to be given in the java code?

aliyesami · ‎02-26-2019

I am not specifying anything about the hbase server or the zookeeper but this code works fine and hbase table gets created , how does the code know where to connect to hbase server? [root@hadoop1 ~]# more TestHbaseTable.java import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.client.HBaseAdmin; public class TestHbaseTable { public static void main(String[] args) throws IOException { HBaseConfiguration hconfig = new HBaseConfiguration(new Configuration()); HTableDescriptor htable = new HTableDescriptor("User"); htable.addFamily( new HColumnDescriptor("Id")); htable.addFamily( new HColumnDescriptor("Name")); System.out.println( "Connecting..." ); HBaseAdmin hbase_admin = new HBaseAdmin( hconfig ); System.out.println( "Creating Table..." ); hbase_admin.createTable( htable ); System.out.println("Done!"); } }

aliyesami · ‎02-25-2019

I have Hadoop classpath defined but when I use it as per documentation I am getting error [root@hadoop1 conf]# hadoop classpath /usr/hdp/2.6.5.0-292/hadoop/conf:/usr/hdp/2.6.5.0-292/hadoop/lib/*:/usr/hdp/2.6.5.0-292/hadoop/.//*:/usr/hdp/2.6.5.0-292/hadoop-hdfs/./:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/*:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//*:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/*:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//*:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/*:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//*:/usr/jdk64/jdk1.8.0_112/lib/tools.jar:mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:ojdbc6.jar:/usr/hdp/2.6.5.0-292/tez/*:/usr/hdp/2.6.5.0-292/tez/lib/*:/usr/hdp/2.6.5.0-292/tez/conf [root@hadoop1 conf]# [root@hadoop1 conf]# [root@hadoop1 conf]# echo $JAVA_HOME /usr/jdk64/jdk1.8.0_112 [root@hadoop1 conf]# [root@hadoop1 conf]# javac -cp `hadoop classpath` TestHbaseTable.java TestHbaseTable.java:3: error: package org.apache.hadoop.hbase does not exist import org.apache.hadoop.hbase.HBaseConfiguration; ^ TestHbaseTable.java:4: error: package org.apache.hadoop.hbase does not exist import org.apache.hadoop.hbase.HColumnDescriptor; ^ TestHbaseTable.java:5: error: package org.apache.hadoop.hbase does not exist import org.apache.hadoop.hbase.HTableDescriptor; ^ TestHbaseTable.java:6: error: package org.apache.hadoop.hbase.client does not exist import org.apache.hadoop.hbase.client.HBaseAdmin; ^ TestHbaseTable.java:12: error: cannot find symbol HBaseConfiguration hconfig = new HBaseConfiguration(new Configuration()); ^ symbol: class HBaseConfiguration location: class TestHbaseTable TestHbaseTable.java:12: error: cannot find symbol HBaseConfiguration hconfig = new HBaseConfiguration(new Configuration()); ^

aliyesami · ‎02-25-2019

I am using HDP2.6, hbase 1.1.2 python 3.3.6 [root@hadoop1 conf]# /usr/hdp/current/hbase-master/bin/hbase-daemon.sh start thrift -p 9090 --infoport 9091starting thrift, logging to /var/log/hbase/hbase-root-thrift-hadoop1.out >>> import happybase >>> c = happybase.Connection('hadoop1',9090, autoconnect=False) >>> c.open() >>> print(c.tables()) Traceback (most recent call last):File "<stdin>", line 1, in <module>File "/usr/local/lib/python3.6/site-packages/happybase/connection.py", line 242, in tablesnames = self.client.getTableNames()File "/usr/local/lib/python3.6/site-packages/thriftpy/thrift.py", line 198, in _reqreturn self._recv(_api)File "/usr/local/lib/python3.6/site-packages/thriftpy/thrift.py", line 210, in _recvfname, mtype, rseqid = self._iprot.read_message_begin()File "thriftpy/protocol/cybin/cybin.pyx", line 439, in cybin.TCyBinaryProtocol.read_message_begin (thriftpy/protocol/cybin/cybin.c:6470)cybin.ProtocolError: No protocol version header>>>

aliyesami · ‎02-25-2019

and can "scan" command be called from a java code ?

aliyesami · ‎02-25-2019

hi Josh how can I do it via 'get' command ? since we will be using java to get these rows

aliyesami · ‎02-25-2019

I can select a particular version from a table , but how can I select a range of versions ? e.g say I want to select all the rows from timestamp 1551108073557 to 1551108018724 ? hbase(main):005:0> scan 'PERSON',{NAME => 'student',VERSIONS => 5} ROW COLUMN+CELL r1 column=student:student_name, timestamp=1551108086986, value=John r1 column=student:student_name, timestamp=1551108073557, value=Mary r1 column=student:student_name, timestamp=1551108037609, value=Nancy r1 column=student:student_name, timestamp=1551108018724, value=Ram r1 column=student:student_name, timestamp=1551108002231, value=Sam 1 row(s) in 0.0190 seconds hbase(main):006:0> hbase(main):007:0* hbase(main):008:0* get 'PERSON','r1',{COLUMN => 'student', TIMESTAMP => 1551108018724} COLUMN CELL student:student_name timestamp=1551108018724, value=Ram

aliyesami · ‎02-20-2019

can someone answer this question ?

aliyesami · ‎02-18-2019

please see the attached screen shots of a job running in our 7 data node cluster . Looking from the job do you see anything that we can tune or doesn't seem right ? the jobs takes a long time to run even though we have a powerful cluster . TEZ.JPG

aliyesami · ‎12-07-2018

can someone please advise ?

aliyesami · ‎11-30-2018

how can we bring updates into hive without loading the complete table and checking for the differences ?the incremental append will only look for a date column for newly added records and bring those records only.we can always create a trigger on the source table to create an additional record on any updates but whats the solution without having to create a trigger ? e.g lets say we have a table TEST (ID int , NAME string, LOGIN_TIME timesteamp) we do incremental appends using LOGIN_TIME colum so all new records will get moved to hive but what if someone modifies the NAME column ? how will sqoop pick it up ?

aliyesami · ‎11-27-2018

after starting the ldap server for hadoop i am getting different error now, also i had to replace the clustername "ftehdp" with "default" ? curl -i -k -u admin:admin-password -X GET 'https://localhost:8443/gateway/default/webhdfs/v1/tmp/uname.txt?op=OPEN' HTTP/1.1 307 Temporary Redirect Date: Tue, 27 Nov 2018 16:21:44 GMT Set-Cookie: JSESSIONID=1219u2f8zreb11eu9fuxlggxhq;Path=/gateway/default;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Mon, 26-Nov-2018 16:21:44 GMT Cache-Control: no-cache Expires: Tue, 27 Nov 2018 16:21:44 GMT Date: Tue, 27 Nov 2018 16:21:44 GMT Pragma: no-cache Expires: Tue, 27 Nov 2018 16:21:44 GMT Date: Tue, 27 Nov 2018 16:21:44 GMT Pragma: no-cache X-FRAME-OPTIONS: SAMEORIGIN Location: https://hadoop1:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/tmp/uname.txt?_=AAAACAAAABAAAACgLvtILkFAljr5PIP7MVSOAump8j0kSwFCPdGCP2R_b1tCZ0V2KGOQuiRiI4_IU7GDG6NqRtK2Vu7DOZeOhbuQUaP1FYtD_-IV3P-VXMbOFbPfbwpNseAuN-RyQduRm5S1mrk0GVbYKQg4NscgsoF0GGsvqKDyPtECwhwkX96E37Jc5_yCnlkw3LVKUY41Hg6LOt96W8-3rTmnrbo7o26dOcpPv1_uv4Q1F18b4yk5N5BNf6HTZdVZ6Q Content-Type: application/octet-stream Server: Jetty(6.1.26.hwx) Content-Length: 0

aliyesami · ‎11-27-2018

I am trying to create a file in hdfs via the knox command but i am getting error : $ curl -i -k -u admin:adminxxxx -X PUT 'https://localhost:8443/gateway/ftehdp/namenode/api/v1/tmp/README?op=CREATE' HTTP/1.1 404 Not Found Date: Tue, 27 Nov 2018 15:33:47 GMT Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html; charset=ISO-8859-1 Content-Length: 319 Server: Jetty(9.2.15.v20160210) <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /gateway/ftehdp/namenode/api/v1/tmp/README. Reason: <pre> Not Found</pre></p> <hr /><i><small>Powered by Jetty://</small></i> </body> </html>

aliyesami · ‎10-19-2018

package com.demo; import org.apache.hadoop.hive.ql.exec.UDF; import java.sql.*; import oracle.jdbc.*; import org.apache.hadoop.io.Text; public class queryp extends UDF { public Text evaluate(final Text s) throws SQLException { //----- below code works fine as a seperate class till EOF ------------ System.setProperty("oracle.net.tns_admin","/u01/oracle/admin"); System.setProperty("oracle.net.wallet_location","/u01/oracle/admin"); String url = "jdbc:oracle:thin:/@secure"; DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver()); Connection conn = DriverManager.getConnection(url); Statement stmt = conn.createStatement (); ResultSet rset = stmt.executeQuery ("select HOST_NAME,INSTANCE_NAME FROM V$INSTANCE"); while (rset.next ()) System.out.println (rset.getString (1)); // -------------------------- EOF ------------------------------ return null; } } > add jar /tmp/hive-udf-0.1-SNAPSHOT.jar; Added [/tmp/hive-udf-0.1-SNAPSHOT.jar] to class path Added resources: [/tmp/hive-udf-0.1-SNAPSHOT.jar] hive> create temporary function qp as 'com.demo.queryp'; OK Time taken: 0.356 seconds hive> select qp(); FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments 'qp': No matching method for class com.fdot.demo.querypatron with (). Possible choices: _FUNC_(string) hive>

aliyesami · ‎10-19-2018

can anyone answer this question please?

aliyesami · ‎10-15-2018

I get this error if I try to run the function with no arguments (since I don't have any) > select qp(); FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments 'qp': No matching method for class com.demo.queryp with (). Possible choices: _FUNC_(string) hive>

aliyesami · ‎10-15-2018

I don't have any arguments? I want to add some constant numbers inside the function and display the results . this is a very valid situation where you don't have any arguments and jjust want to do something inside the function .

aliyesami · ‎10-15-2018

for hive UDFs do you have to select from a table or can they be written to do things like just adding two numbers and displaying the result? if yes then how will I call them from hive? e.g below is the code that connects to the oracle database and queries the data dic. it runs fine as a separate class and I am trying to use it as a UDF in hive. when I run it I get this error package com.demo; import org.apache.hadoop.hive.ql.exec.UDF; import java.sql.*; import oracle.jdbc.*; import org.apache.hadoop.io.Text; public class queryp extends UDF { public Text evaluate(final Text s) throws SQLException { //----- below code works fine as a seperate class till EOF ------------ System.setProperty("oracle.net.tns_admin","/u01/oracle/admin"); System.setProperty("oracle.net.wallet_location","/u01/oracle/admin"); String url = "jdbc:oracle:thin:/@secure"; DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver()); Connection conn = DriverManager.getConnection(url); Statement stmt = conn.createStatement (); ResultSet rset = stmt.executeQuery ("select HOST_NAME,INSTANCE_NAME FROM V$INSTANCE"); while (rset.next ()) System.out.println (rset.getString (1)); // -------------------------- EOF ------------------------------ return null; } } > add jar /tmp/hive-udf-0.1-SNAPSHOT.jar; Added [/tmp/hive-udf-0.1-SNAPSHOT.jar] to class path Added resources: [/tmp/hive-udf-0.1-SNAPSHOT.jar] hive> create temporary function qp as 'com.demo.queryp'; OK Time taken: 0.356 seconds hive> select qp(); FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments 'qp': No matching method for class com.fdot.demo.querypatron with (). Possible choices: _FUNC_(string) hive> NOTE: I do not have any arguments . this class queries the external database n returns results.

Online	Offline
Last Visited	‎11-03-2016 03:37 PM

Date Registered	‎04-22-2016 08:38 AM
Last Visited	‎11-03-2016 03:37 PM
Total Messages Posted	931
Total Kudos Received	46

Re: NON-ANSI JOIN in hive

Re: insert query hangs from hive view ONLY

Re: cant start rest server

Re: hbase rest api failing

Re: how to read time reported by yarn

Re: hive 2.3.2 with HDP 3.0

hive 2.3.2 with HDP 3.0

which version of mysql-connector-java to use with ...

Re: connection to hiveserver2 failing

Re: connection to hiveserver2 failing

Re: connection to hiveserver2 failing

connection to hiveserver2 failing

Re: getting hive exception

getting hive exception

Re: apahce Drill install issues

apahce Drill install issues

hbase not showing ascii value for ROW key

Re: how is this code working ?

how is this code working ?

error compiling hbase java code

happybase not working

Re: versioning question in hbase

Re: versioning question in hbase

versioning question in hbase

Re: performance related question

performance related question

Re: how to get updates into hive

how to get updates into hive

Re: knox PUT request failing

knox PUT request failing

what is wrong with this UDF ?

Re: UDF question 2

Re: UDF question

Re: UDF question

UDF question 2