About m7mdsyam_cs

m7mdsyam_cs · ‎02-17-2018

Hi All, I'm trying to evaluate IBM Big SQL on HDP using TPC-H and TPC-DS datasets, I'm now trying to setup IBM Big SQL on HDP but it seems the only way to do that is by purshasing it, Is ther any free way to use IBM Big SQL for testing and evaluation purposes? Thanks, Mohammed

m7mdsyam_cs · ‎01-14-2018

I'm trying to use the following repo to build and load tpc-h data into hdp: https://github.com/hortonworks/hive-testbench I ran ./tpch-build.sh and it was executed successfully, but when I tried to run ./tpch-setup.sh 2, I got the following error: 18/01/14 15:07:09 ERROR hdfs.DFSClient: Failed to close inode 38075 org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/all_2-1515942428 could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1641) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3198) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3122) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:843) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:500) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552) at org.apache.hadoop.ipc.Client.call(Client.java:1496) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy10.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:457) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176) at com.sun.proxy.$Proxy11.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1489) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1284) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:463) ls: `/tmp/tpch-generate/2/lineitem': No such file or directory Anyone can help?

m7mdsyam_cs · ‎01-12-2018

@Jay Kumar SenSharma I followed your instructions and following what I got: I'm more a UI guy so I am not very familiar with command based systems, so the problem now when I connect to port 2222 I can't see the folder that contains the project , would you please help on this, appricate the help, [root@sandbox ~]# pwd /root [root@sandbox ~]# ls anaconda-ks.cfg Desktop initial-setup-ks.cfg Public testbench apache-maven-3.0.5 Documents Music start_scripts Videos apache-maven-3.0.5-bin.tar.gz Downloads Pictures Templates [root@sandbox ~]# ssh root@localhost -p 2222 root@localhost's password: Last login: Fri Jan 12 11:17:06 2018 from 172.17.0.1 [root@sandbox ~]# pwd /root [root@sandbox ~]# ls anaconda-ks.cfg build.out install.log sandbox.info start_hbase.sh blueprint.json hdp install.log.syslog start_ambari.sh [root@sandbox ~]# which hive /usr/bin/hive [root@sandbox ~]# echo $PATH /usr/lib64/qt-3.3/bin:/usr/lib/jvm/java/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/hdp/current/falcon-client/bin:/usr/hdp/current/hadoop-mapreduce-historyserver/bin:/usr/hdp/current/oozie-client/bin:/usr/hdp/current/falcon-server/bin:/usr/hdp/current/hadoop-yarn-client/bin:/usr/hdp/current/oozie-server/bin:/usr/hdp/current/flume-client/bin:/usr/hdp/current/hadoop-yarn-nodemanager/bin:/usr/hdp/current/pig-client/bin:/usr/hdp/current/flume-server/bin:/usr/hdp/current/hadoop-yarn-resourcemanager/bin:/usr/hdp/current/slider-client/bin:/usr/hdp/current/hadoop-client/bin:/usr/hdp/current/hadoop-yarn-timelineserver/bin:/usr/hdp/current/sqoop-client/bin:/usr/hdp/current/hadoop-hdfs-client/bin:/usr/hdp/current/hbase-client/bin:/usr/hdp/current/sqoop-server/bin:/usr/hdp/current/hadoop-hdfs-datanode/bin:/usr/hdp/current/hbase-master/bin:/usr/hdp/current/storm-client/bin:/usr/hdp/current/hadoop-hdfs-journalnode/bin:/usr/hdp/current/hbase-regionserver/bin:/usr/hdp/current/storm-nimbus/bin:/usr/hdp/current/hadoop-hdfs-namenode/bin:/usr/hdp/current/hive-client/bin:/usr/hdp/current/storm-supervisor/bin:/usr/hdp/current/hadoop-hdfs-nfs3/bin:/usr/hdp/current/hive-metastore/bin:/usr/hdp/current/zookeeper-client/bin:/usr/hdp/current/hadoop-hdfs-portmap/bin:/usr/hdp/current/hive-server2/bin:/usr/hdp/current/zookeeper-server/bin:/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin:/usr/hdp/current/hive-webhcat/bin:/usr/hdp/current/hadoop-mapreduce-client/bin:/usr/hdp/current/knox-server/bin:/usr/hdp/current/hadoop-client/sbin:/usr/hdp/current/hadoop-hdfs-nfs3/sbin:/usr/hdp/current/hadoop-yarn-client/sbin:/usr/hdp/current/hadoop-hdfs-client/sbin:/usr/hdp/current/hadoop-hdfs-portmap/sbin:/usr/hdp/current/hadoop-yarn-nodemanager/sbin:/usr/hdp/current/hadoop-hdfs-datanode/sbin:/usr/hdp/current/hadoop-hdfs-secondarynamenode/sbin:/usr/hdp/current/hadoop-yarn-resourcemanager/sbin:/usr/hdp/current/hadoop-hdfs-journalnode/sbin:/usr/hdp/current/hadoop-mapreduce-client/sbin:/usr/hdp/current/hadoop-yarn-timelineserver/sbin:/usr/hdp/current/hadoop-hdfs-namenode/sbin:/usr/hdp/current/hadoop-mapreduce-historyserver/sbin:/usr/hdp/current/hive-webhcat/sbin:/root/bin

m7mdsyam_cs · ‎01-10-2018

@dthakkar By login to host you mean the VMWare or the sandbox itself? I am already doing this. I'm using the terminal inside the sandbox to run the script, also I ran the cmmand "which hive" and got the following: usr/bin/which: no hive in (/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/root/bin)

m7mdsyam_cs · ‎01-10-2018

@Jay Kumar SenSharma When running "which hive" I got the following: usr/bin/which: no hive in (/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/root/bin) Not sure what I should do next? please notice that I'm using HDP sandbox. Thanks for help.

m7mdsyam_cs · ‎01-08-2018

I'm trying to build and generate data using the following repository: https://github.com/cartershanklin/hive-testbench But when ever I tried to run ./tpch-setup.sh 2 as example I got the following message: Script must be run where Hive is installed Anyone one can help me with this? Thanks, Mohammed

m7mdsyam_cs · ‎12-24-2017

Thank you @Shu So if I want to compare the response time for retrieving data (select) between two SQL ON Hadoop engines, Is there a possible way that I can measure that?, How I can compare?

m7mdsyam_cs · ‎12-24-2017

Thanks @Geoffrey Shelton Okot I already did this, I want to know if there is any documentation of how I can upload TPC-H or TPC-DS data-sets and start querying them.

m7mdsyam_cs · ‎12-23-2017

Thank you @Shu So I am running a select statment as example but I can't see it in the TEZ view. I can see only the insert statmtnet I perfoemed earlier, Am I missing something here?

m7mdsyam_cs · ‎12-23-2017

I'm a beginner with HDP and I'm trying to measure the response time when query execution, Are there a build In matrices where I can see how good was the execution of my query? I'm running my query using Hive View on the Ambari UI. Thanks

m7mdsyam_cs · ‎12-23-2017

I would like to familiarize myself more with the HDP platform, Would you please guide me where is the best place of documentation, examples or demos to start. Mainly I would like to to have APACHE Hive, Cloudera Impala, Spark SQL, and SPARK/Shark in HDP and loading TPC-H and TPC-DS or anyother workloads and start query these datasets using the mentioned SQL on hadoop engines. I tried one of the samples with uploading csv files and query them using hive, I would like to have more examples with the above. Many Thanks and appricate the help. Regards, Mohammed

m7mdsyam_cs · ‎12-19-2017

Hi Scott, Are any of the following SQL On Hadoop engines available within HDP: Concurrent Lingual, Hadapt, InfiniDB, MammothDB, MemSQL, Pivotal HawQ, Progress DataDirect, ScleraDB, Simba Splice Machine Spark SQL FacebookPresto. * LinkedIn Tajo Stinger, * Cloudera Impala, ** Platfora, JethroData, HAWQ, CitusDB, Rainstor, MapR Apache Drill Microsoft PolyBase, TeraData SQL-H, Spark/Shark * Hadoop++/HAIL Thanks, Mohammed

m7mdsyam_cs · ‎12-17-2017

I just want to ask what are the available SQL On Hadoop engines in Hortonworks Data Platform? Thanks,

Online	Offline
Last Visited	‎02-17-2018 12:58 PM

Date Registered	‎12-17-2017 04:48 PM
Last Visited	‎02-17-2018 12:58 PM
Total Messages Posted	13
Total Kudos Received	1

IBM Big SQL on HDP

Loading tpc-h data error

Re: Script must be run where Hive is installed

Re: Script must be run where Hive is installed

Re: Script must be run where Hive is installed

Script must be run where Hive is installed

Re: How I can measure reponse time for a query exe...

Re: Using different SQL On Hadoop engines to query...

Re: How I can measure reponse time for a query exe...

How I can measure reponse time for a query executi...

Using different SQL On Hadoop engines to query TPC...

Re: what are the available SQL On Hadoop engines i...

what are the available SQL On Hadoop engines in Ho...