Member since
12-17-2017
13
Posts
1
Kudos Received
0
Solutions
02-17-2018
12:58 PM
Hi All, I'm trying to evaluate IBM Big SQL on HDP using TPC-H and TPC-DS datasets, I'm now trying to setup IBM Big SQL on HDP but it seems the only way to do that is by purshasing it, Is ther any free way to use IBM Big SQL for testing and evaluation purposes? Thanks, Mohammed
... View more
- Tags:
- Hadoop Core
- hdp-2.5.0
Labels:
01-14-2018
03:24 PM
I'm trying to use the following repo to build and load tpc-h data into hdp: https://github.com/hortonworks/hive-testbench I ran ./tpch-build.sh and it was executed successfully, but when I tried to run ./tpch-setup.sh 2, I got the following error: 18/01/14 15:07:09 ERROR hdfs.DFSClient: Failed to close inode 38075 org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/all_2-1515942428 could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1641) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3198) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3122) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:843) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:500) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552) at org.apache.hadoop.ipc.Client.call(Client.java:1496) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy10.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:457) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176) at com.sun.proxy.$Proxy11.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1489) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1284) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:463) ls: `/tmp/tpch-generate/2/lineitem': No such file or directory Anyone can help?
... View more
01-12-2018
11:21 AM
@Jay Kumar SenSharma I followed your instructions and following what I got: I'm more a UI guy so I am not very familiar with command based systems, so the problem now when I connect to port 2222 I can't see the folder that contains the project , would you please help on this, appricate the help, [root@sandbox ~]# pwd /root [root@sandbox ~]# ls anaconda-ks.cfg Desktop initial-setup-ks.cfg Public testbench apache-maven-3.0.5 Documents Music start_scripts Videos apache-maven-3.0.5-bin.tar.gz Downloads Pictures Templates [root@sandbox ~]# ssh root@localhost -p 2222 root@localhost's password: Last login: Fri Jan 12 11:17:06 2018 from 172.17.0.1 [root@sandbox ~]# pwd /root [root@sandbox ~]# ls anaconda-ks.cfg build.out install.log sandbox.info start_hbase.sh blueprint.json hdp install.log.syslog start_ambari.sh [root@sandbox ~]# which hive /usr/bin/hive [root@sandbox ~]# echo $PATH /usr/lib64/qt-3.3/bin:/usr/lib/jvm/java/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/hdp/current/falcon-client/bin:/usr/hdp/current/hadoop-mapreduce-historyserver/bin:/usr/hdp/current/oozie-client/bin:/usr/hdp/current/falcon-server/bin:/usr/hdp/current/hadoop-yarn-client/bin:/usr/hdp/current/oozie-server/bin:/usr/hdp/current/flume-client/bin:/usr/hdp/current/hadoop-yarn-nodemanager/bin:/usr/hdp/current/pig-client/bin:/usr/hdp/current/flume-server/bin:/usr/hdp/current/hadoop-yarn-resourcemanager/bin:/usr/hdp/current/slider-client/bin:/usr/hdp/current/hadoop-client/bin:/usr/hdp/current/hadoop-yarn-timelineserver/bin:/usr/hdp/current/sqoop-client/bin:/usr/hdp/current/hadoop-hdfs-client/bin:/usr/hdp/current/hbase-client/bin:/usr/hdp/current/sqoop-server/bin:/usr/hdp/current/hadoop-hdfs-datanode/bin:/usr/hdp/current/hbase-master/bin:/usr/hdp/current/storm-client/bin:/usr/hdp/current/hadoop-hdfs-journalnode/bin:/usr/hdp/current/hbase-regionserver/bin:/usr/hdp/current/storm-nimbus/bin:/usr/hdp/current/hadoop-hdfs-namenode/bin:/usr/hdp/current/hive-client/bin:/usr/hdp/current/storm-supervisor/bin:/usr/hdp/current/hadoop-hdfs-nfs3/bin:/usr/hdp/current/hive-metastore/bin:/usr/hdp/current/zookeeper-client/bin:/usr/hdp/current/hadoop-hdfs-portmap/bin:/usr/hdp/current/hive-server2/bin:/usr/hdp/current/zookeeper-server/bin:/usr/hdp/current/hadoop-hdfs-secondarynamenode/bin:/usr/hdp/current/hive-webhcat/bin:/usr/hdp/current/hadoop-mapreduce-client/bin:/usr/hdp/current/knox-server/bin:/usr/hdp/current/hadoop-client/sbin:/usr/hdp/current/hadoop-hdfs-nfs3/sbin:/usr/hdp/current/hadoop-yarn-client/sbin:/usr/hdp/current/hadoop-hdfs-client/sbin:/usr/hdp/current/hadoop-hdfs-portmap/sbin:/usr/hdp/current/hadoop-yarn-nodemanager/sbin:/usr/hdp/current/hadoop-hdfs-datanode/sbin:/usr/hdp/current/hadoop-hdfs-secondarynamenode/sbin:/usr/hdp/current/hadoop-yarn-resourcemanager/sbin:/usr/hdp/current/hadoop-hdfs-journalnode/sbin:/usr/hdp/current/hadoop-mapreduce-client/sbin:/usr/hdp/current/hadoop-yarn-timelineserver/sbin:/usr/hdp/current/hadoop-hdfs-namenode/sbin:/usr/hdp/current/hadoop-mapreduce-historyserver/sbin:/usr/hdp/current/hive-webhcat/sbin:/root/bin
... View more
01-10-2018
06:28 PM
@dthakkar By login to host you mean the VMWare or the sandbox itself? I am already doing this. I'm using the terminal inside the sandbox to run the script, also I ran the cmmand "which hive" and got the following: usr/bin/which: no hive in (/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/root/bin)
... View more
01-10-2018
06:16 PM
@Jay Kumar SenSharma When running "which hive" I got the following: usr/bin/which: no hive in (/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/root/bin) Not sure what I should do next? please notice that I'm using HDP sandbox. Thanks for help.
... View more
01-08-2018
10:30 PM
I'm trying to build and generate data using the following repository: https://github.com/cartershanklin/hive-testbench But when ever I tried to run ./tpch-setup.sh 2 as example I got the following message: Script must be run where Hive is installed Anyone one can help me with this? Thanks, Mohammed
... View more
- Tags:
- Data Processing
- Hive
Labels:
12-24-2017
03:23 PM
Thank you @Shu So if I want to compare the response time for retrieving data (select) between two SQL ON Hadoop engines, Is there a possible way that I can measure that?, How I can compare?
... View more
12-24-2017
03:20 PM
Thanks @Geoffrey Shelton Okot I already did this, I want to know if there is any documentation of how I can upload TPC-H or TPC-DS data-sets and start querying them.
... View more
12-23-2017
05:19 PM
Thank you @Shu So I am running a select statment as example but I can't see it in the TEZ view. I can see only the insert statmtnet I perfoemed earlier, Am I missing something here?
... View more
12-23-2017
04:39 PM
I'm a beginner with HDP and I'm trying to measure the response time when query execution, Are there a build In matrices where I can see how good was the execution of my query? I'm running my query using Hive View on the Ambari UI. Thanks
... View more
12-23-2017
02:25 PM
I would like to familiarize myself more with the HDP platform, Would you please guide me where is the best place of documentation, examples or demos to start. Mainly I would like to to have APACHE Hive, Cloudera Impala, Spark SQL, and SPARK/Shark in HDP and loading TPC-H and TPC-DS or anyother workloads and start query these datasets using the mentioned SQL on hadoop engines. I tried one of the samples with uploading csv files and query them using hive, I would like to have more examples with the above. Many Thanks and appricate the help. Regards, Mohammed
... View more
Labels:
12-19-2017
01:45 PM
Hi Scott, Are any of the following SQL On Hadoop engines available within HDP: Concurrent Lingual, Hadapt, InfiniDB, MammothDB, MemSQL, Pivotal HawQ, Progress DataDirect, ScleraDB, Simba Splice Machine Spark SQL FacebookPresto. * LinkedIn Tajo Stinger, * Cloudera Impala, ** Platfora, JethroData, HAWQ, CitusDB, Rainstor, MapR Apache Drill Microsoft PolyBase, TeraData SQL-H, Spark/Shark * Hadoop++/HAIL Thanks, Mohammed
... View more
12-17-2017
04:50 PM
1 Kudo
I just want to ask what are the available SQL On Hadoop engines in Hortonworks Data Platform? Thanks,
... View more
Labels: