Member since
02-23-2019
29
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3256 | 07-08-2016 02:32 AM |
03-16-2019
07:37 PM
It should be OK because MIT Kerberos client is running fine. [cloudera@~]$ telnet quickstart.cloudera 88
Trying 10.10.10.190...
Connected to quickstart.cloudera.
Escape character is '^]'.
^]
... View more
02-09-2019
07:05 AM
@Howchoy Nice to know it worked but the real issue is that the tookit.sh interprets the $ sign as a special character that's the reason you MUST use an escape character for it to work and the length of more than 13 characters. I am sure if you tried "Ce\$18C" it won't work either.
... View more
04-09-2019
02:44 PM
Hi guys, i followed the above steps, and was able to execute commands like ( show databases, show tables) successfully, also created a database from spark-shell and created a table and inserted some data in it, but i am not able to query the data either from the newly created table from spark, nor the tables that already exists in hive, and getting this error java.lang.AbstractMethodError: Method com/hortonworks/spark/sql/hive/llap/HiveWarehouseDataSourceReader.createBatchDataReaderFactories()Ljava/util/List; is abstract at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.createBatchDataReaderFactories(HiveWarehouseDataSourceReader.java) the commands is as below: import com.hortonworks.hwc.HiveWarehouseSession val hive = HiveWarehouseSession.session(spark).build() hive.createTable("hwx_table").column("value", "string").create() hive.executeUpdate("insert into hwx_table values('1')") hive.executeQuery("select * from hwx_table").show then the error appears, i am using the below command to start spark-shell spark-shell --master yarn --jars /usr/hdp/current/hive-warehouse-connector/hive-warehouse-connector_2.11-1.0.0.3.1.2.0-4.jar --conf spark.security.credentials.hiveserver2.enabled=false
... View more
10-08-2018
07:47 AM
It has been a while and I believed you already got it working. But for those people want to know how to do it, I am going to show you how I did it. Before I get started, I wanted to let you know I found the informative link by googling "setup Jupyter notebook at Hortonworks sandbox". Based on the link and I made some minor changes, I got it working. ####========================================================
### login as root
####========================================================
sandbox-version
== Sandbox Information ==
Platform: hdp-security
Build date: 06-18-2018
Ambari version: 2.6.2.0-155
Hadoop version: Hadoop 2.7.3.2.6.5.0-292
OS: CentOS Linux release 7.5.1804 (Core)
====
####========================================================
### Install Jupyter Dependencies
####========================================================
pip install --ignore-installed pyparsing
yum install epel-release
sudo wget https://bootstrap.pypa.io/ez_setup.py -O - | python ;sudo yum install python-pip python-wheel python-devel gcc
pip install --upgrade pip
pip install --upgrade pip wheel pandas numpy scipy scikit-learn matplotlib virtualenv
####========================================================
### Install Jupyter
####========================================================
pip install jupyter
####========================================================
### Setup folders and files
####========================================================
jupyter notebook --generate-config
sudo mkdir -p /ibm/conf
sudo chown -R spark:hadoop /ibm
cp ~/.jupyter/jupyter_notebook_config.py /ibm/conf/
####========================================================
### Setup startup shell script
####========================================================
vi /ibm/scripts/start_jupyter.sh
#copy the paste the following contents
#! bin/bash
set -x
USER=$1
JUPYTER_HOST=sandbox-hdp.hortonworks.com
JUPYTER_PORT=8889
su - ${USER} << EOF
export SPARK_HOME=/usr/hdp/current/spark-client
export PYSPARK_SUBMIT_ARGS="--master yarn-client pyspark-shell"
export HADOOP_HOME=/usr/hdp/current/hadoop-client
export HADOOP_CONF_DIR=/usr/hdp/current/hadoop-client/conf
export PYTHONPATH="/usr/hdp/current/spark-client/python:/usr/hdp/current/spark-client/python/lib/py4j-0.9-src.zip"
export PYTHONSTARTUP=/usr/hdp/current/spark-client/python/pyspark/shell.py
export PYSPARK_SUBMIT_ARGS="--master yarn-client pyspark-shell"
echo "Starting Jupyter daemon on HDP Cluster ..."
jupyter notebook --config=/ibm/conf/jupyter_notebook_config.py --ip=${JUPYTER_HOST} --port=${JUPYTER_PORT}&
EOF
exit 0
####========================================================
### Run startup shell script
####========================================================
chown -R spark:hadoop /ibm
chmod 777 /ibm/script/start_jupyter.sh
cd /ibm/scripts
./start_jupyter.sh spark
####========================================================
### Copy the link from above step's output and paste to your computer's browser
####========================================================
# make sure you define sandbox.hortonworks.com in your hosts file
http://sandbox.hortonworks.com:8889/?token=c982c0f95222abcf2900e3aeb9d9c59cc0386cc04c6c154d Test in Jupyter.
... View more
02-01-2018
10:46 PM
Hi, this is my Scala code to read Hbase tables. It's working with Hbase latest version 1.1.2.2.6.4.0-91 (HDP 2.6.4, Ambari 2.6.1). The key parameter is: conf.set("zookeeper.znode.parent", "/hbase-unsecure") because zookeeper doesn't hold the Hbase master detail. Check: # /usr/hdp/2.6.4.0-91/zookeeper/bin/zkCli.sh -server <server>.hortonworks.com:2181 # ls /hbase-unsecure/master return [] import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.client.Connection
import org.apache.hadoop.hbase.client.ConnectionFactory
import org.apache.hadoop.hbase.TableName
object Hbase {
val conf: Configuration = HBaseConfiguration.create()
def main(args: Array[String]): Unit = {
//conf.set("hbase.master", "<server>.hortonworks.com" + ":" + "60000")
conf.setInt("timeout", 120000)
conf.set("hbase.zookeeper.quorum", "<server>.hortonworks.com")
conf.set("zookeeper.znode.parent", "/hbase-unsecure") // IMPORTANT!!!
conf.setInt("hbase.client.scanner.caching", 10000)
val connection: Connection = ConnectionFactory.createConnection(conf)
val table = connection.getTable(TableName.valueOf("trading"))
print("connection created")
val admin = connection.getAdmin
// List the tables.
val listtables = admin.listTables()
listtables.foreach(println)
connection.close()
}
} Result: 'trading', {NAME => 'ca', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
... View more
07-12-2016
09:04 AM
Please see this for example https://dzone.com/articles/using-libjars-option-hadoop And http://stackoverflow.com/questions/6890087/problem-with-libjars-in-hadoop
... View more
06-29-2017
07:28 AM
Hi Rahul Pathak, I followed your instructions to setup Cassandra to my Hortonworks Sandbox (HDP_2.6_vmware_19_04_2017_20_25_43_hdp_ambari_2_5_0_5_1 ) and got an error - Connection failed: [Errno 111] Connection refused to sandbox.hortonworks.com:7000. The following is the steps (login as root): 1) Added file datastax.repo vi /etc/yum.repos.d/datastax.repo [datastax] name = DataStax Repo for Apache Cassandra baseurl = http://rpm.datastax.com/community enabled = 1gpgcheck = 0 2) Install Python requests easy_install-2.6 pip pip install requests 3) Downloaded the Cassandra service folder VERSION=`hdp-select status hadoop-client | sed 's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'` git clone https://github.com/Symantec/ambari-cassandra-service.git /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/CASSANDRA 4) Restart Ambari service ambari restart 5) Configured Cassandra: 'Add Service' from the 'Actions' dropdown menu in the bottom left of the Ambari dashboard Set seed_provider_parameters_seeds to "sandbox" 6) Restarted VM and Restarted servcies 7) Got a error in Cassandra, see attached pictures. Please shed some light on this issue. Thank you in advance for your reply.
... View more
06-05-2016
07:26 AM
Finally, I got it. This is what I’ve done to compile (mvn install –DskipTests) without any errors: 1) Open command prompt with administrator 2) Change repo link to: http://repo.hortonworks.com/content/groups/public 3) Require protoc.exe version 2.5.0 4) Require msbuild.exe from Visual Studio 2010 5) Install cmake.exe It should be mentioned in BUILDING.txt.
... View more