Member since
04-27-2016
61
Posts
61
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3368 | 09-19-2016 05:42 PM | |
971 | 06-11-2016 06:41 AM | |
2722 | 06-10-2016 05:17 PM |
05-31-2018
06:16 PM
While trying to start Namenode from Ambari, its failing with the below error: raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/3.0.0.0-1371/hadoop/bin/hdfs --config /usr/hdp/3.0.0.0-1371/hadoop/conf --daemon start namenode'' returned 1. WARNING: HADOOP_NAMENODE_OPTS has been replaced by HDFS_NAMENODE_OPTS. Using value of HADOOP_NAMENODE_OPTS.
ERROR: Cannot set priority of namenode process 355
Does anybody know a solution Thanks in advance
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
01-23-2017
06:36 PM
1 Kudo
Looking for help with the below support related questions 1. What versions of the software do you currently provide support for? 2. What was the release data of the oldest version currently supported?
... View more
- Tags:
- Hadoop Core
- hdp-2.3.4
- support
- Upgrade to HDP 2.5.3 : ConcurrentModificationException When Executing Insert Overwrite : Hive
- version
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
12-21-2016
07:16 PM
I went back to HDP 2.5.0 and no errors now. Looks like 2.5.3 has this start service issue. Thanks for the help
... View more
12-20-2016
10:45 PM
@Michael Young Thanks. I went ahead and did a 'restart all' on all those services that failed to start. It worked for some of them .However for few others like history server, hive server2, Namenode etc., i am still getting the following error in the log: Could you help me there, Thanks
History Server:
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.5.3.0-37/hadoop/mapreduce.tar.gz 'http://ganne-test0.field.hortonworks.com:50070/webhdfs/v1/hdp/apps/2.5.3.0-37/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444' 1>/tmp/tmpCBuCHv 2>/tmp/tmp76U3W6' returned 52. curl: (52) Empty reply from server
100
Hiveserver 2:
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://ganne-test0.field.hortonworks.com:50070/webhdfs/v1/user/hcat?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmpNYUL1Z 2>/tmp/tmpij3JWm' returned 7. curl: (7) Failed connect to ganne-test0.field.hortonworks.com:50070; Connection refused
000
... View more
12-20-2016
10:07 PM
1 Kudo
Repo Description The DiP API has been tested on below mentioned HDP 2.4 components: Apache Hadoop 2.7.1.2.4 Apache Kafka 0.9.0.2.4 Apache Apex 3.4.0 Apache Hbase 1.1.2.2.4 Apache Hive 1.2.1.2.4 Apache Zeppelin 0.6.0.2.4 Apache Tomcat Server 8.0 Apache Phoenix 4.4.0.2.4 Apache Maven Java 1.7 or later Repo Info Github Repo URL https://github.com/XavientInformationSystems/Data-Ingestion-Platform/tree/master/dataingest-apex Github account name XavientInformationSystems/Data-Ingestion-Platform/tree/master Repo name dataingest-apex
... View more
- Find more articles tagged with:
- ambari-extensions
- Data Ingestion & Streaming
Labels:
12-20-2016
10:02 PM
3 Kudos
While trying to install HDP 2.5.3 on a 4 node cluster via Ambari Wizard, I passed all the steps and got till this point of 'install, start and test' but it gives this warning that many services failed to start. Please see the attachments for pictures. Did any one face this issue?
... View more
Labels:
12-20-2016
09:43 PM
1 Kudo
Repo Description You can now deploy DataTorrent RTS within the Ambari stack. Such a deployment ensures simplified management of the DataTorrent RTS setup. Note: DataTorrent RTS, powered by Apache Apex, provides a high-performing, fault-tolerant, scalable, easy-to-use data processing platform for batch and streaming workloads. It includes advanced management, monitoring, development, visualization, data ingestion, and distribution features. Repo Info Github Repo URL https://github.com/DataTorrent/ambari-datatorrent-service Github account name DataTorrent Repo name ambari-datatorrent-service
... View more
- Find more articles tagged with:
- ambari-extensions
- Cloud & Operations
Labels:
11-14-2016
03:42 PM
Appreciate your input . Yes the issue was the key tab and principal. Had to kinit with the specific principal instead of the default one and the error was gone.thanks
... View more
11-11-2016
07:31 PM
Running into the errors in the pictures and log file , while trying to submit a topology on a kerberised cluster. Nimbus is running on lake2.field.hortonworks.com. Can some one shed some light here, who are kerberos gurus. Appreciate any help!
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Storm
09-28-2016
02:20 AM
@Timothy Spann yes that code was written for storm 0.10. Now I am trying to test that for 1.0.1. Updated the Pom with necessary storm and Kafka versions and added guava dependency as suggested in the above link. Still I am getting build errors
... View more
09-27-2016
11:30 PM
1 Kudo
I am facing the classic storm compatibility issue here with a demo. There is a java file here thats doing " import org.apache.storm.guava.collect.Lists;" With the new 2.5 Sandbox, Storm version is 1.0.1.2.5.0.0-1245. When i am compiling the code by running 'mvn clean package', i am running into compilation and build errors, as you can see in the image attached. I also attached the pom.xml file. Its complaining about guava dependencies, that are shaded in the new storm release. Please help to resolve the dependency issue. Thanks pom.xml
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Storm
09-19-2016
06:34 PM
What you can do as a work around is to ssh into the sandbox using "ssh root@127.0.0.1 -p 2222" and do the ambari-admin-password-reset .This generally works
... View more
09-19-2016
05:42 PM
4 Kudos
Go to VirtualBox Preferences>Network>Host-only Networks and Add host-only network for vboxnet0. This will solve the problem
... View more
09-19-2016
05:37 PM
1 Kudo
On HDP 2.5 Sandbox, my YARN application is submitted using Slider, but it is not running for more than 2 mins. 3 Containers are allocated. It goes to fails after the 2min 32sec mark and goes to finished state. yarn-log.zipI am attaching the YARN application log for reference. I can provide other additional logs/ details if required. Thanks in advance for the help.
... View more
Labels:
08-29-2016
12:17 AM
@Ted Yu Thanks. I see that my hbase-default.xml which is packaged in the "hbase-common-1.1.2.2.5.0.0-817.jar" has the 'hbase.master.logcleaner.plugins' value set to 'org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner'. Attaching my hbase-site.xml and hbase-default.xml files. Please let me know if you can find the issue. I also keep getting the below error in a storm topology: 'java.lang.RuntimeException: hbase-default.xml file seems to be for an older version of HBase (1.1.2), this version is 1.1.2.2.5.0.0-817'
... View more
08-26-2016
08:36 PM
Sure. I am attaching the pom.xml. Yes kafka dependecies are taken care of in the file. Infact this topology runs completely fine on HDP 2.4, 2.3. Please let me know any inputs.
... View more
08-26-2016
06:31 PM
Thanks. Added the .out files. Please let me know your thoughts
... View more
08-26-2016
06:27 PM
On HDP 2.5 TP Sandbox, running a storm topology for a telcom demo is giving me the below error: java.lang.NoSuchFieldError: PLAINTEXTSASL at kafka.utils.CoreUtils$.isSaslProtocol(CoreUtils.scala:282) at kafka.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:46) at kafka.javaapi.consumer.Si.. Attaching a picture. Can anybody help me identify the issue? Is it related to Kerberos and SSL on Kafka? Thanks
... View more
Labels:
08-26-2016
06:19 PM
On HDP 2.5 TP sandbox, HBase 's both master and Region servers are shutting down continuously. Attempting to restart the service from ambari also resulted in service going down after few seconds. Attaching the logs for reference. Any inputs appreciated. Thanks.
... View more
Labels:
06-21-2016
06:45 PM
4 Kudos
Try the below steps : 1.CREATE DATABASE IF NOT EXISTS test_1 2. DROP DATABASE IF EXISTS test_1 CASCADE 3.I see the error message has Meta Exception, so guessing it could be because of metastore not running. So do this service hive-metastore status check for the result is not process not started or metastore dead then restart the service service hive-metastore start
... View more
06-21-2016
05:51 PM
1 Kudo
I am on a mission to convert mysql based CLIs to direct Nova API calls in a certain openstack- DNS related shell scripts. So does anybody know some good places to get started with hands on working examples or tutorials of Nova APIs? i am aware of the openstack documentation ,but right now looking for more complex ones. Thanks!
... View more
06-20-2016
08:52 PM
Thanks for letting me know this.
... View more
06-15-2016
01:21 AM
5 Kudos
Often times, maintaining a hadoop cluster for longer periods on cloud is an expensive task. Sometimes engineers might also encounter situations of not having immediate access to cloud environment, in order to quickly spin up their own cluster and play arround. As an easy alternative, vagrant with virtual box as a provider, HDP cluster can be set up on your own laptop. Step 1 - Install prerequisites
Download and install from here Vagrant.
Download and install Oracle VirtualBox as the Vagrant Provider. Step 2 - Generation of Vagrantfile
Create a working directory for Vagrant file generation and initiating the deployment vis vagrant $ mkdir hdp22
$ cd hdp22 Following command will generate the Vagrantfile in current directory. This file will define VMs that are to be on the cluster. $ vagrant init
$ vi Vagrantfile Step 3 - Configuration of VMs in Vagrantfile Lets configure Vagrant to use CentOS 6.6/ CentOS 6.7 as the base box $ config.vm.box = "bento/centos-6.7" or
$ config.vm.box = "chef/centos-6.6" The below script should be included in the Vagrantfile to allow some basic provisioning for VMs like 1. Install NTP service 2. Disable firewall, SElinux 3.(Optional) Install wget $script = <<SCRIPT
sudo yum -y install ntp
sudo chkconfig ntpd on
sudo chkconfig iptables off
sudo /etc/init.d/iptables stop
sudo setenforce 0
sudo sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
sudo sh -c 'echo "* soft nofile 10000" >> /etc/security/limits.conf'
sudo sh -c 'echo "* hard nofile 10000" >> /etc/security/limits.conf'
sudo sh -c 'echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag'
sudo sh -c 'echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled'
SCRIPT
config.vm.provision "shell", inline:
$script
Step 4 - Configure the definition of VMs The following configurations will define 4 Virtual machines to be used in the HDP cluster, 1 Ambari server 1 Hadoop master 2 slaves The machines defined have below hostnames: 1. ambari1.mycluster 2. master1.mycluster 3. slave1.mycluster 4. slave2.mycluster # Ambari1
config.vm.define :ambari1 do |a1|
a1.vm.hostname = "ambari1.mycluster"
a1.vm.network :private_network, ip: "192.168.0.11"
a1.vm.provider :virtualbox do |vb|
vb.memory = "2048"
end
a1.vm.network "forwarded_port", guest: 8080, host: 8080
a1.vm.network "forwarded_port", guest: 80, host: 80
end
# Master1
config.vm.define :master1 do |m1|
m1.vm.hostname = "master1.mycluster"
m1.vm.network :private_network, ip: "192.168.0.12"
m1.vm.provider :virtualbox do |vb|
vb.memory = "4096"
end
end
# Slave1
config.vm.define :slave1 do |s1|
s1.vm.hostname = "slave1.mycluster"
s1.vm.network :private_network, ip: "192.168.0.21"
s1.vm.provider :virtualbox do |vb|
vb.memory = "2048"
end
end
# Slave2
config.vm.define :slave2 do |s2|
s2.vm.hostname = "slave2.mycluster"
s2.vm.network :private_network, ip: "192.168.0.22"
s2.vm.provider :virtualbox do |vb|
vb.memory = "2048"
end
end Step 5 - Start the Machines and Install Ambari Server Vagrant will automatically run the provision defined in Vagrantfile by Shell Provisioner to start the Ambari server machine from Vagrant. And then SSH to the Ambari server $ vagrant up ambari1
$ vagrant ssh ambari1 As a root user, run the below commands # Install
wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo
yum -y install ambari-server
sudo service ntpd start
# Setup. There are several options to configure during setup.
ambari-server setup
# Start Ambari Server
ambari-server start Add the following FQDN to each the /etc/hosts file on each VM. 192.168.0.11 ambari1.mycluster ambari1
192.168.0.12 master1.mycluster master1
192.168.0.21 slave1.mycluster slave1
192.168.0.22 slave2.mycluster slave2 Set up a password less SSHing from Ambari Node to all other nodes(VMs) $ ssh-keygen
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys Copy the ambari server's public key to other nodes' authorized keys to allow communication later. Step 6 - Deploy HDP Cluster We are ready to deploy a HDP cluster from Ambari Web UI. Because the UI is really simple, I would omit the screenshots here.
Access http://192.168.0.11:8080/ from your laptop. The username and password is admin and admin respectively. Give a cluster name. Select the latest HDP version Input hostname of the VMs (one per line) and the SSH private key of Ambari server. SSH user should be vagrant . Accept the default options for rest of the wizard. Complete the wizard. It takes about 30m to finish up. Now we are all set and a 4 nodes HDP cluster ready on your local machine!
... View more
- Find more articles tagged with:
- hadoop
- How-ToTutorial
- Linux
- solutions
- vagrant
- virtualbox
Labels:
06-13-2016
02:44 PM
4 Kudos
Repo DescriptionRepo Info Github Repo URL https://github.com/geetha-hortonworks/hive-testbench Github account name geetha-hortonworks Repo name hive-testbench
... View more
- Find more articles tagged with:
- benchmark
- Data Processing
- Hive
- hive performance
- sample-aps
Labels:
06-13-2016
02:42 PM
3 Kudos
Repo Description Building a ' n ' nodes HDP (Hortonworks Data Platform) cluster on your laptop with Apache Ambari and Vagrant is pretty simple. This is come handy when you are want to spin up a cluster and have no cloud access. Useful for developing and testing Hadoop applications. Repo Info Github Repo URL https://github.com/constantin-ungureanu-github/Hadoop-Cluster Github account name constantin-ungureanu-github Repo name Hadoop-Cluster
... View more
- Find more articles tagged with:
- Hadoop Core
- utilities
06-11-2016
06:41 AM
2 Kudos
1. H2O is an open source in-memory solution from 0xdata for predictive analytics on big data. With familiar APIs like R and JSON, as well as common storage method of using HDFS, H2O can bring the ability to do advance analysis. Please refer to the below tutorial from Hortonworks to get started http://hortonworks.com/hadoop-tutorial/predictive-analytics-h2o-hortonworks-data-platform/ 2. Apache Spark's MLLib is a very popular one used in predictive analytics 3. scikit-learn is another open source Machine learning tool
... View more
06-10-2016
10:39 PM
5 Kudos
This beautiful demo is courtesy of Vadim Vaks. It utilizes a Lambda architecture built using the Hortonworks Data Platform and Hortonworks Data Flow. The demo shows how a Telecom can manage customer device outages using predictive maintenance and a connected workforce. Overview: Customer devices that are simulated in this specific telecom usecase are SetTopBoxes(STB) in individual homes that might need assistance from a technician, when something goes wrong. Attributes associated with STB are: 1. SignalStrength 2. Internal Temperature 3. Status Location of technician is tracked and plotted on MapUI using the latitude and longitudes. Two cycles of operation of STBs are: 1. Normal Cycle: When status is normal and internal temperature of STB fluctuates up and down 2. Failure Cycle: When status is not normal and internal temperature of STBox incrementally goes high until 109 degrees.
Step 1: Prerequisites for the Demo Set Up:
For instructions to install this demo on an HDP 2.4 Sandbox,a good place to start is the README here DeviceMangerDemo 1. Clone the DeviceManagerDemo repository from here and follow the steps suggested below code: git clone https://github.com/vakshorton/DeviceManagerDemo.git
cd DeviceManagerDemo
./install.sh 2. The install.sh handles the installation and starting of artifacts necessary for the demo onto the Sandbox
Looking for the latest Hortonworks sandbox version
Creating the NiFi service configuration, installing and starting it using the Ambari ReST API
Importing the DeviceMangerDemo NiFi template, instantiating and starting the NiFi Flow using the NiFi ReST API
TEMPLATEID=$(curl -v -F template=@"Nifi/template/DeviceManagerDemo.xml" -X POST http://sandbox.hortonworks.com:9090/nifi-api/controller/templates | grep -Po '<id>([a-z0-9-]+)' | grep -Po '>([a-z0-9-]+)' | grep -Po '([a-z0-9-]+)')
REVISION=$(curl -u admin:admin -i -X GET http://sandbox.hortonworks.com:9090/nifi-api/controller/revision |grep -Po '\"version\":([0-9]+)' | grep -Po '([0-9]+)')
curl -u admin:admin -i -H "Content-Type:application/x-www-form-urlencoded" -d "templateId=$TEMPLATEID&originX=100&originY=100&version=$REVISION" -X POST http://sandbox.hortonworks.com:9090/nifi-api/controller/process-groups/root/template-instance
Starting the Kafka Ambari service using the Ambari ReST API and configuring the TechnicianEvent and DeviceEvents topics using the kafka-topics shell script
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper sandbox.hortonworks.com:2181 --replication-factor 1 --partitions 1 --topic DeviceEvents
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper sandbox.hortonworks.com:2181 --replication-factor 1 --partitions 1 --topic TechnicianEvent
Changing YARN Container Memory Size -/var/lib/ambari-server/resources/scripts/configs.sh set sandbox.hortonworks.com Sandbox yarn-site "yarn.scheduler.maximum-allocation-mb""6144" Starting the HBase service using the Ambari ReST API Installing ad starting Docker service, download docker images, creating the working folder with the slider for MapUI, Starting the Storm service using the Ambari ReST API, and deploying the storm topology here storm jar /home/storm/DeviceMonitor-0.0.1-SNAPSHOT.jar com.hortonworks.iot.topology.DeviceMonitorTopology 3. Install.sh reboots the ambari-server, wait for that and then run the below steps cd DeviceManagerDemo
./startDemoServices.sh 4.The startDemoServices.sh should be run each time the Sandbox VM is (re)started, after all of the default Sandbox services come up successfully. It handles the initialization of all of the application-specific components of the demo. The script starts the following Amabari Services via ReST API
Kafka
NiFi
Storm
Docker
UI Servlet and CometD Server on YARN using Slider
HBase
Step 2: Understanding the code, nifi processors and then navigate to the UI 1.First,make sure the kafka events are created by the install.sh script by doing below. You should see the two event names
cd /usr/hdp/current/kafkabroker/bin/
./kafkatopics.sh list zookeeper localhost:2181 2.The install.sh script that was run in the previous section creates and submits a storm topology named DeviceMonitor that has spouts and bolts Spouts: 1. DeviceSpout 2. Technician Spout Bolts:
EnrichDeviceStatus.java
IncidentDetector.java
PersistTechnicianLocation.java
PrintDeviceAlert.java
PublishDeviceStatus.java What each of the spouts does is simply look for the status of the device('Normal' or not) and technician( 'Assigned' or not) and enqueue or emit the status to bolts for future event decisions. Bolts process the data based on the event type (device/technician) like publishing the device status and technician location, updating the Hbase tables, enriching the device status, intelligent incident detection, printing alerts, routing the technician etc. Various configurations are done in DeviceMonitorTopology (detailed code here )that set the bolts and spouts using several methods like TopologyBuilder() ,setBolt(), setSpout(), SpoutConfig() etc 3. Verify the storm topology is properly submitted, by going to the StormUI quick link on Ambari and you should see below: Spouts and Bolts on the Storm UI: 4. Inoder to see the events get simulated,Run the simulator jars from two different CLIs: cd DeviceManagerDemo
java -jar DeviceSimulator-0.0.1-SNAPSHOT-jar-with-dependencies.jar Technician 1000 Simulation
java -jar DeviceSimulator-0.0.1-SNAPSHOT-jar-with-dependencies.jar STB 1000 Simulation DeviceStatus and TechnicianLocation Events are generated by the jars: Technician: 5. After importing the Devicemangerdemo Nifi template, it look like below, with several processors connected and running the data flow in sequential fashion.
6.Some of the major decision making processors involved and by viewing their configurations,one can see the content in 'properties' tab as below:
1.RouteOnAttribute:
ChannelTuneEventSearchIndex :${routingTarget:equalsIgnoreCase('ChannelTuneEventSearchIndex')}
DeviceEvents :${routingTarget:equalsIgnoreCase('DeviceEvents')}
TechnicianEvents :${routingTarget:equalsIgnoreCase('TechnicianEvents')}
DeviceInterface :${routingTarget:equalsIgnoreCase('DeviceInterface')}
2.DeviceHTTPInterface:
HTTP Method: POST
RemoteURL: ${targetProtocol:append(${targetIpAddress}):append(':'):append(${targetPort}):append('/server/TechnicianService')}
3.TechnicianlocationEvents: These technician information is pushed to the corresponding Kafka topic. 4.DeviceStatusEvents : Device status and other related event data is pushed to the corresponding Kafka topic. 5. EvaluateJSONPath Several jsonpaths that are user-defined properties like channel_i,deviceId_s,eventTimeStamp_I etc are evaluated against the content of the flowfile and written to the corresponding attributes in the next step of dataflow. 7. Now navigate to the MapUI to see the car of the technician moving arround, which was initiated by Slider here http://sandbox.hortonworks.com:8090/MapUI/DeviceMap 8. DeviceMonitorNostradamus section of the code utilizes the spark streaming and prediction capabilities. The enriched technician data from HBase is streamed into a Spark data model to predict possible outage of a customer device and later publish the predictions to MapUI web application using CometD server. Conclusion: This telecom demo gives overview of an IoT data application scenario and exposes the power of hortonworks dataflow technologies like Nifi, Kafka,Storm along with HBase, Spark and Slider.
... View more
- Find more articles tagged with:
- Data Processing
- HBase
- How-ToTutorial
- Kafka
- NiFi
- Storm
06-10-2016
05:17 PM
1 Kudo
Can you try --map-column-hive option. This will Overridethe default mapping from SQL type to Hive type for configured columns Refer to the documentation here https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_dataintegration/content/using_sqoop_to_move_data_into_hive.html
... View more
06-10-2016
05:16 PM
Can you try --map-column-hive option. This will Overridethe default mapping from SQL type to Hive type for configured columns Refer to the documentation here https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_dataintegration/content/using_sqoop_to_move_data_into_hive.html
... View more