Member since
01-06-2017
10
Posts
0
Kudos Received
0
Solutions
10-26-2017
02:04 AM
Yup. We got it resolved earlier this evening and that's what we did. Is this fact mentioned somewhere in the yarn installation/configuration portion? It's not like people setup kerberos right when they setup all the components before they are Kerberized, right? Anyhew, thanks for the response.
... View more
10-25-2017
05:17 PM
Hello, We just enabled FreeIPA integration in our Hortonworks cluster (HDP 2.5.3). We understand that part of the kerberization of Yarn, Ambari/yarn will try and delete the yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs. The directories defined for these configs in our environment are actual mount points, so as expected Yarn throws this error message "OSError: [Errno 16] Device or resource busy: '/hadoop/yarn/local/01". Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py", line 161, in <module>
Nodemanager().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py", line 51, in start
self.configure(env) # FOR SECURITY
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py", line 57, in configure
yarn(name="nodemanager")
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py", line 168, in yarn
action='delete'
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 114, in __new__
cls(names_list.pop(0), env, provider, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 208, in action_delete
sudo.rmtree(path)
File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 102, in rmtree
shutil.rmtree(path)
File "/usr/lib64/python2.7/shutil.py", line 256, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/lib64/python2.7/shutil.py", line 254, in rmtree
os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/hadoop/yarn/local/01' 2017-10-25 18:19:36,019 - checked_call returned (0, '')
2017-10-25 18:19:36,019 - Ensuring that hadoop has the correct symlink structure
2017-10-25 18:19:36,019 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-10-25 18:19:36,025 - Directory['/hadoop/yarn/local/01'] {'action': ['delete']}
2017-10-25 18:19:36,026 - Removing directory Directory['/hadoop/yarn/local/01'] and all its content
Command failed after 1 tries Any idea how do we get around this? Is there some script/flag we can modify for it to avoid attempting to delete these directories? Just an FYI, we can start the NM service on our DN manually. Any help would be appreciated. Thanks! K
... View more
Labels:
01-24-2017
05:21 PM
Are you going through a proxy or firewall to the internal repos ? - Nope The Ambari server logs should log all the errors, so don't ignore it can you copy&paste you last log? I shall check them out. Will have to reinstall. Will post later today. Can you make sure all the 8 nodes in your cluster OS RPM's/patches are at the same level and can access the internal repo webserver if you have pdsh installed you can quickly compare the nodes rpm status? They should be since they have no way of updating to a newer version but I shall check it out. can you share your ansible script? Sorry I can't do that right now. Corporate legal issues but I do intend to open source it sometime in the future.
... View more
01-23-2017
10:12 PM
Some background info. I am not doing these things manually. I have created Ansible deployment scripts that deploy the whole cluster using Ambari blueprints and MySQL HA. Hive and Oozie both talk to MySQL. I also have an uninstallation script that cleans up any HDP/Ambari footprints on the servers. It drops the relevant MySQL DB's as well. In 15-20 mins you have an 8 node cluster up and running from scratch. These nodes do not have internet connectivity and use an internal repo setup for my team. For now I am modifying my installation scripts to install the 4 RPM's before Ambari kicks off the Service install. The same RPM's failing to install every time gives me the impression its some dependency issue. The folders or permissions arent setup when these RPM's are being attempted to install and they are fixed further down which causes the rest of the RPM's to install without any issues. I would have resolved this issue If I had Ambari logs that captured the output of the RPM installs. My hostgroup setup is posted below. I use these to generate the blueprint. host_groups:
- name: Master_Nodes
cardinality: '1'
components:
- name: SPARK_CLIENT
- name: YARN_CLIENT
- name: HDFS_CLIENT
- name: SQOOP
- name: METRICS_MONITOR
- name: HIVE_METASTORE
- name: NAMENODE
- name: TEZ_CLIENT
- name: ZOOKEEPER_CLIENT
- name: HCAT
- name: PIG
- name: HST_AGENT
- name: MAPREDUCE2_CLIENT
- name: OOZIE_CLIENT
- name: HBASE_CLIENT
- name: HIVE_CLIENT
- name: Secondary_Master_Nodes
cardinality: '1'
components:
- name: AMBARI_SERVER
- name: HIVE_SERVER
- name: METRICS_MONITOR
- name: HBASE_MASTER
- name: TEZ_CLIENT
- name: ZOOKEEPER_CLIENT
- name: HCAT
- name: WEBHCAT_SERVER
- name: SECONDARY_NAMENODE
- name: HST_AGENT
- name: OOZIE_CLIENT
- name: METRICS_COLLECTOR
- name: METRICS_GRAFANA
- name: SPARK_CLIENT
- name: YARN_CLIENT
- name: HDFS_CLIENT
- name: SQOOP
- name: HST_SERVER
- name: HISTORYSERVER
- name: PIG
- name: MAPREDUCE2_CLIENT
- name: HBASE_CLIENT
- name: SPARK_JOBHISTORYSERVER
- name: OOZIE_SERVER
- name: APP_TIMELINE_SERVER
- name: HIVE_CLIENT
- name: RESOURCEMANAGER
- name: Kafka_Brokers
cardinality: '3'
components:
- name: KAFKA_BROKER
- name: HST_AGENT
- name: ZOOKEEPER_SERVER
- name: METRICS_MONITOR
- name: Slave_Nodes
cardinality: '3'
components:
- name: NODEMANAGER
- name: SPARK_CLIENT
- name: YARN_CLIENT
- name: HDFS_CLIENT
- name: SQOOP
- name: METRICS_MONITOR
- name: TEZ_CLIENT
- name: ZOOKEEPER_CLIENT
- name: HCAT
- name: PIG
- name: HST_AGENT
- name: MAPREDUCE2_CLIENT
- name: OOZIE_CLIENT
- name: HBASE_REGIONSERVER
- name: DATANODE
- name: HBASE_CLIENT
- name: HIVE_CLIENT
... View more
01-23-2017
02:27 AM
I spent all day today cleaning up the repos, disabling any unnecessary repos. Uninstalling every component and reinstalling the whole cluster multiple times. No dice. It's the same components that fail repeatedly and it's the same process to fix them. Reinstall the same RPM's again and restart the service. Yarn Nodemanagers fail to start SmartSense HST Agent fail to start Kafka Brokers fail to start. Metrics Collector fails to start.
If I don't have a solution I might just have to write a script to reinstall the RPM's cause I can't rely on Ambari. Edit: Adding the fact that all the files are missing are jar files. In case it helps. Ex of RPM missing files. Is the ambari agent deleting files once they are installed? Preinstalling the rpm's before I setup HDP on the cluster results in no errors. [root@<host> /]# yum info spark_2_4_2_0_258-yarn-shuffle.noarch
Loaded plugins: fastestmirror, langpacks, verify
Loading mirror speeds from cached hostfile
Installed Packages
Name : spark_2_4_2_0_258-yarn-shuffle
Arch : noarch
Version : 1.6.1.2.4.2.0
Release : 258.el6
Size : 6.6 M
Repo : installed
From repo : HDP-2.4
Summary : Spark Yarn Shuffle
URL : http://spark.apache.org/
License : ASL 2.0
Description : Spark Yarn Shuffle Jar
[root@st1-bdp-hadoop-03 /]# yum verify spark_2_4_2_0_258-yarn-shuffle.noarch
Loaded plugins: fastestmirror, langpacks, verify
==================== Installed Packages ====================
spark_2_4_2_0_258-yarn-shuffle.noarch : Spark Yarn Shuffle
File: /usr/hdp/2.4.2.0-258/hadoop/lib/spark-yarn-shuffle.jar
Problem: file is missing
File: /usr/hdp/2.4.2.0-258/spark/lib/spark-1.6.1.2.4.2.0-258-yarn-shuffle.jar
Problem: file is missing
verify done
[root@<host> ambari-agent]# yum verify smartsense-hst.x86_64
Loaded plugins: fastestmirror, langpacks, verify
==================== Installed Packages ====================
smartsense-hst.x86_64 : Hortonworks SmartSense Tool
File: /etc/hst/conf/anonymization_rules.json
Problem: checksum does not match
Current: sha256:0ead985082991351d6fc9f7b1ed7243a06d13f549f63584d23652961dce9d0a1
Original: sha256:9d5402ba85c07665b0b1315661866437804256a2b868688d5e0aaad6e2520752
--------
Problem: size does not match
Current: 882
Original: 877
--------
Problem: mode does not match
Current: user:wr-, group:-r-, other:-r-
Original: user:wrx, group:-rx, other:-rx
--------
Problem: mtime does not match
Current: Thu Jan 19 21:53:36 2017 (259 days, 3:29:14 later)
Original: Thu May 5 18:24:22 2016
File: /etc/hst/conf/capture_levels.json
Problem: checksum does not match
Current: sha256:9d0afaff37419601af7f7629a001369512dc00d53066249e6c91200cae07a04b
Original: sha256:5e51a1cdd967eeae34474e7fbebf16ecbea7e4623bf66fe56d2655d9fb1b4555
--------
Problem: size does not match
Current: 632
Original: 642
--------
Problem: mode does not match
Current: user:wr-, group:-r-, other:-r-
Original: user:wrx, group:-rx, other:-rx
--------
Problem: mtime does not match
Current: Thu Jan 19 21:53:36 2017 (259 days, 3:29:14 later)
Original: Thu May 5 18:24:22 2016
File: /etc/hst/conf/hst-agent.ini
Problem: checksum does not match
Current: sha256:839b03625393b77cdd9fdf0edd167c305b2245c7522ff476381aad1abfd5c799
Original: sha256:b7c5113089ef60a9e36dfb2645902ccc26de810fd927b699dcb94b5b098514cb
--------
Problem: size does not match
Current: 3.9 k
Original: 6.0 k
--------
Problem: mode does not match
Current: user:wr-, group:-r-, other:-r-
Original: user:wrx, group:-rx, other:-rx
--------
Problem: mtime does not match
Current: Mon Jan 23 02:09:59 2017 (262 days, 7:45:37 later)
Original: Thu May 5 18:24:22 2016
File: /etc/hst/conf/log4j.properties
Problem: checksum does not match
Current: sha256:6b49a2e1df7aa81ba3ff9f10fd4794fe5f9c78ea683c95c66ec1ff7cf7dea6ec
Original: sha256:d0d45c50bb5876ef6dfd3f2ebd14444ab2037e5d177cfd13e7eb897afa43a4c4
--------
Problem: size does not match
Current: 1450 B
Original: 1440 B
--------
Problem: mode does not match
Current: user:wr-, group:-r-, other:-r-
Original: user:wrx, group:-rx, other:-rx
--------
Problem: mtime does not match
Current: Thu Jan 19 21:53:36 2017 (259 days, 3:29:14 later)
Original: Thu May 5 18:24:22 2016
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/alerts.json
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/configuration
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/configuration/anonymization-rules.xml
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/configuration/capture-levels.xml
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/configuration/hst-agent-conf.xml
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/configuration/hst-common-conf.xml
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/configuration/hst-log4j.xml
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/configuration/hst-server-conf.xml
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/metainfo.xml
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/files
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/files/.gitignore
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/files/rpm
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/files/rpm/.gitignore
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/files/view
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/files/view/.gitignore
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/files/view/smartsense-ambari-view-1.2.2.0-460.jar
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/files/view/view_default.xml
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/files/view/view_instance.xml
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/scripts
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/scripts/hst.py
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/scripts/hst_agent.py
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/scripts/hst_script.py
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/scripts/hst_server.py
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/scripts/hst_service.py
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/scripts/params.py
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/scripts/service_check.py
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/package/scripts/utils.py
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/role_command_order.json
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/themes
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/themes/theme.json
Problem: file is missing
File: /usr/hdp/share/hst/ambari-service/SMARTSENSE/themes/theme_v2.json
Problem: file is missing
File: /usr/hdp/share/hst/bin
Problem: file is missing
File: /usr/hdp/share/hst/bin/hst-env.sh
Problem: file is missing
File: /usr/hdp/share/hst/bin/hst-scheduled-capture.sh
Problem: file is missing
File: /usr/hdp/share/hst/bin/install-helper.sh
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/Bootstrap.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/Bootstrap.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/Controller.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/Controller.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/HSTAgent.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/HSTConfig.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/HSTConfig.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/PropertyReader.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/ServerAPI.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/ServerAPI.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/__init__.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/__init__.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/anonymize.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/AgentUpgradeCommand.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/AnonymizeBundleCommand.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/ArchiveDataCommand.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/CancelRequestCommand.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/CleanupCommand.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/CollectDataCommand.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/CollectDataCommand.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/DownloadScriptCommand.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/GetScriptsCommand.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/UpdateConfigCommand.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/UploadCommand.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/__init__.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/command/__init__.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/commands.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/commands.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/cron.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/heartbeat.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/heartbeat.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/hostname.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/levels.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/levels.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/lock.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/lock.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/main.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/oscheck.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/oscheck.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/patch.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/request.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/script.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/script.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/security.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/security.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/shell.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/shell.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/upgrade
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/upgrade/RequestWithMethod.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/upgrade/UpgradeConfig120.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/upgrade/UpgradeService120.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/upgrade/__init__.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/upgrade/upgrade.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/util.py
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/lib/hst_agent/util.pyc
Problem: file is missing
File: /usr/hdp/share/hst/hst-agent/python-wrap
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/activation-1.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/ambari-views-1.7.0.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/anonymization-1.2.2.0-460.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/aopalliance-1.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/asm-3.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/asm-3.3.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/avalon-framework-4.1.3.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/cglib-2.2.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/cglib-nodep-2.2.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/com.sun.el-1.0.0.v201105211818.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commonj.sdo-2.1.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-1.2.2.0-460.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-beanutils-1.7.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-beanutils-core-1.8.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-cli-1.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-codec-1.9.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-collections-3.2.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-compress-1.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-configuration-1.6.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-digester-1.8.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-io-2.4.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-lang-2.6.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-lang3-3.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-lang3-3.3.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-logging-1.1.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-logging-1.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-net-1.4.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/commons-vfs2-2.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/cron-parser-core-2.8.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/cron-utils-3.1.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/derby-10.9.1.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/easymock-3.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/eclipselink-2.4.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/gatewayClient-1.2.2.0-460.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/gson-2.2.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/guava-14.0.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/guice-3.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/guice-assistedinject-3.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/guice-multibindings-3.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/guice-persist-3.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/guice-servlet-3.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/hamcrest-core-1.3.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/htime-1.0.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/httpclient-4.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/httpcore-4.4.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/httpmime-4.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jackson-core-asl-1.9.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jackson-jaxrs-1.9.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jackson-mapper-asl-1.9.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jackson-xc-1.9.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/javassist-3.17.1-GA.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/javax.el-2.1.0.v201105211819.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/javax.inject-1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/javax.persistence-2.0.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/javax.servlet-2.5.0.v201103041518.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/javax.servlet.jsp-2.1.0.v201105211820.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/javax.servlet.jsp.jstl-1.2.0.v201105211821.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jaxb-api-2.2.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jaxb-impl-2.2.3-1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jersey-client-1.11.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jersey-core-1.11.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jersey-guice-1.11.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jersey-json-1.11.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jersey-multipart-1.11.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jersey-server-1.11.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jersey-servlet-1.11.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jettison-1.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-client-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-continuation-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-http-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-io-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-jsp-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-security-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-server-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-servlet-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-servlets-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-util-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-webapp-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jetty-xml-7.6.7.v20120910.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/joda-time-2.3.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jsch-0.1.52.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jsoup-1.8.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jsp-api-2.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jsr250-api-1.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/jsr305-1.3.9.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/junit-4.11.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/log4j-1.2.17.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/logkit-1.0.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/mailapi-1.5.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/maven-scm-api-1.4.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/maven-scm-provider-svn-commons-1.4.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/maven-scm-provider-svnexe-1.4.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/mimepull-1.6.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/objenesis-1.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/objenesis-tck-1.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/org.apache.jasper.glassfish-2.1.0.v201110031002.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/org.apache.taglibs.standard.glassfish-1.2.0.v201112081803.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/org.eclipse.jdt.core-3.7.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/oro-2.0.8.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/plexus-utils-1.5.6.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/powermock-api-easymock-1.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/powermock-api-support-1.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/powermock-core-1.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/powermock-module-junit4-1.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/powermock-module-junit4-common-1.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/powermock-reflect-1.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/regexp-1.3.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/servlet-api-2.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/slf4j-api-1.7.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/slf4j-jdk14-1.7.7.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/slf4j-log4j12-1.7.5.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/smartsense-gateway-1.2.2.0-460.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/smartsense-hst-1.2.2.0-460.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/smtp-1.5.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-aop-3.0.7.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-asm-3.0.7.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-beans-3.0.7.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-context-3.0.7.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-core-3.0.7.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-expression-3.0.7.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-jdbc-3.0.7.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-security-config-3.1.2.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-security-core-3.1.2.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-security-web-3.1.2.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-tx-3.0.7.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/spring-web-3.0.7.RELEASE.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/stateless4j-2.5.0.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/stax-api-1.0-2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/stax-api-1.0.1.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-common/lib/xz-1.2.jar
Problem: file is missing
File: /usr/hdp/share/hst/hst-gateway/python-wrap
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/WEB-INF
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets/hstapp-0d8dee290917e84534d554e3393e2598.css
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets/hstapp-5db66ede969d324e420f4053a0139e13.js
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets/images
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets/images/hortonworks-logo-555555-cb831a5b1cf99fd004f671962de2798a.png
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets/images/hortonworks-logo-79c433-e25f943124d2e451123d2c930a09ede1.png
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets/images/hortonworks-logo-9d9d9d-0756bc37e8a7cbebb3d93872c1b3bf28.png
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets/images/hortonworks-logo-ffffff-e2c63b1388ee67626d284d0a070780ad.png
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets/images/hortonworks-logo-green-d3847d21b50b9afdd214edccdfaf2f79.png
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets/images/hortonworks-logo-wht-1082af8aadaefc935ffca99cb96bb2a8.png
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets/vendor-644a1cfbdf04d5c5c317d4de5f3b18f5.js
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/assets/vendor-6889eac3f8cd620d9f9066ac4297e9ca.css
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/crossdomain.xml
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/docs
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/docs/LICENSE
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/favicon.ico
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/fonts
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/fonts/bootstrap
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/fonts/bootstrap/glyphicons-halflings-regular.eot
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/fonts/bootstrap/glyphicons-halflings-regular.svg
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/fonts/bootstrap/glyphicons-halflings-regular.ttf
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/fonts/bootstrap/glyphicons-halflings-regular.woff
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/fonts/bootstrap/glyphicons-halflings-regular.woff2
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/index.html
Problem: file is missing
File: /usr/hdp/share/hst/hst-server/web/robots.txt
Problem: file is missing
File: /usr/hdp/share/hst/parallel-sh.py
Problem: file is missing
File: /usr/hdp/share/hst/patches
Problem: file is missing
File: /usr/hdp/share/hst/patches/hdp-select-patchST37.sh
Problem: file is missing
File: /var/run/hst
Problem: file is missing
verify done
... View more
01-22-2017
12:00 PM
This is a sample output from running yum verify-all on one of my cluster data nodes. Any ideas about why I am hitting this issue repeatedly? The RPM was installed with files missing. A reinstall fixes the issues. So why does it fail during its first install? ambari-metrics-hadoop-sink.x86_64 : Ambari Metrics Assembly
File: /usr/lib/ambari-metrics-hadoop-sink
Problem: file is missing
File: /usr/lib/ambari-metrics-hadoop-sink/ambari-metrics-hadoop-sink-with-common-2.2.2.0.460.jar
Problem: file is missing
File: /usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink-with-common-2.2.2.0.460.jar
Problem: file is missing
File: /usr/lib/ambari-metrics-kafka-sink/lib
Problem: file is missing
File: /usr/lib/ambari-metrics-kafka-sink/lib/commons-lang-2.6.jar
Problem: file is missing
File: /usr/lib/ambari-metrics-kafka-sink/lib/commons-logging-1.1.1.jar
Problem: file is missing
File: /usr/lib/ambari-metrics-kafka-sink/lib/jackson-core-asl-1.9.13.jar
Problem: file is missing
File: /usr/lib/ambari-metrics-kafka-sink/lib/jackson-mapper-asl-1.9.13.jar
Problem: file is missing
File: /usr/lib/ambari-metrics-kafka-sink/lib/jackson-xc-1.9.13.jar
Problem: file is missing
File: /usr/lib/flume/lib
Problem: file is missing
File: /usr/lib/flume/lib/ambari-metrics-flume-sink-with-common-2.2.2.0.460.jar
Problem: file is missing
File: /usr/lib/storm/lib
Problem: file is missing
File: /usr/lib/storm/lib/ambari-metrics-storm-sink-with-common-2.2.2.0.460.jar
Problem: file is missing
... View more
01-21-2017
09:49 PM
Thanks for the quick response. I shall try it out but it doesn't answer my questions. Why would only part of the files that comprise an RPM be copied to their appropriate locations? Are these installation errors logged somewhere ?
... View more
01-21-2017
08:09 PM
Hello, I am having issues during HDP install. Services have trouble coming online because some RPM's have files missing when after they are installed. For instance. Yarn NodeManager would fail to come online cause of a missing spark-yarn-shuffle.jar. The RPM that is supposed to contain the jar is installed according to yum. Removing that RPM using with --no-deps option and reinstalling it fixes this problem. The RPM uninstall shows what files were missing which includes the jar in question. The same thing happens when bringing up the Kafka brokers (KafkametricsSink jar) There is no guarantee that the same services fail each time but the same culprits tend to fail more often than not. This may or may not be an Ambari or HDP related issue but I am asking if anyone encountered similar issues and how they ensured that the HDP services install was copacetic? Any ideas on what I could check are also welcome. The servers do not have internet access and a local repo was setup for us.
... View more
Labels:
01-09-2017
05:26 PM
Thanks! I shall work with our ops guys and test this out.
... View more
01-06-2017
11:36 PM
Hello, I am trying to create an ansible deployment script for HDP using Ambari blueprints. So far I can deploy clusters on an environment where I have internet access. However, I am unable to do so on environments that have their own local cached repo and no internet access for the servers. I get the following error when I am trying to modify the base repo url "org.apache.ambari.server.controller.spi.NoSuchResourceException: The specified resource doesn't exist: Stack data, Stack HDP 2.4 is not found in Ambari metainfo" This is when I set the repositories base url to our own internal repo: http://<internal_URL>:8080/repos/hortonworks/HDP/centos7/2.x/updates/2.4.2.0 FYI, I have confirmed that the above repo works. I understand I am getting the above exception because the info isn't there in the metainfo file. When I log on to the Ambari UI, I see the following stacks: Stacks
HDP 2.3
HDP 2.2
HDP 2.1 So I am trying to figure out if I am missing any steps after I do the ambari-server setup? Is my base URL wrong or have I missed changing any config file? I don't know where is it getting the previous versions of HDP. I can see them in the metainfo file under /var/lib/ but why are there previous versions of HDP present there. This is the repo file setup for me. #VERSION_NUMBER=2.2.0.0-1310
[Updates-ambari-2.2.0.0]
name=ambari-2.2.0.0 - Updates
baseurl=http://<repo_url>/repos/hortonworks/ambari-2.2.0.0/centos7/2.2.0.0-1310
gpgcheck=1
gpgkey=http://<repo_url>/repos/keys/RPM-GPG-KEY-ambari
enabled=1
priority=1
#VERSION_NUMBER=2.4.2.0-258
[HDP-2.4.2.0]
name=HDP Version - HDP-2.4.2.0
baseurl=http://<repo_url>/repos/hortonworks/HDP/centos7/2.x/updates/2.4.2.0
gpgcheck=1
gpgkey=http://<repo_url>/repos/keys/RPM-GPG-KEY-HDP
enabled=1
priority=1
[HDP-UTILS-1.1.0.20]
name=HDP Utils Version - HDP-UTILS-1.1.0.20
baseurl=http://<repo_url>/repos/hortonworks/HDP-UTILS-1.1.0.20/repos/centos7gpgcheck=1
gpgkey=http://<repo_url>/repos/keys/RPM-GPG-KEY-HDP-UTILSenabled=1
priority=1 I would be really grateful If anyone can point me to documentation which explains how I can setup a stack after I complete the Ambari server setup. I have seen the wiki (https://cwiki.apache.org/confluence/display/AMBARI/How-To+Define+Stacks+and+Services). There is nothing there that explains on how I can setup an HDP 2.4 Stack. Thank you!
... View more
- Tags:
- stackaccessexception
