Member since
05-07-2014
17
Posts
3
Kudos Received
0
Solutions
12-07-2017
09:01 PM
Thanks ghagleitner for the reply. It looks like we don't have a choice apart from using hive managed tables. I understand the rationale.
... View more
12-07-2017
11:04 AM
Hi There, I have gone through the blogs, docs, tutorials (links below) and concluding that ACID transactions works with Internal Hive tables with ORC format. They do not work with External Hive Tables. Please let me know if I have misunderstood and there is a way to make ACID transacitions work with External Hive Tables. https://hortonworks.com/tutorial/using-hive-acid-transactions-to-insert-update-and-delete-data/ https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_data-access/content/ch02s05s01.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_data-access/content/creating-hive-acid-transaction-tables.html Many thanks. Cheers -Kiks.
... View more
- Tags:
- Data Processing
- Hive
Labels:
11-20-2017
12:20 PM
Sorry ignore my silly question. Unlike HDP, DPS downloads is via customer portal.
... View more
11-07-2017
10:18 PM
Hi, Just wondering where the public repositories for DPS are. I can't find the tarballs to download from the documentation available here. https://docs.hortonworks.com/HDPDocuments/DPS1/DPS-1.0.0/bk_dps-installation/content/ch_using-local-repos.html Am I right in assuming that DPS is also open source s/w like HDP and free to use but pay for the support? Please correct me if I have got it wrong and there is a different licensing arrangement for DPS. Cheers,
... View more
07-28-2017
01:18 PM
Hi There, Just wondering if there is a support roadmap for various HDP versions i.e
Is HDP 2.3 version still supported by hortonworks?
When will hortonworks stop supporting this version? If this version is already out of support, what was the date? Is there a concept of extended support for clients who are still on this version? 2. Same information for 2.5 3. Same information for 2.6. I know this is fairly newer version but wondering if there is a roadmap for support. I know we could liaise with our account manager for this but wondering why is it not easily available information. Tried searching but could only find that hortonwork provide support on rolling window for two previous versions but doesn't answer all my questions. Please let me know if you have it. Cheers, -Kiks.
... View more
- Tags:
- support
07-28-2017
01:12 PM
Hi There,
Just wondering if there is a support roadmap for various HDP versions i.e
Is HDP 2.3 version still supported by hortonworks?
When will hortonworks stop supporting this version? If this version is already out of support, what was the date? Is there a concept of extended support for clients who are still on this version?
2. Same information for 2.5
3. Same information for 2.6. I know this is fairly newer version but wondering if there is a roadmap for support.
I know we could liaise with our account manager for this but wondering why is it not easily available information. Tried all sorts of searches but not finding the right information. Please let me know if you have it.
Cheers,
-Kiks.
... View more
- Tags:
- support
01-04-2017
06:09 PM
1 Kudo
Absolutely. My thinking (although very hard to implement due to various constraints) 1. Exploit the newer technology as much as you can. 2. Empower and encourage the administrators, developers, tester, release managers to automate Infra, test, deploy etc. Encourage the teams to embrace devops culture. 3. Empower them do intelligent stuff with good quality rather than repetitive stuff with poor quality. 4. Spend money on training people rather than spending it on the tin. 5. Cutdown the lead times for doing things making organisation turely agile. 6. Cutdown the total cost of ownership for the business & to the projects. We are now living in "disruptive era" so let our actions also be disruptive. These things will help organisations in the long run as business and projects do not cut corners while delivering projects. As an architect, I work hard in protecting the architecture view when there is a delivery cost and delivery risk double edge sword on your neck.
... View more
01-04-2017
09:26 AM
2 Kudos
Good answer @bpreachuk. Agreed and I would be tempted to go down the
segregated clusters path with the constraints I work in. But given a
chance, I want to move away from this thinking/culture. Today's technology support IaaC. If Organisation build true
devops culture, #1 and #2 can easily be addressed by spinning up a cluster on
the click of a button. I see them as small short lived projects with a defined
start and a defined end. Also dedicated environments may not be needed for
something which is infrequent. #3 can be addressed using various combinations of the
security features available out there and I see it is no different to two
teams/users having different access privileges to the data in the same
environment e.g production. #4 can be addressed partially by change in culture and
partially by technology segregating the environments within the cluster at storage
layer and processing layer. I have been in situations where different parts of organisations
(sometime different programmes within the same sub-organisations) do not agree
on things. e.g one wanting to upgrade for the new features available in the newer release and another not willing to pay for regression testing of their workloads resulting in proliferation of the software in production environments. I have also known some
example of proliferation HDP clusters in production environment resulting in data ponds
instead of building one big datalakes with a single HDP cluster. Sometimes I wonder if
we are replicating the traditional problems in the big data set-ups. In short, if you have a strong devops teams with strong expertise
and appetite to IaaC, CI/CD and depending on your organisation's culture, I
think single environment is definitely be possible. I believe if we aim high,
we will achieve high. Let me know your views. Cheers, -KK.
... View more
12-13-2016
10:33 PM
Hi, Does anyone know the certification matrix between hortonworks HDP versions and SAS dataloader for hadoop versions? Tried searching but could not find this information. Cheers KK
... View more
11-07-2016
01:25 PM
Hi Jonathan, Please let me know your findings. I will share mine if I progress on it. I have a test workload as well to run it on my test cluster. Need to upgrade to HDP2.5 before I could run the tests. Currently snowed under at work so not finding time to do it. It is one of my list of things to do. Cheers, KK
... View more
10-26-2016
05:01 PM
Hi, With HDP2.5 version supporting both spark versions - i.e. 1.6.2 & 2.0 (TP) - has anyone tried running both versions of spark code (not necessarity same spark application) on the same cluster? Are there any lessons learnt? I heard there are problems when both versions of the code executed on the same cluster but does not have further details. Just wondering if anyone has run some tests/evaluating 2.0 version usage for building future workloads but no immediate urge to convert all the current workloads to be migrated from oldversion to 2.0. Let me know your views. Cheers, KK.
... View more
06-02-2014
10:50 AM
Hi Is there a number I can reach you on? I have teamviewer details. Cheers, -Kiran.
... View more
06-02-2014
10:22 AM
Hi, Thanks for the reply. 1. Both of the vm's are bridged network 2. I am able to ping both machine from both VM using the hostname and ip address. 3. /etc/hosts on both vm'shave the following 127.0.0.1 localhost 192.168.1.100 cloudera1-VirtualBox cloudera1 192.168.1.103 cloudera2-virtualBox cloudera2 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters still no luck with the installation. Cheers, -Kiran.
... View more
05-29-2014
02:13 PM
When I click details on the failed node, it seems to be referring to 192.168.1.100... Please see the below log. Failed to receive heartbeat from agent. (Current Step) Last Refreshed: May 29, 2014 10:05:43 PM BST /tmp/scm_prepare_node.4g47aDjH using SSH_CLIENT to get the SCM hostname: 192.168.1.100 47634 22 opening logging file descriptor Starting installation script... Acquiring installation lock... BEGIN flock 4 END (0) Detecting root privileges... effective UID is 0 Detecting distribution... BEGIN grep 'Ubuntu' /etc/lsb-release END (0) DISTRIB_ID=Ubuntu DISTRIB_DESCRIPTION="Ubuntu 12.04.4 LTS" BEGIN grep DISTRIB_CODENAME /etc/lsb-release DISTRIB_CODENAME=precise END (0) BEGIN echo DISTRIB_CODENAME=precise | cut -d = -f 2 END (0) precise Detecting Cloudera Manager Server... BEGIN host -t PTR 192.168.1.100 Host 100.1.168.192.in-addr.arpa. not found: 3(NXDOMAIN) END (1) BEGIN which python END (0) BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' 192.168.1.100 7182 /usr/bin/python END (0) BEGIN which wget END (0) /usr/bin/wget BEGIN wget -qO- -T 1 -t 1 http://169.254.169.254/latest/meta-data/public-hostname && /bin/echo END (4) Installing package repositories... validating format of repository file /tmp/scm_prepare_node.4g47aDjH/repos/ubuntu_precise/cloudera-manager.list installing repository file /tmp/scm_prepare_node.4g47aDjH/repos/ubuntu_precise/cloudera-manager.list repository file /tmp/scm_prepare_node.4g47aDjH/repos/ubuntu_precise/cloudera-manager.list installed BEGIN apt-key add /tmp/scm_prepare_node.4g47aDjH/archive.key OK END (0) Refreshing package metadata... BEGIN apt-get update Hit http://extras.ubuntu.com precise Release.gpg Hit http://extras.ubuntu.com precise Release Hit http://extras.ubuntu.com precise/main Sources Hit http://extras.ubuntu.com precise/main amd64 Packages Hit http://extras.ubuntu.com precise/main i386 Packages Ign http://extras.ubuntu.com precise/main TranslationIndex Hit http://gb.archive.ubuntu.com precise Release.gpg Hit http://gb.archive.ubuntu.com precise-updates Release.gpg Hit http://gb.archive.ubuntu.com precise-backports Release.gpg Hit http://gb.archive.ubuntu.com precise-backports/restricted amd64 Packages Hit http://gb.archive.ubuntu.com precise-backports/universe amd64 Packages Hit http://gb.archive.ubuntu.com precise-backports/multiverse amd64 Packages Hit http://gb.archive.ubuntu.com precise-backports/main i386 Packages Hit http://gb.archive.ubuntu.com precise-backports/restricted i386 Packages Hit http://gb.archive.ubuntu.com precise-backports/universe i386 Packages Hit http://gb.archive.ubuntu.com precise-backports/multiverse i386 Packages Hit http://gb.archive.ubuntu.com precise-backports/main TranslationIndex Hit http://gb.archive.ubuntu.com precise-backports/multiverse TranslationIndex Hit http://gb.archive.ubuntu.com precise-backports/restricted TranslationIndex Hit http://security.ubuntu.com precise-security Release Hit http://gb.archive.ubuntu.com precise-backports/universe TranslationIndex Hit http://gb.archive.ubuntu.com precise/main Translation-en_GB Hit http://gb.archive.ubuntu.com precise/main Translation-en Hit http://gb.archive.ubuntu.com precise/multiverse Translation-en_GB Hit http://gb.archive.ubuntu.com precise/multiverse Translation-en Hit http://gb.archive.ubuntu.com precise/restricted Translation-en_GB Hit http://gb.archive.ubuntu.com precise/restricted Translation-en Hit http://gb.archive.ubuntu.com precise/universe Translation-en_GB Hit http://gb.archive.ubuntu.com precise/universe Translation-en Hit http://gb.archive.ubuntu.com precise-updates/main Translation-en_GB Hit http://gb.archive.ubuntu.com precise-updates/main Translation-en Hit http://gb.archive.ubuntu.com precise-updates/multiverse Translation-en_GB Hit http://gb.archive.ubuntu.com precise-updates/multiverse Translation-en Hit http://gb.archive.ubuntu.com precise-updates/restricted Translation-en_GB Hit http://gb.archive.ubuntu.com precise-updates/restricted Translation-en Hit http://gb.archive.ubuntu.com precise-updates/universe Translation-en_GB Hit http://gb.archive.ubuntu.com precise-updates/universe Translation-en Hit http://gb.archive.ubuntu.com precise-backports/main Translation-en Hit http://gb.archive.ubuntu.com precise-backports/multiverse Translation-en Hit http://gb.archive.ubuntu.com precise-backports/restricted Translation-en Hit http://gb.archive.ubuntu.com precise-backports/universe Translation-en Hit http://archive.cloudera.com precise-cm5.0.1 Release Hit http://security.ubuntu.com precise-security/main Sources Hit http://security.ubuntu.com precise-security/restricted Sources Hit http://security.ubuntu.com precise-security/universe Sources Hit http://security.ubuntu.com precise-security/multiverse Sources Hit http://security.ubuntu.com precise-security/main amd64 Packages Hit http://security.ubuntu.com precise-security/restricted amd64 Packages Hit http://security.ubuntu.com precise-security/universe amd64 Packages Hit http://security.ubuntu.com precise-security/multiverse amd64 Packages Hit http://security.ubuntu.com precise-security/main i386 Packages Hit http://security.ubuntu.com precise-security/restricted Translation-en Ign http://archive.cloudera.com precise-cm5.0.1/contrib Translation-en_GB Ign http://archive.cloudera.com precise-cm5.0.1/contrib Translation-en Hit http://security.ubuntu.com precise-security/universe Translation-en Reading package lists... END (0) BEGIN apt-get update Hit http://extras.ubuntu.com precise Release.gpg Hit http://extras.ubuntu.com precise Release Hit http://extras.ubuntu.com precise/main Sources Hit http://security.ubuntu.com precise-security Release.gpg Hit http://extras.ubuntu.com precise/main amd64 Packages Hit http://extras.ubuntu.com precise/main i386 Packages Ign http://extras.ubuntu.com precise/main TranslationIndex Hit http://security.ubuntu.com precise-security Release Hit http://gb.archive.ubuntu.com precise Release.gpg Hit http://gb.archive.ubuntu.com precise-updates Release.gpg Hit http://gb.archive.ubuntu.com precise-backports Release.gpg Hit http://security.ubuntu.com precise-security/main Sources Hit http://gb.archive.ubuntu.com precise Release Hit http://gb.archive.ubuntu.com precise-updates Release Hit http://security.ubuntu.com precise-security/restricted Sources Hit http://security.ubuntu.com precise-security/universe Sources Hit http://security.ubuntu.com precise-security/multiverse Sources Hit http://security.ubuntu.com precise-security/main amd64 Packages Hit http://security.ubuntu.com precise-security/restricted amd64 Packages Hit http://security.ubuntu.com precise-security/universe amd64 Packages Hit http://security.ubuntu.com precise-security/multiverse amd64 Packages Hit http://security.ubuntu.com precise-security/main i386 Packages Hit http://security.ubuntu.com precise-security/restricted i386 Packages Hit http://security.ubuntu.com precise-security/universe i386 Packages Hit http://gb.archive.ubuntu.com precise-backports Release Hit http://security.ubuntu.com precise-security/multiverse i386 Packages Hit http://security.ubuntu.com precise-security/main TranslationIndex Hit http://gb.archive.ubuntu.com precise/main Sources Hit http://gb.archive.ubuntu.com precise/restricted Sources Hit http://gb.archive.ubuntu.com precise/universe Sources Hit http://gb.archive.ubuntu.com precise/multiverse Sources Hit http://gb.archive.ubuntu.com precise/main amd64 Packages Hit http://gb.archive.ubuntu.com precise/restricted amd64 Packages Hit http://gb.archive.ubuntu.com precise/universe amd64 Packages Hit http://gb.archive.ubuntu.com precise/multiverse amd64 Packages Hit http://gb.archive.ubuntu.com precise/main i386 Packages Hit http://gb.archive.ubuntu.com precise/restricted i386 Packages Hit http://security.ubuntu.com precise-security/multiverse TranslationIndex Hit http://security.ubuntu.com precise-security/restricted TranslationIndex Hit http://security.ubuntu.com precise-security/universe TranslationIndex Hit http://archive.cloudera.com precise-cm5.0.1 Release.gpg Hit http://security.ubuntu.com precise-security/main Translation-en Hit http://security.ubuntu.com precise-security/multiverse Translation-en Hit http://security.ubuntu.com precise-security/restricted Translation-en Hit http://gb.archive.ubuntu.com precise/universe i386 Packages Hit http://gb.archive.ubuntu.com precise/multiverse i386 Packages Hit http://gb.archive.ubuntu.com precise/main TranslationIndex Hit http://gb.archive.ubuntu.com precise/multiverse TranslationIndex Hit http://gb.archive.ubuntu.com precise/restricted TranslationIndex Hit http://security.ubuntu.com precise-security/universe Translation-en Hit http://gb.archive.ubuntu.com precise/universe TranslationIndex Hit http://gb.archive.ubuntu.com precise-updates/main Sources Hit http://gb.archive.ubuntu.com precise-updates/restricted Sources Hit http://gb.archive.ubuntu.com precise-updates/universe Sources Hit http://gb.archive.ubuntu.com precise-updates/multiverse Sources Ign http://extras.ubuntu.com precise/main Translation-en_GB Hit http://gb.archive.ubuntu.com precise-updates/main amd64 Packages Hit http://gb.archive.ubuntu.com precise-updates/restricted amd64 Packages Hit http://gb.archive.ubuntu.com precise-updates/universe amd64 Packages Hit http://gb.archive.ubuntu.com precise-updates/multiverse amd64 Packages Hit http://gb.archive.ubuntu.com precise-updates/main i386 Packages Hit http://gb.archive.ubuntu.com precise-updates/restricted i386 Packages Hit http://gb.archive.ubuntu.com precise-updates/universe i386 Packages Hit http://gb.archive.ubuntu.com precise-updates/multiverse i386 Packages Ign http://extras.ubuntu.com precise/main Translation-en Hit http://gb.archive.ubuntu.com precise-updates/main TranslationIndex Hit http://gb.archive.ubuntu.com precise-updates/multiverse TranslationIndex Hit http://archive.cloudera.com precise-cm5.0.1 Release Hit http://gb.archive.ubuntu.com precise-updates/restricted TranslationIndex Hit http://gb.archive.ubuntu.com precise-updates/universe TranslationIndex Hit http://gb.archive.ubuntu.com precise-backports/main Sources Hit http://gb.archive.ubuntu.com precise-backports/restricted Sources Hit http://gb.archive.ubuntu.com precise-backports/universe Sources Hit http://gb.archive.ubuntu.com precise-backports/multiverse Sources Hit http://gb.archive.ubuntu.com precise-backports/main amd64 Packages Hit http://gb.archive.ubuntu.com precise-backports/restricted amd64 Packages Hit http://gb.archive.ubuntu.com precise-backports/universe amd64 Packages Hit http://gb.archive.ubuntu.com precise-backports/multiverse amd64 Packages Hit http://gb.archive.ubuntu.com precise-backports/main i386 Packages Hit http://gb.archive.ubuntu.com precise-backports/restricted i386 Packages Hit http://gb.archive.ubuntu.com precise-backports/universe i386 Packages Hit http://gb.archive.ubuntu.com precise-backports/multiverse i386 Packages Hit http://gb.archive.ubuntu.com precise-backports/main TranslationIndex Hit http://gb.archive.ubuntu.com precise-backports/multiverse TranslationIndex Hit http://gb.archive.ubuntu.com precise-backports/restricted TranslationIndex Hit http://gb.archive.ubuntu.com precise-backports/universe TranslationIndex Hit http://gb.archive.ubuntu.com precise/main Translation-en_GB Hit http://gb.archive.ubuntu.com precise/main Translation-en Hit http://gb.archive.ubuntu.com precise/multiverse Translation-en_GB Hit http://gb.archive.ubuntu.com precise/multiverse Translation-en Hit http://gb.archive.ubuntu.com precise/restricted Translation-en_GB Hit http://gb.archive.ubuntu.com precise/restricted Translation-en Hit http://gb.archive.ubuntu.com precise/universe Translation-en_GB Hit http://gb.archive.ubuntu.com precise/universe Translation-en Hit http://gb.archive.ubuntu.com precise-updates/main Translation-en_GB Hit http://gb.archive.ubuntu.com precise-updates/main Translation-en Hit http://gb.archive.ubuntu.com precise-updates/multiverse Translation-en_GB Hit http://gb.archive.ubuntu.com precise-updates/multiverse Translation-en Hit http://gb.archive.ubuntu.com precise-updates/restricted Translation-en_GB Hit http://archive.cloudera.com precise-cm5.0.1/contrib amd64 Packages Hit http://gb.archive.ubuntu.com precise-updates/restricted Translation-en Hit http://gb.archive.ubuntu.com precise-updates/universe Translation-en_GB Hit http://gb.archive.ubuntu.com precise-updates/universe Translation-en Hit http://gb.archive.ubuntu.com precise-backports/main Translation-en Hit http://gb.archive.ubuntu.com precise-backports/multiverse Translation-en Ign http://archive.cloudera.com precise-cm5.0.1/contrib TranslationIndex Hit http://gb.archive.ubuntu.com precise-backports/restricted Translation-en Hit http://gb.archive.ubuntu.com precise-backports/universe Translation-en Ign http://archive.cloudera.com precise-cm5.0.1/contrib Translation-en_GB Ign http://archive.cloudera.com precise-cm5.0.1/contrib Translation-en Reading package lists... END (0) Installing oracle-j2sdk1.6 package... BEGIN dpkg -l oracle-j2sdk1.6 | grep -E '^ii[[:space:]]*oracle-j2sdk1.6[[:space:]]*' ii oracle-j2sdk1.6 1.6.0+update31 Java(TM) JDK, Standard Edition, Oracle(TM) END (0) BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep oracle-j2sdk1.6 END (0) oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons BEGIN apt-cache show oracle-j2sdk1.6 Package: oracle-j2sdk1.6 Version: 1.6.0+update31 Architecture: amd64 Maintainer: Debian Java Maintainers <pkg-java-maintainers@lists.alioth.debian.org> Installed-Size: 224994 Depends: libasound2 (>= 1.0.16), libc6 (>= 2.2.5), libx11-6, libxext6, libxi6, libxt6, libxtst6 Recommends: netbase, libgtk1.2, libstdc++5 Replaces: oracle-j2sdk1.6debian Provides: j2re1.6, j2sdk1.6, java-browser-plugin, java-compiler, java-runtime, java-runtime-headless, java-sdk, java-virtual-machine, java2-compiler, java2-runtime, java2-runtime-headless, java2-sdk Priority: optional Section: non-free/devel Filename: pool/contrib/o/oracle-j2sdk1.6/oracle-j2sdk1.6_1.6.0+update31_amd64.deb Size: 96896162 SHA256: e3af578020ca3d595fbf1da7f006dde783fa10ace7e147ea135e5d09a8ad32b9 SHA1: 4b4cd040b2b71f1b754180d89c8e69f5940d27a5 MD5sum: 183149f3ddc42fcc03e137bea41cf85e Description: Java(TM) JDK, Standard Edition, Oracle(TM) The Java(TM) 2 SDK is a development environment for building applications, applets, and components that can be deployed on the Java(TM) platform. . The Java(TM) 2 SDK software includes tools useful for developing and testing programs written in the Java programming language and running on the Java platform. These tools are designed to be used from the command line. Except for appletviewer, these tools do not provide a graphical user interface. . This package has been automatically created with java-package (0.50). END (0) BEGIN apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install oracle-j2sdk1.6 Reading package lists... Building dependency tree... Reading state information... oracle-j2sdk1.6 is already the newest version. 0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade. END (0) remote package oracle-j2sdk1.6 installed Installing oracle-j2sdk1.7 package... BEGIN dpkg -l oracle-j2sdk1.7 | grep -E '^ii[[:space:]]*oracle-j2sdk1.7[[:space:]]*' ii oracle-j2sdk1.7 1.7.0+update45-1 no description given END (0) BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep oracle-j2sdk1.7 oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons END (0) BEGIN apt-cache show oracle-j2sdk1.7 Package: oracle-j2sdk1.7 Version: 1.7.0+update45-1 License: unknown Vendor: root@debslave.ent.cloudera.com Architecture: amd64 Maintainer: <scm-users@cloudera.org> Installed-Size: 280104 Provides: java Homepage: http://example.com/no-uri-given Priority: extra Section: default Filename: pool/contrib/o/oracle-j2sdk1.7/oracle-j2sdk1.7_1.7.0+update45-1_amd64.deb Size: 138179316 SHA256: 058589470101f0bee52d833a021014ffeaa49570b68ba63c5735df4e0e07d526 SHA1: 64be1db632ee78f66c55d5f27e5c8addf9535f14 MD5sum: 19668aeb116fd07d9cd79711f0f9ed17 Description: no description given END (0) BEGIN apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install oracle-j2sdk1.7 Reading package lists... Building dependency tree... Reading state information... oracle-j2sdk1.7 is already the newest version. 0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade. END (0) remote package oracle-j2sdk1.7 installed Installing cloudera-manager-agent package... BEGIN dpkg -l cloudera-manager-agent | grep -E '^ii[[:space:]]*cloudera-manager-agent[[:space:]]*' ii cloudera-manager-agent 5.0.1-1.cm501.p0.257~precise-cm5.0.1 The Cloudera Manager Agent END (0) BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-agent END (0) BEGIN apt-cache show cloudera-manager-agent oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons Package: cloudera-manager-agent Source: enterprise Version: 5.0.1-1.cm501.p0.257~precise-cm5.0.1 Architecture: amd64 Maintainer: Cloudera Inc. <https://issues.cloudera.org> Installed-Size: 22483 Depends: libc6 (>= 2.15), libcomerr2 (>= 1.01), libgcc1 (>= 1:4.1.1), libgssapi-krb5-2 (>= 1.10+dfsg~), libkrb5-3 (>= 1.6.dfsg.2), libssl1.0.0 (>= 1.0.0), zlib1g (>= 1:1.2.0), lsb-base, psmisc, bash, libsasl2-modules, libsasl2-modules-gssapi-mit, libxslt1.1, libsqlite3-0, libfuse2, fuse-utils, rpcbind Homepage: http://www.cloudera.com Priority: extra Section: misc Filename: pool/contrib/e/enterprise/cloudera-manager-agent_5.0.1-1.cm501.p0.257~precise-cm5.0.1_amd64.deb Size: 5849080 SHA256: e3add46932ad38614afae865a9fd495fafa9f19573c473458cd127e68b147de9 SHA1: 4fa93da9c286f481492d556b0bed11f3e150ff4f MD5sum: 662d7bc9c505918622c324d21ecd61fc Description: The Cloudera Manager Agent The Agent is deployed to machines running services managed by Cloudera Manager. END (0) Version: 5.0.1-1.cm501.p0.257~precise-cm5.0.1 BEGIN apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-agent Reading package lists... Building dependency tree... Reading state information... cloudera-manager-agent is already the newest version. 0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade. END (0) remote package cloudera-manager-agent installed Installing cloudera-manager-daemons package... BEGIN dpkg -l cloudera-manager-daemons | grep -E '^ii[[:space:]]*cloudera-manager-daemons[[:space:]]*' ii cloudera-manager-daemons 5.0.1-1.cm501.p0.257~precise-cm5.0.1 Provides daemons for monitoring Hadoop and related tools. END (0) BEGIN echo oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-daemons oracle-j2sdk1.6 oracle-j2sdk1.7 cloudera-manager-agent cloudera-manager-daemons END (0) BEGIN apt-cache show cloudera-manager-daemons Package: cloudera-manager-daemons Source: enterprise Version: 5.0.1-1.cm501.p0.257~precise-cm5.0.1 Architecture: all Maintainer: Cloudera Inc. <https://issues.cloudera.org> Installed-Size: 438979 Homepage: http://www.cloudera.com Priority: extra Section: misc Filename: pool/contrib/e/enterprise/cloudera-manager-daemons_5.0.1-1.cm501.p0.257~precise-cm5.0.1_all.deb Size: 337113456 SHA256: 1c505a54fed3263071ae3d1ccf55592648e56370488f5bc4584eb9ba9fc91a9e SHA1: e4ab508c0a35ae4d6c2a8e8c3458dca50157a193 MD5sum: eb2d039fc7065bba02b98e6697092863 Description: Provides daemons for monitoring Hadoop and related tools. Provides daemons for monitoring Hadoop and related tools. END (0) Version: 5.0.1-1.cm501.p0.257~precise-cm5.0.1 BEGIN apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install cloudera-manager-daemons Reading package lists... Building dependency tree... Reading state information... cloudera-manager-daemons is already the newest version. 0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade. END (0) remote package cloudera-manager-daemons installed Installing Unlimited Strength Encryption policy files. BEGIN dpkg -L oracle-j2sdk1.6 | grep "/usr/lib/jvm/j" | sort | head -n 1 /usr/lib/jvm/j2sdk1.6-oracle END (0) BEGIN dpkg -L oracle-j2sdk1.7 | grep "/usr/lib/jvm/j" | sort | head -n 1 /usr/lib/jvm/java-7-oracle-cloudera END (0) Java 6 prefix is /usr/lib/jvm/j2sdk1.6-oracle Java 7 prefix is /usr/lib/jvm/java-7-oracle-cloudera Installing unlimited strength local_policy.jar for Java 6 BEGIN cp /tmp/scm_prepare_node.4g47aDjH/local_policy.jar.6 /usr/lib/jvm/j2sdk1.6-oracle/jre/lib/security/local_policy.jar END (0) Installing unlimited strength US_export_policy.jar for Java 6 BEGIN cp /tmp/scm_prepare_node.4g47aDjH/US_export_policy.jar.6 /usr/lib/jvm/j2sdk1.6-oracle/jre/lib/security/US_export_policy.jar END (0) Installing unlimited strength local_policy.jar for Java 7 BEGIN cp /tmp/scm_prepare_node.4g47aDjH/local_policy.jar.7 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/security/local_policy.jar END (0) Installing unlimited strength US_export_policy.jar for Java 7 BEGIN cp /tmp/scm_prepare_node.4g47aDjH/US_export_policy.jar.7 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/security/US_export_policy.jar END (0) Configuring Cloudera Manager Agent... BEGIN grep server_host=192.168.1.100 /etc/cloudera-scm-agent/config.ini END (0) scm agent is already configured Starting Cloudera Manager Agent... BEGIN /usr/sbin/service cloudera-scm-agent status | grep running server_host=192.168.1.100 END (0) BEGIN /usr/sbin/service cloudera-scm-agent restart Checking for service cloudera-scm-agent * cloudera-scm-agent is running Stopping cloudera-scm-agent: * cloudera-scm-agent stopped Starting cloudera-scm-agent: * cloudera-scm-agent started END (0) agent logs: BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/' >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO SCM Agent Version: 5.0.1 >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Agent Protocol Version: 4 >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Using Host ID: 587d1ad4-7da3-4cd4-948e-1413ff582cea >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Using directory: /run/cloudera-scm-agent >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Using supervisor binary path: /usr/lib/cmf/agent/src/cmf/../../build/env/bin/supervisord >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Adding env vars that start with CMF_AGENT_ >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log END (0) BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/' >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO SCM Agent Version: 5.0.1 >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Agent Protocol Version: 4 >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Using Host ID: 587d1ad4-7da3-4cd4-948e-1413ff582cea >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Using directory: /run/cloudera-scm-agent >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Using supervisor binary path: /usr/lib/cmf/agent/src/cmf/../../build/env/bin/supervisord >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Adding env vars that start with CMF_AGENT_ >>[29/May/2014 21:49:24 +0000] 5396 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log >>[29/May/2014 21:49:26 +0000] 5396 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/memory >>[29/May/2014 21:49:26 +0000] 5396 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpu >>[29/May/2014 21:49:26 +0000] 5396 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpuacct >>[29/May/2014 21:49:26 +0000] 5396 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/blkio >>[29/May/2014 21:49:26 +0000] 5396 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True} >>[29/May/2014 21:49:26 +0000] 5396 MainThread agent INFO Setting up supervisord event monitor. >>[29/May/2014 21:49:26 +0000] 5396 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] >>[29/May/2014 21:49:26 +0000] 5396 MainThread filesystem_map INFO Using timeout of 2.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread filesystem_map INFO Using join timeout of 0.100000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread filesystem_map INFO Using tolerance of 60.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib/cmf/agent/src/cmf/monitor/tasktracker/schema.json >>[29/May/2014 21:49:26 +0000] 5396 MainThread dns_names INFO Using timeout of 2.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread ntp_monitor INFO Using timeout of 2.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread __init__ INFO Importing metric schema from file /usr/lib/cmf/agent/src/cmf/monitor/schema.json >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service', 'CDH_SPARK_HOME': '/usr/lib/spark', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_HUE_HOME': '/usr/lib/hue', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_FLUME_HOME': '/usr/lib/flume-ng', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'CDH_HCAT_HOME':>>[29/May/2014 21:49:26 +0000] 5396 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/memory >>[29/May/2014 21:49:26 +0000] 5396 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpu >>[29/May/2014 21:49:26 +0000] 5396 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpuacct >>[29/May/2014 21:49:26 +0000] 5396 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/blkio >>[29/May/2014 21:49:26 +0000] 5396 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft_limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True} >>[29/May/2014 21:49:26 +0000] 5396 MainThread agent INFO Setting up supervisord event monitor. >>[29/May/2014 21:49:26 +0000] 5396 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] >>[29/May/2014 21:49:26 +0000] 5396 MainThread filesystem_map INFO Using timeout of 2.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread filesystem_map INFO Using join timeout of 0.100000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread filesystem_map INFO Using tolerance of 60.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib/cmf/agent/src/cmf/monitor/tasktracker/schema.json >>[29/May/2014 21:49:26 +0000] 5396 MainThread dns_names INFO Using timeout of 2.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread ntp_monitor INFO Using timeout of 2.000000 >>[29/May/2014 21:49:26 +0000] 5396 MainThread __init__ INFO Importing metric schema from file /usr/lib/cmf/agent/src/cmf/monitor/schema.json >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service', 'CDH_SPARK_HOME': '/usr/lib/spark', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_HUE_HOME': '/usr/lib/hue', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_FLUME_HOME': '/usr/lib/flume-ng', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_LLAMA_HOME': '/usr/lib/llama/'} >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/process >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent ERROR Failed to connect to previous supervisor. >>Traceback (most recent call last): >> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 1236, in find_or_start_supervisor >> self.get_supervisor_process_info() >> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 1423, in get_supervisor_process_info >> self.identifier = self.supervisor_client.supervisor.getIdentification() >> File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__ >> return self.__send(self.__name, args) >> File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request >> verbose=self.__verbose >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request >> self.connection.request('POST', handler, request_body, self.headers) >> File "/usr/lib/python2.7/httplib.py", line 958, in request >> self._send_request(method, url, body, headers) >> File "/usr/lib/python2.7/httplib.py", line 992, in _send_request >> self.endheaders(body) >> File "/usr/lib/python2.7/httplib.py", line 954, in endheaders >> self._send_output(message_body) >> File "/usr/lib/python2.7/httplib.py", line 814, in _send_output >> self.send(msg) >> File "/usr/lib/python2.7/httplib.py", line 776, in send >> self.connect() >> File "/usr/lib/python2.7/httplib.py", line 757, in connect >> self.timeout, self.source_address) >> File "/usr/lib/python2.7/socket.py", line 571, in create_connection >> raise err >>error: [Errno 111] Connection refused >>[29/May/2014 21:49:27 +0000] 5396 MainThread tmpfs INFO Reusing mounted tmpfs at /run/cloudera-scm-agent/process '/usr/lib/hive-hcatalog', 'CDH_LLAMA_HOME': '/usr/lib/llama/'} >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/process >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent ERROR Failed to connect to previous supervisor. >>Traceback (most recent call last): >> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 1236, in find_or_start_supervisor >> self.get_supervisor_process_info() >> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 1423, in get_supervisor_process_info >> self.identifier = self.supervisor_client.supervisor.getIdentification() >> File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__ >> return self.__send(self.__name, args) >> File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request >> verbose=self.__verbose >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request >> self.connection.request('POST', handler, request_body, self.headers) >> File "/usr/lib/python2.7/httplib.py", line 958, in request >> self._send_request(method, url, body, headers) >> File "/usr/lib/python2.7/httplib.py", line 992, in _send_request >> self.endheaders(body) >> File "/usr/lib/python2.7/httplib.py", line 954, in endheaders >> self._send_output(message_body) >> File "/usr/lib/python2.7/httplib.py", line 814, in _send_output >> self.send(msg) >> File "/usr/lib/python2.7/httplib.py", line 776, in send >> self.connect() >> File "/usr/lib/python2.7/httplib.py", line 757, in connect >> self.timeout, self.source_address) >> File "/usr/lib/python2.7/socket.py", line 571, in create_connection >> raise err >>error: [Errno 111] Connection refused >>[29/May/2014 21:49:27 +0000] 5396 MainThread tmpfs INFO Reusing mounted tmpfs at /run/cloudera-scm-agent/process END (0) end of agent logs. scm agent restarted Installation script completed successfully. all done closing logging file descriptor 1 2 3 4 5 6 Could you please help if I have done something wrong? Cheers, -Kiran.
... View more
05-29-2014
02:00 PM
Hi I am trying CDH5 automatic installation. I have created two ubuntu Precise 12.04 LTS desktop computers using oracle virtualbox. I have built the ubuntu virtual machines as follows 1. Bridge Networked virtual machines. ifconfig gives me the following ips 1.a 192.168.1.100 & 1.b 192.168.1.103 2. The following commands are executed on the two ubuntun machines 2.a $ sudo passwd root - to enable root access 2.b $ sudo ufw allow 7180 - to open port 7180 2.c $ sudo ufw allow 7182 - to open port 7182 2.d $ sudo ufw allow 9000 - to open port 9000 2.e $ sudo ufw allow 9001 - to open port 9001 2.f $ sudo apt-get install openssh-server - to install openssh 2.g $ sudo ufw disable - to disable the firewall completely as I couldn't progress by just opening 7180, 7182, 9000 and 9001 ports 3. Edited /etc/hosts files on both machines to have the following lines 127.0.0.1 localhost 192.168.1.100 cloudera1-VirtualBox cloudera1 192.168.1.103 cloudera2-virtualBox cloudera2 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 4. $ sudo chmod u+x cloudera-manager-installer.bin 5. $ sudo ./cloudera-manager-installer.bin and followed the on screen instructions as instructions on per http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Installation-Guide/cm5ig_install_path_A.html?scroll=cmig_topic_6_5 6. Cloudera manager installed and the firefox browser started and after a while I could access the console using http://192.168.1.100:7180/cmf/login admin/admin 7. Specified the hosts for the cloudera cluster installation. 8. cluster installation progressed on both hosts i.e 192.168.1.100 & 192.168.1.103 9. Installation failed at the heartbeat step for 192.168.1.100 but I couldn't find any errors on the details. So I clicked retry and the installation is complete second time round for this node. The installation was still progressing on second node 192.168.1.103. Finally this installation also failed at the heartbeat step. Clicked retry but no luck second time, third time as well. The following is the excerpt from the details. >>[29/May/2014 21:49:27 +0000] 5396 MainThread agent ERROR Failed to connect to previous supervisor. >>Traceback (most recent call last): >> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 1236, in find_or_start_supervisor >> self.get_supervisor_process_info() >> File "/usr/lib/cmf/agent/src/cmf/agent.py", line 1423, in get_supervisor_process_info >> self.identifier = self.supervisor_client.supervisor.getIdentification() >> File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__ >> return self.__send(self.__name, args) >> File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request >> verbose=self.__verbose >> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request >> self.connection.request('POST', handler, request_body, self.headers) >> File "/usr/lib/python2.7/httplib.py", line 958, in request >> self._send_request(method, url, body, headers) >> File "/usr/lib/python2.7/httplib.py", line 992, in _send_request >> self.endheaders(body) >> File "/usr/lib/python2.7/httplib.py", line 954, in endheaders >> self._send_output(message_body) >> File "/usr/lib/python2.7/httplib.py", line 814, in _send_output >> self.send(msg) >> File "/usr/lib/python2.7/httplib.py", line 776, in send >> self.connect() >> File "/usr/lib/python2.7/httplib.py", line 757, in connect >> self.timeout, self.source_address) >> File "/usr/lib/python2.7/socket.py", line 571, in create_connection >> raise err >>error: [Errno 111] Connection refused >>[29/May/2014 21:49:27 +0000] 5396 MainThread tmpfs INFO Reusing mounted tmpfs at /run/cloudera-scm-agent/process 10. I checked $ sudo ufw status and the result in inactive on both virtual machines 11. when I check $ sudo service cloudera-scm-server status on 192.168.1.103 it comes as unrecognized service. Run the command on 192.168.1.100, it says running. Could you please let me know where I went wrong in installing cloudera in a clustered environment? Thanks & Regards, -K
... View more
