Member since
02-22-2016
5
Posts
0
Kudos Received
0
Solutions
03-07-2016
07:54 AM
Quick follow-up: I have confirmed that the combination of updating /etc/cloudera-scm-agent/config.ini AND adding --lib_dir= to /etc/defaults/cloudera-scm-agent successfully corrects the startup problem when relocating /var/lib/cloudera-scm-agent. Here is what I did: 1. Stop agent: sudo service cloudera-scm-agent stop 2. Update /etc/cloudera-scm-agent/config.ini: # Particularly, the agent's UUID is stored here. # lib_dir=/var/lib/cloudera-scm-agent lib_dir=/NEW_DIR_PATH 3. Update /etc/default/cloudera-scm-agent: # Specify any command line arguments for the Cloudera SCM Agent here. # CMF_AGENT_ARGS="--lib_dir=/NEW_DIR_PATH " 4. Copy old content of /var/lib/coudera-scm-agent to new location /NEW_DIR_PATH: sudo cp /var/lib/cloudera-scm-agent/* /NEW_DIR_PATH 5. Rename old content (in case we mess up): sudo mv /var/lib/cloudera-scm-agent /var/lib/cloudera-scm-agent-del 6. Restart agent: sudo /sbin/service cloudera-scm-agent start 7. If you are happy and you know it, delete the old lib: sudo rm-rf /var/lib/cloudera-scm-agent-del I am happy. And I appreciate the really quick turn-around!! -Hugh
... View more
03-03-2016
08:02 AM
Hi, Harsh! At least I am not losing my mind 🙂 Ok. I will set this up in my lab and let you know what the outcome is. Thank you for getting back to me on this!! Regards! -Hugh
... View more
02-26-2016
02:38 PM
CM 552 SCM Agent Version: 5.5.3 Centos5.11 We needed to relocate the agent outputs from /var/log and /var/lib because of space restrictions on the root mount. Using config.ini, we were able to sucessfully move log_file to a new location. However, changes to the lib_dir property are wholly ignored and we are fairly sure we are not victims of a phat-phinger attack. Could someone check on this? Thanks! Hugh
... View more
02-26-2016
02:27 PM
Hi, Harsh! Thank you for replying. Here is the info you requested: [jamiesoh@cdhcm01dxdu]$ll /opt/cloudera/parcel-repo/ total 1418740 drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Feb 16 14:05 . drwxr-xr-x 5 cloudera-scm cloudera-scm 4096 Feb 24 11:44 .. -rw-r----- 1 cloudera-scm cloudera-scm 1452773214 Feb 23 14:54 CDH-5.5.2-1.cdh5.5.2.p0.4-el5.parcel -rw-r----- 1 cloudera-scm cloudera-scm 41 Feb 23 14:54 CDH-5.5.2-1.cdh5.5.2.p0.4-el5.parcel.sha [jamiesoh@cdhcm01dxdu]$ However, I read that some of the cm installer packages had been recalled, so I wiped everthing and started over by downloading the 5.5.2 installer for CM. I had much better results with this install. I still see the messages regarding the installed JDKs (I deleted all but 2), so the log now shows: [26/Feb/2016 17:23:18 +0000] 8933 MainThread version_detector INFO Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_75 java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) for requested version 7. [26/Feb/2016 17:23:18 +0000] 8933 MainThread version_detector INFO Identified java component java8 with full version JAVA_HOME=/usr/java/jdk1.8.0_45 java version "1.8.0_45" Java(TM) SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) for requested version 8. [26/Feb/2016 17:25:18 +0000] 8933 MainThread version_detector INFO Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_75 java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) for requested version 7. [26/Feb/2016 17:25:18 +0000] 8933 MainThread version_detector INFO Identified java component java8 with full version JAVA_HOME=/usr/java/jdk1.8.0_45 java version "1.8.0_45" Java(TM) SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) for requested version 8. This seems to have no effect on the agent activity; it's just logging. Thank you! Hugh
... View more
02-22-2016
09:47 AM
CM Agent 5.5.2 Centos7 During cluster node installation, agent is successfully installed and started, and does some activity, but the CM screen displays "Installing Selected Parcels..." for days. Each cloudera-scm-agent log in /var/log has the following at the end, repeatedly: [22/Feb/2016 12:35:26 +0000] 17982 DnsResolutionMonitor throttling_logger INFO Using java location: '/usr/java/jdk1.7.0_75/bin/java'. [22/Feb/2016 12:36:57 +0000] 17982 MainThread version_detector INFO Identified java component java6 with full version JAVA_HOME=/usr/java/jdk1.6_prev java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) for requested version 6. [22/Feb/2016 12:36:57 +0000] 17982 MainThread version_detector INFO Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_75 java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) for requested version 7. [22/Feb/2016 12:36:57 +0000] 17982 MainThread version_detector INFO Identified java component java8 with full version JAVA_HOME=/usr/java/jdk1.8.0_45 java version "1.8.0_45" Java(TM) SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) for requested version 8. [22/Feb/2016 12:38:57 +0000] 17982 MainThread version_detector INFO Identified java component java6 with full version JAVA_HOME=/usr/java/jdk1.6_prev java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) for requested version 6. [22/Feb/2016 12:38:57 +0000] 17982 MainThread version_detector INFO Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_75 java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) for requested version 7. [22/Feb/2016 12:38:57 +0000] 17982 MainThread version_detector INFO Identified java component java8 with full version JAVA_HOME=/usr/java/jdk1.8.0_45 java version "1.8.0_45" Java(TM) SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) for requested version 8. [22/Feb/2016 12:40:57 +0000] 17982 MainThread version_detector INFO Identified java component java6 with full version JAVA_HOME=/usr/java/jdk1.6_prev java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) for requested version 6. [22/Feb/2016 12:40:57 +0000] 17982 MainThread version_detector INFO Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_75 java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) for requested version 7. [22/Feb/2016 12:40:57 +0000] 17982 MainThread version_detector INFO Identified java component java8 with full version JAVA_HOME=/usr/java/jdk1.8.0_45 java version "1.8.0_45" Java(TM) SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) for requested version 8. The tail of cloudera-scm-server: 2016-02-22 12:46:02,751 WARN 1821765743@scm-web-306:com.cloudera.server.web.cmf.WebController: Unknown parcel CDH-5.5.1-1.cdh5.5.1.p0.11 for cluster Cluster 1 2016-02-22 12:46:04,751 WARN 1821765743@scm-web-306:com.cloudera.server.web.cmf.WebController: Unknown parcel CDH-5.5.1-1.cdh5.5.1.p0.11 for cluster Cluster 1 2016-02-22 12:46:08,268 WARN 1821765743@scm-web-306:com.cloudera.server.web.cmf.WebController: Unknown parcel CDH-5.5.1-1.cdh5.5.1.p0.11 for cluster Cluster 1 2016-02-22 12:46:19,751 WARN 1821765743@scm-web-306:com.cloudera.server.web.cmf.WebController: Unknown parcel CDH-5.5.1-1.cdh5.5.1.p0.11 for cluster Cluster 1 2016-02-22 12:46:29,816 WARN 1821765743@scm-web-306:com.cloudera.server.web.cmf.WebController: Unknown parcel CDH-5.5.1-1.cdh5.5.1.p0.11 for cluster Cluster 1 2016-02-22 12:46:39,882 WARN 1821765743@scm-web-306:com.cloudera.server.web.cmf.WebController: Unknown parcel CDH-5.5.1-1.cdh5.5.1.p0.11 for cluster Cluster 1 In fact, there are many JDKs installed on each of these machines. Not exactly sure how to get process to move forward.
... View more
02-22-2016
07:59 AM
Hi! My name is Hugh Jamieson and I am the Hadoop Principle Engineer at OCLC.org, a non-profit org that serves libraries around the world. We are heavily invested in big data to support the data processes our community needs, processes that previously used to take months to complete! We manage library information at all levels and support sharing of resources at cloud scale. We have a considerable investment in HBase and are true believers. Like many organizations, we are struggling with the appetite our org has for big data and the velocity at which new projects arrive. We have many clusters arranged around geographical and organizational boundaries, and managing these little beauties is a real chore. We are looking for ways to improve our velocity getting new features and tools into production. We have some roll-your-own utilities for cluster management that simply do not scale. So, we are looking at CM in the hope that it will help us speed up our deployments and make them much less complex. Like other big data fans, I have no life and seldom see the sun. JK; I love my job. I stumbled upon Hadoop 5 years ago and was completely hooked on its design and capabilities. I have travelled thru many environments, from mainframe to SGI, and I can say I am a total Hadoop fan-boy. Such a nerd. In my spare time I like to evangelize Scala and Spark, Streaming, performance, reactive programming, and immutability. My favorite color is blue. I have my own cluster in my basement. Yeah, I'm that sad.
... View more
