hi can I have the parcels located on an FS so the installation will include a copy instead of downloading from a remote URL? I believe it will be faster solution this is for testing cluster environments better still - can I copy the parcels to the /opt/cloudera/parcels myself - while I'm preparing the VM - will cdp know that the file is already there or do I need to update also the psql metadata db? thanks doron ... View more

hi I created a CentOS 7.7 Linux machine (but also happens with RHEL 7.4) on Azure. I installed the cloudera-manager RPM packages via yum When i try to start the cloudera-scm-server-db service I get "permission denied" on /var/lib/cloudera-scm-server-db/data I added -x flag to the /usr/share/cmf/bin/initialize_embedded_db.sh, and added the "strace" command before the initdb command. I see that the command is running as cloudera-scm user (in the /var/log/messages file) and Is see in the strace output, that it fails on "chmod("/var/lib/cloudera-scm-server-db/data", 0700) = -1 EACCES (Permission denied)" When I run the command from Linux CLI as user cloudera-scm it runs OK. Does the Linux behave differently on Azure (i.e. more security features) I don't think it relates to firewall I see in the /var/log/messages the following messages: Oct 22 12:53:14 cdh-doxi-1 systemd: Created slice User Slice of cloudera-scm. Oct 22 12:53:14 cdh-doxi-1 systemd: Started Session c1 of user cloudera-scm. Oct 22 12:53:16 cdh-doxi-1 systemd: Removed slice User Slice of cloudera-scm. could this slices are doing the damage? I see it relates to cgroups (but I'm not sure what it actually means) - can I turn this off somewhere? do i need to reinstall the Linux with different flags? TIA Doron ... View more

hi in the upgrade documentation the following is stated: "Upgrades from Cloudera Enterprise 5.15 to 6.0.0 Not supported" does it mean specifically to cdh 6.0.0 version or to any 6.x version? thanks Doron ... View more

hi We've got the following error: org.apache.openjpa.persistence.PersistenceException: ERROR: value too long for type character varying(255) {prepstmnt 190305649 INSERT INTO WF_JOBS (id, app_name, app_path, conf, created_time, end_time, external_id, group_name, last_modified_time, log_token, parent_id, proto_action_conf, run, sla_xml, start_time, status, user_name, wf_instance) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)} [code=0, state=22001] and to solve it we've altered the WF_JOBS table, resizing the app_path and app_name to be varchar(1024). Is it acceptable to modify the oozie database? Will it cause us problems in other parts of oozie? We did while using postgreSQL for oozie db. Thanks Doron ... View more

In our testing environments we're aleays short on space... So we thought that we can remove the /opt/cloudera/parcel-cache and /opt/cloudera/parcel-repo - after all the parcel is already installed and distributed. Big mistake... i failed to start the cloudera manager services and got error like: ========= [29/Jun/2015 10:57:41 +0000] 2917 MainThread util INFO Using generic parser for process cloudera-mgmt-HOSTMONITOR [29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Activating Process 1019-cloudera-mgmt-HOSTMONITOR [29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Created /var/run/cloudera-scm-agent/process/1019-cloudera-mgmt-HOSTMONITOR [29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Chowning /var/run/cloudera-scm-agent/process/1019-cloudera-mgmt-HOSTMONITOR to cloudera-scm (781) cloudera-scm (501) [29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/1019-cloudera-mgmt-HOSTMONITOR to 0751 [29/Jun/2015 10:57:41 +0000] 2917 MainThread agent ERROR Failed to activate {u'refresh_files': [u'cloudera-stack-monitor.properties'], u'config_generation': 0, u'auto_restart': True, u'running': True, u'required_tags': [], u'user': u'cloudera-scm', u'special_file_info': [], u'group': u'cloudera-scm', u'id': 1019, u'status_links': {u'status': u'http://ilmtxbda146.corp.amdocs.com:8091/'}, u'name': u'cloudera-mgmt-HOSTMONITOR', u'run_generation': 1, u'environment': {u'MGMT_LOG_FILE': u'mgmt-cmf-mgmt-HOSTMONITOR-ilmtxbda146.corp.amdocs.com.log.out', u'CDH_VERSION': u'-1', u'FIREHOSE_JAVA_OPTS': u'-Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError={{AGENT_COMMON_DIR}}/killparent.sh'}, u'optional_tags': [], u'program': u'mgmt/mgmt.sh', u'arguments': [u'firehose', u'--pipeline-type', u'HOST_MONITORING'], u'parcels': {u'CDH': u'5.3.0-1.cdh5.3.0.p0.30'}, u'resources': [{u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': False, u'rlimits': {u'limit_memlock': None, u'limit_fds': None}, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/log/cloudera-scm-firehose', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 509}, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'10.233.164.119', u'port': 8091}, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'10.233.164.119', u'port': 9995}, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'10.233.164.119', u'port': 9994}, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/lib/cloudera-host-monitor', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 509}, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': True, u'rlimits': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/log/cloudera-scm-firehose/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 493}, u'cpu': None}], u'one_off': False} Traceback (most recent call last): File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1234, in handle_heartbeat_response new_process.activate() File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2438, in activate self.update_process_environment_and_parcels() File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2601, in update_process_environment_and_parcels parcels_in_use = self.agent.repo.prepare_environment(self.raw['parcels'], AttributeError: 'NoneType' object has no attribute 'prepare_environment' [29/Jun/2015 10:57:57 +0000] 2917 CP Server Thread-12 _cplogging INFO 10.233.164.119 - - [29/Jun/2015:10:57:57] "GET /download_log?path=%2Fvar%2Flog%2Fcloudera-scm-firehose%2Fmgmt-cmf-mgmt-HOSTMONITOR-ilmtxbda146.corp.amdocs.com.log.out&onlyTail=true&compress=false HTTP/1.1" 200 2011 "" "Java/1.7.0_10" [29/Jun/2015 11:07:28 +0000] 2917 MonitorDaemon-Reporter throttling_logger ERROR (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-14f3c3b6446d19e4dcc4dab16355bb71 Traceback (most recent call last): File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 70, in _send self._port) File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 464, in __init__ self.conn.connect() File "/usr/lib64/python2.6/httplib.py", line 720, in connect self.timeout) File "/usr/lib64/python2.6/socket.py", line 567, in create_connection raise error, msg error: [Errno 111] Connection refused =========   Luckily for me, I had the same parcel on a different environment so I could copy it and environment was happy again.   My questions for cloudera people: * Why do I need the parcels after it was successfully installed? (all the jars and exe are available from /opt/cloudera/parcels/CDH) * Do you have any suggestions for saving space? - again this is for small testing environments and not for production.   Thanks Doron ... View more

Can I update oozie-site.xml using REST api?   We added our own external function to oozie. After each installation we need to go manually and change things in oozie via the CM web interface: Oozie Server Configuration Safety Valve for oozie-site.xml - add two properties oozie.service.ActionService.executor.ext.classes - add additional value oozie SchemaService Workflow Extension Schemas - add additional value i'd like to be able to do this via scripting of any kind.   So far i could not find such an option - neither in CM REST API (http://cloudera.github.io/cm_api/apidocs/v10/index.html) nor OOZIE REST API (http://archive.cloudera.com/cdh4/cdh/4/oozie/WebServicesAPI.html)   Thanks Doron ... View more

We have the postgres db crashes occasional. There is no core dump. Can we add the -c flag to the pg_ctl command in the /etc/init.d/cloudera-scm-server-db file? $SU -c "pg_ctl start -c -w -D $DATA_DIR -l $SERVER_LOGFILE"   can we keep this change permanent?   Thanks Doron ... View more