About doronve

doronve · ‎10-16-2019

Hi In our development site we have a concept of prod-like environment. These are large clusters (10-15 nodes) where we run various benchmark tests for our products. Until now we used physical servers - 2 U rack-mount servers with multiple internal disks configured as JBOD. now we have increased the requirements of the number of environments - we need to have 5-6 such environment for the latest product and we need to have available environments for existing customer with older version of our products. so, I was thinking of moving from physical servers to use ESX hypervisor and use VMs. this will enable me to use less resource and be more flexibile in the configuration. We got recommendation to use HPE Synergy, which can mix blades and Disks in the same chassis. Can anyone share if they handle similar issues and how they resolved them? Thanks Doron PS we're using both CDH and HDP (existing customers have different versions), main components are Kafka, HBase, Hive and SPARK, and additional SW like Cassandra. Elasticsearch and more.

doronve · ‎09-24-2019

We have a cluster with Name Node HA. We're using HDFS FUSE to point to the nameservice, and we configured the mount in /etc/fstab. All works well, except if we reboot the namenode server. After reboot, the name node is not up yet, but the mount point tries to be set and fails. We have to do a manual umount and mount afterwards. Is there a way to delay the mount until after the namenode is up? I've seen autofs, but I don't like the mount to be gone if not in use. Is there another way? Thanks Doron

doronve · ‎06-27-2019

thanks for the response. the error occured from java - but not from hive cli (and I did not check beelonw) recently we moved from Oracle JDK to OpenJDK - could this be an issue? for now I've added hive.exec.drop.ignorenonexistent and it seems to help

doronve · ‎06-26-2019

hi In our application (java code) we're running: DROP FUNCTION IF EXISTS udf_for_STORE_AGENT_VISIT and we still get: 2019-06-26 13:49:25,683 ERROR org.apache.hadoop.hive.ql.exec.Task: [HiveServer2-Background-Pool: Thread-6397]: FAILED: error during drop function: org.apache.hadoop.hive.ql.metadata.HiveException: NoSuchObjectException(message:Function udf_for_STORE_AGENT_VISIT does not exist) shouldn't the error be skipped if we have the "IF EXISTS" in the command? cluster is CDH 5.16.1 Thanks Doron

doronve · ‎12-11-2018

Hi I'm trying to download and install ready-made anaconda parcels for cloudera from a cloudera site (https://www.cloudera.com/downloads/partner/anaconda.html) it referes to anaconda site (https://docs.continuum.io/anaconda/cloudera) the latter URL is not available is it cloudera issue or anaconda issue? thanks I've found the parcels here - https://repo.continuum.io/pkgs/misc/parcels/ but the full documentations are missing thanks doron

doronve · ‎11-26-2018

great! thanks for clarifying.

doronve · ‎11-25-2018

hi in the upgrade documentation the following is stated: "Upgrades from Cloudera Enterprise 5.15 to 6.0.0 Not supported" does it mean specifically to cdh 6.0.0 version or to any 6.x version? thanks Doron

doronve · ‎07-04-2018

@Geoffrey Shelton Okot thanks again - but we're still looking for other solution before we'll have to upgrade our code.

doronve · ‎07-03-2018

hi @Sindhu thanks again for the quick response. OK - so my question is - can I install a new HDP 2.6.4 cluster with older version of HIVE? regardless of the existing cluster. no upgrade... If i'll need the data from the old one, I'll copy all relevant data between them. Thanks Doron

doronve · ‎07-03-2018

hi Thanks for the answer. We do have smaller cluster for tests. However, we prefer to try to install older version of HIVE than upgrading our code. Is it possible at all? Thanks Doron

doronve · ‎07-03-2018

we need to upgrade to HDP 2.6.4, but our code still use older HIVE version. can we upgrade HDP and keep old HIVE? currently we're using HDP with HIVE version 0.14.0.2.2 while the new HDP brings HIVE 1.2.1000

doronve · ‎05-10-2018

I've upgraded CM to latest (5.14.3) and it works. although - i'd still like to know if it is supported on lower CM version or not.

doronve · ‎05-09-2018

hi I've followed instruction from here - https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html Downloaded parcel, replaced csd jar, distributed, activated, restarted but when restarting cloudera-scm-server I get in its log file /var/log/cloudera-scm-server/cloudera-scm-server.log: 2018-05-09 14:59:31,361 WARN main:com.cloudera.csd.components.CsdLocalRepository: Skipping [/opt/cloudera/csd/SPARK2_ON_YARN-2.3.0.cloudera2.jar]: { Could not read service descriptor for csd [SPARK2_ON_YARN-2.3.0.cloudera2]: Could not resolve type id 'provided' into a subtype of [simple type, class com.cloudera.csd.descriptors.parameters.Parameter<java.lang.Object>] at [Source: [B@44b2dfa7; line: 124, column: 45] (through reference chain: ServiceDescriptor["parameters"]) } and i cannot use the spark2 service in the cluster. In file SPARK23/scripts/common.sh I see: local MIN_CM_VERSION=$(parse_version_string "5.14.0") Should i really upgrade to latest CM version?? I prefer staying on current cm version until we decide to upgrade the whole lot. Or is there another issue here? Thanks in advance Doron

doronve · ‎02-21-2018

@Poojan Khanpara Here is what I got from HDP experts: Work-around: 1. Edit /var/lib/ambari-server/resources/common-services/HDFS/x.x.x.x.x/configuration/hdfs-site.xml 2. Modify the XML for commenting the <property-type>GROUP</property-type> tag. Like this: <property> <name>dfs.permissions.superusergroup</name> <value>hdfs</value>  <description>The name of the group of super-users.</description> <on-ambari-upgrade add="true"/> </property> 3. Restart Ambari ambari-server restart

doronve · ‎12-27-2017

hi I cannot find dfs.permissions.superusergroup via ambari (HDFS-->config) in HDP 2.6.3 When I search "dfs.permissions" I see only the parameter "dfs.permissions.enabled" If i try to add it to the custom hdfs-site I get "Line 1: Property "dfs.permissions.superusergroup" is already defined in this configuration file" Checking on the local file system I do see that the parameter exists but I cannot modify it via ambari. I have cluster with HDP 2.6.1 and it is modifiable there. OS is RHEL 7.3 Any help will be appriciated. Thanks

doronve · ‎10-18-2017

hi we're checking the encryption of hbase tables. we generate hbase.jks and put it under /etc/hbase/conf then we set hbase.crypto.keyprovider.parameters=jceks:///etc/hbase/conf/hbase.jks?password=password however, the /etc/hbase/conf/hbase.jks file is being deleted from time to time I suspect that this folder is managed by cloduera manager which cleans it up. should we locate the file in another directory? Thanks Doron

doronve · ‎10-05-2017

Hi We're using CDH 5.10.1 on RHEL7. we've started with default the spark 1.6 and then we've decided to go with spark 2.2 When we create a cluster CDH 5.10.1 with default spark and then we upgrade to spark 2.2 (add percel, add csd, and keep following innstructions), the folder /user/spark/spark2ApplicationHistory has permissions "drwxrwxrwt" However, whne we install the cluster and install directly spark 2.2, without the default one first, the same folder gets "drwxr-xr-x" permissions. (both time owner and group are "spark") i guess some chmod command is missing during the installation path of spark2.2, or do we do something incorrect? did someone got same issues? this happened in our testing environments and caused some spark jobs to fail, until we changed the permissions mode of the folder. thanks Doron

doronve · ‎08-04-2016

Hi I have an existing cluster with full HA and securtity is on using kerberos. The Hive metastore was on PostgreSQL and I'm moving it to Oracle. I managed to run "Create the Hive Metastore Database Tables" But when I fail with "Update Hive Metastore NameNodes" It seems like it cannot connect the database, or that it fails with the kerberos ticket. Had anyone try this scenario? Using CDH 5.5.1 on RHEL 7.1 thanks Doron

doronve · ‎05-29-2016

Robert thanks for your reply. Indeed, I fear the exact issues you mentioned of future upgrades of this DB. However the names of the workflows are generated by some code of ours and they tend to become very long - more than 256. I'm trying to convince or development team to shorten this but it will take them time. Now - except for future changes in the database - could it be that other parts of oozie might be affected? some code that reads this table and has hard-coded 256 arrays size instead of strings? If there is no such issue, we'll go with our modification - we did the following: ALTER TABLE wf_jobs ALTER COLUMN app_path TYPE varchar(1024); ALTER TABLE wf_jobs ALTER COLUMN app_name TYPE varchar(1024); I do hope you can reproduce the issue and recommend this change. 🙂 Thanks Doron

doronve · ‎05-22-2016

hi We've got the following error: org.apache.openjpa.persistence.PersistenceException: ERROR: value too long for type character varying(255) {prepstmnt 190305649 INSERT INTO WF_JOBS (id, app_name, app_path, conf, created_time, end_time, external_id, group_name, last_modified_time, log_token, parent_id, proto_action_conf, run, sla_xml, start_time, status, user_name, wf_instance) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)} [code=0, state=22001] and to solve it we've altered the WF_JOBS table, resizing the app_path and app_name to be varchar(1024). Is it acceptable to modify the oozie database? Will it cause us problems in other parts of oozie? We did while using postgreSQL for oozie db. Thanks Doron

doronve · ‎05-22-2016

hi We've got the following error: org.apache.openjpa.persistence.PersistenceException: ERROR: value too long for type character varying(255) {prepstmnt 190305649 INSERT INTO WF_JOBS (id, app_name, app_path, conf, created_time, end_time, external_id, group_name, last_modified_time, log_token, parent_id, proto_action_conf, run, sla_xml, start_time, status, user_name, wf_instance) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)} [code=0, state=22001] and to solve it we've altered the WF_JOBS table, resizing the app_path and app_name to be varchar(1024). Is it acceptable to modify the oozie database? Will it cause us problems in other parts of oozie? We did while using postgreSQL for oozie db. Thanks Doron

doronve · ‎06-29-2015

Hi Thanks for the quick replay. I knew that the fault was mine, but it took me a while to find it. What I would like to be able to do is to have the parcels located on a shared storage, so I cam have a single copy of the parcel file (1.5GB per parcel) for my various clusters. In any case i have my testing clusters installed on VMs, so it does make sense to do this for them. For real Production environment i can't argue with your solution. Cheers Doron

doronve · ‎06-29-2015

In our testing environments we're aleays short on space... So we thought that we can remove the /opt/cloudera/parcel-cache and /opt/cloudera/parcel-repo - after all the parcel is already installed and distributed. Big mistake... i failed to start the cloudera manager services and got error like: ========= [29/Jun/2015 10:57:41 +0000] 2917 MainThread util INFO Using generic parser for process cloudera-mgmt-HOSTMONITOR [29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Activating Process 1019-cloudera-mgmt-HOSTMONITOR [29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Created /var/run/cloudera-scm-agent/process/1019-cloudera-mgmt-HOSTMONITOR [29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Chowning /var/run/cloudera-scm-agent/process/1019-cloudera-mgmt-HOSTMONITOR to cloudera-scm (781) cloudera-scm (501) [29/Jun/2015 10:57:41 +0000] 2917 MainThread agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/1019-cloudera-mgmt-HOSTMONITOR to 0751 [29/Jun/2015 10:57:41 +0000] 2917 MainThread agent ERROR Failed to activate {u'refresh_files': [u'cloudera-stack-monitor.properties'], u'config_generation': 0, u'auto_restart': True, u'running': True, u'required_tags': [], u'user': u'cloudera-scm', u'special_file_info': [], u'group': u'cloudera-scm', u'id': 1019, u'status_links': {u'status': u'http://ilmtxbda146.corp.amdocs.com:8091/'}, u'name': u'cloudera-mgmt-HOSTMONITOR', u'run_generation': 1, u'environment': {u'MGMT_LOG_FILE': u'mgmt-cmf-mgmt-HOSTMONITOR-ilmtxbda146.corp.amdocs.com.log.out', u'CDH_VERSION': u'-1', u'FIREHOSE_JAVA_OPTS': u'-Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError={{AGENT_COMMON_DIR}}/killparent.sh'}, u'optional_tags': [], u'program': u'mgmt/mgmt.sh', u'arguments': [u'firehose', u'--pipeline-type', u'HOST_MONITORING'], u'parcels': {u'CDH': u'5.3.0-1.cdh5.3.0.p0.30'}, u'resources': [{u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': False, u'rlimits': {u'limit_memlock': None, u'limit_fds': None}, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/log/cloudera-scm-firehose', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 509}, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'10.233.164.119', u'port': 8091}, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'10.233.164.119', u'port': 9995}, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': {u'bind_address': u'10.233.164.119', u'port': 9994}, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': None, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': False, u'rlimits': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/lib/cloudera-host-monitor', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 509}, u'cpu': None}, {u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': True, u'rlimits': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/log/cloudera-scm-firehose/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 493}, u'cpu': None}], u'one_off': False} Traceback (most recent call last): File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1234, in handle_heartbeat_response new_process.activate() File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2438, in activate self.update_process_environment_and_parcels() File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2601, in update_process_environment_and_parcels parcels_in_use = self.agent.repo.prepare_environment(self.raw['parcels'], AttributeError: 'NoneType' object has no attribute 'prepare_environment' [29/Jun/2015 10:57:57 +0000] 2917 CP Server Thread-12 _cplogging INFO 10.233.164.119 - - [29/Jun/2015:10:57:57] "GET /download_log?path=%2Fvar%2Flog%2Fcloudera-scm-firehose%2Fmgmt-cmf-mgmt-HOSTMONITOR-ilmtxbda146.corp.amdocs.com.log.out&onlyTail=true&compress=false HTTP/1.1" 200 2011 "" "Java/1.7.0_10" [29/Jun/2015 11:07:28 +0000] 2917 MonitorDaemon-Reporter throttling_logger ERROR (9 skipped) Error sending messages to firehose: mgmt-HOSTMONITOR-14f3c3b6446d19e4dcc4dab16355bb71 Traceback (most recent call last): File "/usr/lib64/cmf/agent/src/cmf/monitor/firehose.py", line 70, in _send self._port) File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 464, in __init__ self.conn.connect() File "/usr/lib64/python2.6/httplib.py", line 720, in connect self.timeout) File "/usr/lib64/python2.6/socket.py", line 567, in create_connection raise error, msg error: [Errno 111] Connection refused ========= Luckily for me, I had the same parcel on a different environment so I could copy it and environment was happy again. My questions for cloudera people: * Why do I need the parcels after it was successfully installed? (all the jars and exe are available from /opt/cloudera/parcels/CDH) * Do you have any suggestions for saving space? - again this is for small testing environments and not for production. Thanks Doron

doronve · ‎05-26-2015

I manage to create python script to change the configuration. here is an example: https://linuxlabrat.wordpress.com/2015/05/26/updating-cdh-configuration-using-python-and-rest-apis/

doronve · ‎05-17-2015

Hi Darren, just now got to check it and It worked! Thanks!! Doron

doronve · ‎05-13-2015

Can I update oozie-site.xml using REST api? We added our own external function to oozie. After each installation we need to go manually and change things in oozie via the CM web interface: Oozie Server Configuration Safety Valve for oozie-site.xml - add two properties oozie.service.ActionService.executor.ext.classes - add additional value oozie SchemaService Workflow Extension Schemas - add additional value i'd like to be able to do this via scripting of any kind. So far i could not find such an option - neither in CM REST API (http://cloudera.github.io/cm_api/apidocs/v10/index.html) nor OOZIE REST API (http://archive.cloudera.com/cdh4/cdh/4/oozie/WebServicesAPI.html) Thanks Doron

doronve · ‎05-13-2015

Thanks we'll do that. i.e. - currently we'll add the flag under init.d for the long run we'll create an alternative robust DB. Doron

doronve · ‎05-12-2015

Hi I had an issue with uid of users on different servers so i had to delete and reinstall the cluster. unfortunately - I kept the old parcels files with the old UID ls -l /opt/cloudera/parcel-repo/CDH-5.3.2-1.cdh5.3.2.p0.10-el6.parcel* -rw-r----- 1 cloudera-scm cloudera-scm 1558200266 May 12 14:51 /opt/cloudera/parcel-repo/CDH-5.3.2-1.cdh5.3.2.p0.10-el6.parcel -rw-r----- 1 cloudera-scm cloudera-scm 848904192 May 12 14:52 /opt/cloudera/parcel-repo/CDH-5.3.2-1.cdh5.3.2.p0.10-el6.parcel.part -rw-r----- 1 522 522 41 Apr 7 13:33 /opt/cloudera/parcel-repo/CDH-5.3.2-1.cdh5.3.2.p0.10-el6.parcel.sha note that the *.parcel.sha file is with the old UID of cloudera-scm account. I saw that the parcels is being downloaded and redownloded in an endless loop. in the log file i saw: 2015-05-12 14:07:14,322 INFO MainThread:com.cloudera.parcel.components.PeriodicParcelTasks: Set up periodic parcel tasks every 60 minutes. 2015-05-12 14:07:14,337 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files CDH-5.3.2-1.cdh5.3.2.p0. 10-el6.parcel under /opt/cloudera/parcel-repo 2015-05-12 14:07:14,352 WARN ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Error reading hash file: CDH-5.3.2-1 .cdh5.3.2.p0.10-el6.parcel.sha java.io.FileNotFoundException: /opt/cloudera/parcel-repo/CDH-5.3.2-1.cdh5.3.2.p0.10-el6.parcel.sha (Permission denied) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at com.google.common.io.Files$FileByteSource.openStream(Files.java:124) at com.google.common.io.Files$FileByteSource.openStream(Files.java:114) at com.google.common.io.ByteSource$AsCharSource.openStream(ByteSource.java:287) at com.google.common.io.CharSource.openBufferedStream(CharSource.java:80) at com.google.common.io.CharSource.readFirstLine(CharSource.java:157) at com.google.common.io.Files.readFirstLine(Files.java:674) at com.cloudera.parcel.components.LocalParcelManagerImpl.readFirstLineFromFile(LocalParcelManagerImpl.java:392) at com.cloudera.parcel.components.LocalParcelManagerImpl.getParcelHash(LocalParcelManagerImpl.java:348) at com.cloudera.parcel.components.LocalParcelManagerImpl.processParcel(LocalParcelManagerImpl.java:182) at com.cloudera.parcel.components.LocalParcelManagerImpl.scanRepo(LocalParcelManagerImpl.java:142) at com.cloudera.parcel.components.LocalParcelManagerImpl$1.run(LocalParcelManagerImpl.java:155) at com.cloudera.parcel.components.LocalParcelManagerImpl$1.run(LocalParcelManagerImpl.java:152) at com.cloudera.cmf.persist.ReadWriteDatabaseTaskCallable.call(ReadWriteDatabaseTaskCallable.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Once i deleted the old sha file the downloding of the parcel ended and i could continue with the installation. I'd think there should be some kind of warning to the screen that something is wrong so one won't have to wait that long Doron

doronve · ‎05-11-2015

We have the postgres db crashes occasional. There is no core dump. Can we add the -c flag to the pg_ctl command in the /etc/init.d/cloudera-scm-server-db file? $SU -c "pg_ctl start -c -w -D $DATA_DIR -l $SERVER_LOGFILE" can we keep this change permanent? Thanks Doron

doronve · ‎03-16-2015

I've wrote a little script to resolve the issue. although the script is a workaround and I still need to find the proper solution. details - here - https://linuxlabrat.wordpress.com/2015/03/16/removing-alternatives-of-old-cdh-parcels/

Online	Offline
Last Visited	‎10-17-2019 11:38 AM

Date Registered	‎08-11-2014 04:32 AM
Last Visited	‎10-17-2019 11:38 AM
Total Messages Posted	31
Total Kudos Received	2

Hardware recommendation for non-production

Delay HDFS FUSE mount till nameservice is up after...

Re: DROP FUNCTION IF EXISTS udf_... fails with "fu...

DROP FUNCTION IF EXISTS udf_... fails with "functi...

anaconda parcels for cloudera - broken documentati...

Re: upgrade path verification

upgrade path verification

Re: upgrading to HDP 264 and keeping old HIVE

Re: upgrading to HDP 264 and keeping old HIVE

Re: upgrading to HDP 264 and keeping old HIVE

upgrading to HDP 264 and keeping old HIVE

Re: failed to upgrade to spark2.3 on CM 5.10.1

failed to upgrade to spark2.3 on CM 5.10.1

Re: how to modify dfs.permissions.superusergroup i...

how to modify dfs.permissions.superusergroup in HD...

file deleted from /etc/hbase/conf

/user/spark/spark2ApplicationHistory folder permis...

Changing Hive Metastore Database to Oracle when HD...

Re: Oozie - Is it OK to modify its database?

Oozie - Is it OK to modify its database?

Oozie - Is it OK to modify its database?

Re: Weirdest reason for Cloudera Management Servic...

Weirdest reason for Cloudera Management Services n...

Re: updating oozie-site.xml via REST api

Re: updating oozie-site.xml via REST api

updating oozie-site.xml via REST api

Re: We have the postgres db crashes occasional. Th...

small bug when re-downloading parcels.

We have the postgres db crashes occasional. There ...

Re: Incorrect symlinks being created by cloudera-s...