Member since
02-16-2017
20
Posts
1
Kudos Received
0
Solutions
02-19-2021
05:50 AM
hi, when i upgraded to cdh 6.2.0, the namenodes stopped exposing metrics on 9070. do anyone know where the metrics are exposed now?
... View more
Labels:
02-11-2021
06:45 AM
the use case is for adding a host to the cluster. to download the packages, it just connects to local mirror (instead of archive.cloudera) this will also allow for an non-internet connect infrastructure.
... View more
02-10-2021
04:55 AM
am i allowed to publicly mirror archive.cloudera.com?
this is for Express version
... View more
02-10-2021
03:33 AM
this might be an auth problem, i have noticed 404's as well. try to go to https://archive.cloudera.com/ and see you will be asked for login.
... View more
12-08-2020
06:54 AM
i have a long running job, that has survived some upgrades (os and older cdh versions) the command executed: /usr/bin/hadoop jar /opt/cloudera/parcels/CDH/lib/hbase/hbase-server.jar importtsv the exception this produces: Exception in thread "main" java.lang.ClassNotFoundException: importtsv it seems that importtsv is not there? however i have not been able to locate the class responsible for this.
... View more
Labels:
04-29-2020
02:42 AM
is there a place that shows the operating system requirements of future CDH versions? I am wondering specifically if there is a ubuntu focal version spec'd and what other OS's it supports?
... View more
Labels:
03-11-2020
05:44 AM
CDH: 5.13.0
HIVE version:
VER_ID | SCHEMA_VERSION | VERSION_COMMENT | SCHEMA_VERSION_V2 --------+----------------+----------------------------+------------------- 1 | 1.1.0 | Hive release version 1.1.0 | 1.1.0-cdh5.12.0
when trying to run the 'Upgrade Hive Metastore Database Schema' command
through the GUI (CDH) it reports the following:
(STDOUT)
Starting upgrade metastore schema from version 1.1.0-cdh5.12.0 to 1.1.0-cdh5.13.0 schemaTool completed Exit code: 0
However (STDERR) reports sort of the opposite:
INFO metastore.CDHMetaStoreSchemaInfo: Current version is higher than or equal to 1.1.0-cdh5.12.0 Skipping file 1.1.0-to-1.1.0-cdh5.12.0
i want to be on cdh5.13.0 but hive is thinking it's in earlier version.
did i miss a migration/upgrade step somewhere to cause this?
does anybody know how to force an upgrade to 5.13.0?
... View more
- Tags:
- CDH 5.13
- schematool
Labels:
10-24-2019
12:08 AM
thanks, i suppose one could just move that to wherever you want the home to be. this was part of a hack deployment, so ultimately not needed.
... View more
10-09-2019
06:26 AM
Spark2 2.3.0 on cdh.5.13 deployed with CSD and parcels. by default it uses /etc/spark2/conf, but it seems it should use /etc/spark2/conf.cloudera.spark2_on_yarn/ is there a way to set it properly? i could set it in spark2-env.sh but it seems not the appropriate place.
... View more
Labels:
08-14-2019
03:44 AM
on this guide: https://www.cloudera.com/documentation/enterprise/5-3-x/topics/search_hdfsfindtool.html
there is a reference in the -mtime option:
Evaluates as true if the file modification time subtracted
from the start time is n days
what does the "start time" in here reference?
it does not seem to work like normal bash find.
i want functionality like:
if mtime is more than 30 days ago (from current time), print the folder/file.
... View more
Labels:
06-20-2019
01:37 AM
thanks for your answer, do i need the .meta and .meta.tmp files?
... View more
06-19-2019
06:15 AM
so i have a flume agent with a spooldir channel that got too full (too many files error). the fix was to move the files out of the spooldir, move the checkpoint, and restart the agent, resulting in data flowing again. but now i have some flume-formatted data that i would like to ingest. can i move the files into the spooldir 10% at a time fx? would i have to read the files and add them to the flume source? additional notes: the reason for the spool dir was a missing sink (another flume agent- turned off)
... View more
Labels:
11-09-2018
12:59 AM
i have a logging setup where i have a fluent/webhdfs host, for writing logs from fluent to hdfs. a problem is i can only write to the ACTIVE namenode, this sets some requirements to the logic in the fluent/webhdfs host that i would like to avoid. is there a way to make sure youre always posting to the ACTIVE namenode?
... View more
Labels:
06-28-2018
06:06 AM
yes, but that requires cluster restart right? and also the copying on the journal node edit dirs.
... View more
06-28-2018
04:06 AM
thanks, this is what worked in the end, coupled with kill -9 for the really resilient procs.
... View more
06-28-2018
03:59 AM
so i want to move the Journalnode from 1 host to another. this is a cluster running cloudera manager 5.13 it's set up with Quorum Journal. currently there is 3 nodes: A,B,C I want to add node D. and then remove node C. why can i not do the following: 1. add journalnode role to D 2. remove journalnode role from C leaving me with A,B,D and a migrated journalnode.
... View more
Labels:
04-25-2018
06:18 AM
UPDATE: did some experimentations on my own, and i deleted the pkg_resources, in /usr/lib/python2.7/dist-packages this apparantly fixed the issue as hue will now run, and allow me to log in. i am not sure if an update would also work...
... View more
04-24-2018
01:38 AM
upgraded from cdh 5.6 to 5.13
and since then, hue will not start.
stderr from running a "Start" on "Hue" through the Manager- GUI
++ tr '\n' :
++ find /usr/share/cmf/lib/plugins -maxdepth 1 -name '*.jar'
+ ADD_TO_CP=/usr/share/cmf/lib/plugins/tt-instrumentation-5.13.0.jar:/usr/share/cmf/lib/plugins/event-publish-5.13.0-shaded.jar:
+ [[ -n '' ]]
+ eval 'OLD_VALUE=$HADOOP_EXTRA_CLASSPATH_STRING'
++ OLD_VALUE=
+ NEW_VALUE=/usr/share/cmf/lib/plugins/tt-instrumentation-5.13.0.jar:/usr/share/cmf/lib/plugins/event-publish-5.13.0-shaded.jar:
+ export HADOOP_EXTRA_CLASSPATH_STRING=/usr/share/cmf/lib/plugins/tt-instrumentation-5.13.0.jar:/usr/share/cmf/lib/plugins/event-publish-5.13.0-shaded.jar
+ HADOOP_EXTRA_CLASSPATH_STRING=/usr/share/cmf/lib/plugins/tt-instrumentation-5.13.0.jar:/usr/share/cmf/lib/plugins/event-publish-5.13.0-shaded.jar
+ HUE=/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hue/build/env/bin/hue
+ [[ runcpserver == runcpserver ]]
+ grep -q '^\s*ssl_certificate\s*=\s*.\+' hue.ini hue_safety_valve.ini hue_safety_valve_server.ini
+ '[' '!' -z '' ']'
+ run_syncdb_and_migrate_subcommands
+ /opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hue/build/env/bin/hue syncdb --noinput
+ '[' -z '' ']'
+ MERGE_FLAG=--merge
+ '[' 5 -ge 5 ']'
+ /opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hue/build/env/bin/hue migrate --merge
+ '[' dumpdata = runcpserver ']'
+ '[' syncdb = runcpserver ']'
+ '[' ldaptest = runcpserver ']'
+ exec /opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hue/build/env/bin/hue runcpserver
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hue/build/env/bin/hue", line 8, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2707, in <module>
working_set.require(__requires__)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 686, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 584, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: distribute
it seems like it's a bad import in the hue script:
/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/hue/build/env/bin/hue
#!/usr/bin/env python2.7
import os; activate_this=ocdn(os.path.dirname(os.path.realpath(__file__)), 'activate_this.py'); exec(compile(open(activate_this).read(), activate_this, 'exec'), dict(__file__=activate_this)); del os, activate_this
# EASY-INSTALL-ENTRY-SCRIPT: 'desktop==3.9.0','console_scripts','hue'
__requires__ = 'desktop==3.9.0'
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.exit(
load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')()
)
should be changed to( changes on line 6 and 10)
#!/usr/bin/env python2.7
import os; activate_this=ocdn(os.path.dirname(os.path.realpath(__file__)), 'activate_this.py'); exec(compile(open(activate_this).read(), activate_this, 'exec'), dict(__file__=activate_this)); del os, activate_this
# EASY-INSTALL-ENTRY-SCRIPT: 'desktop==3.9.0','console_scripts','hue'
__requires__ = 'desktop==3.9.0'
import sys
import pkg_resources
if __name__ == '__main__':
sys.exit(
pkg_resources.EntryPoint.load('desktop==3.9.0', 'console_scripts', ‘hue’)()
)
is there another way to solve this?
something like putting the old pkg_resource into the path of the new?
... View more
Labels:
11-29-2017
05:30 AM
i am having problems forcing the cloudera/hadoop procs to use the new supervisord and assesing the impact of this not working. i am upgrading from cdh5.6 to cdh5.13 did upgrade all the cloudera manager daemons and agents to 5.13 some nodes are not using the new supervisord, this is seen from the text below, taken from the host inspector, it gives 3 groups using 3 different supervisord group1: Supervisord 3.0-cm5.13.0 group2L Supervisord 3.0-cm5.6.0 group3: Supervisord 3.0 i can move hosts to use the new supervisord, by killing the old process, kill $(cat /run/cloudera-scm-agent/supervisord/supervisord.pid) this works, something starts a new, korrekt version, of supervisord. this seems like a not-nice way to do it, and causes some warning in the manager. is there another way to do this? also, what is the potential risks in not using the same supervisord version, will the nodes be unable to communicate? can i upgrade parcels?
... View more
Labels:
04-05-2017
04:12 AM
1 Kudo
i want to put a 1:1 elastic cluster inside my hadoop cluster, 1 elastic node on each hadoop datanode. to not interfere too much with the hadoop cluster, i would like to run the elastic nodes on a disk of it's own. the setup: CDH 5.6.0 datanode disk layout(in mounted dirs): /data/disk1 . . . /data/disk10 20+ data nodes say i wish to remove disk10 from each datanode, how do i do that without data loss? removing the disk on a decommissioned datanote, and later recomissioning it, takes too long time. - any hint on making this process faster? can i use the rebalancer? ( i saw there is internal datanode balancer in CDH5.8)
... View more
Labels: