Member since
06-13-2016
14
Posts
1
Kudos Received
0
Solutions
09-22-2017
03:58 AM
Hi. We have configured HUE Load Balancer with cloudera manager. And we also have an additional iptables NAT rule that forward 443 -> HUE Load Balancer port for a user friendly experience. We run HUE LB on port tcp/7180. iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 7180 End users access HUE with a CNAME in the browser, let’s stay https://internal-hue.company.net. The actual HUE instance is on AWS and we have a VPN from our internal network so the server is using the standard VPC hostname ip-X-X-X-X.<aws region>.compute.internal. We now get the following warning banner from HUE saying that we are accessing non-optimized hue, even if we are going through the HUE LB. "You are accessing a non-optimized Hue, please switch to one of the available addresses: https://ip-X-X-X-X.<region>..compute.internal:7180" Has anyone seen this problem before when using CNAMES instead of hostname with HUE LB?
... View more
Labels:
12-19-2016
08:10 AM
We are using cloudera manager with CDH 5.8.3. Thanks for the update! I can see now that YARN and HDFS can be refreshed without restarting the cluster. However some services like SPARK2, HIVE, HUE and OOZIE still have the old topology.map file and cloudera manager wants to restart stale services. Is it safe to wait until we have server maintenance or something and do the restart?
... View more
12-15-2016
04:16 AM
1 Kudo
Hi. Downloaded spark 2.0 from http://www.cloudera.com/downloads/spark2/2-0.html and deployed it according to the instructions for cloudera manager. Installed latest livy server from github with options "mvn clean package -DskipTests -Dspark-2.0 -Dscala-2.11". During startup livy server can't figure out the spark-submit version due to a regex problem in livy server LivySparkUtils.scala file. "Fail to parse Spark version from 2.0.0.cloudera1" Here is a similar issue, https://issues.cloudera.org/browse/LIVY-229. Is there any official guidelines from Cloudera on what version of Livy server to use for Spark 2.0? And will livy server be packaged in future CDH releases?
... View more
10-24-2016
06:23 AM
Hi. When adding a new worker node with Datanode, NodeManager and Impalad cloudera manager wants to restart stale services due to the topology.map file. We are deploying a single DC / Rack cluster so there isn't any need for us to divide into racks, all of them are in /default. Is there any way to avoid this behavior? And are there any negative effects ( if only using /default rack ) to not restarting the stale services when asked to update topology.map file? The node seems to be in both YARN and HDFS node list and we can see that jobs and blocks are being copied to the host. /Jesper
... View more
09-08-2016
06:50 AM
Hi. Thanks for your reply! Yes, I tried that. When I add user_attribute_mapping="{'nameId': ('username',), 'emailAddress': ('email',), 'givenName' : ('first_name',), 'surname': ('last_name', )}" to hue_safety_valve_server.ini hue server throws this exemption. + /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue migrate --merge
[08/Sep/2016 12:26:08 +0000] settings DEBUG DESKTOP_DB_TEST_NAME SET: /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/desktop-test.db
[08/Sep/2016 12:26:08 +0000] settings DEBUG DESKTOP_DB_TEST_USER SET: hue_test
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue", line 12, in <module>
load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')()
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/manage_entry.py", line 57, in entry
execute_from_command_line(sys.argv)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 261, in fetch_command
commands = get_commands()
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 107, in get_commands
apps = settings.INSTALLED_APPS
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 54, in __getattr__
self._setup(name)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 49, in _setup
self._wrapped = Settings(settings_module)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 128, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/importlib.py", line 40, in import_module
__import__(name)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/settings.py", line 391, in <module>
from libsaml.saml_settings import *
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/libs/libsaml/src/libsaml/saml_settings.py", line 105, in <module>
SAML_ATTRIBUTE_MAPPING = libsaml.conf.USER_ATTRIBUTE_MAPPING.get()
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 147, in get
return self.config.get_value(data, present=present, prefix=self.prefix, coerce_type=True)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 263, in get_value
return self._coerce_type(raw_val, prefix)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 283, in _coerce_type
return self.type(raw)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/libs/libsaml/src/libsaml/conf.py", line 46, in dict_list_map
for k, v in json.loads(value).iteritems():
File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 381, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 1 column 2 (char 1)
+ '[' dumpdata = runcpserver ']'
+ '[' syncdb = runcpserver ']'
+ exec /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue runcpserver
[08/Sep/2016 12:26:09 +0000] settings DEBUG DESKTOP_DB_TEST_NAME SET: /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/desktop-test.db
[08/Sep/2016 12:26:09 +0000] settings DEBUG DESKTOP_DB_TEST_USER SET: hue_test
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue", line 12, in <module>
load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')()
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/manage_entry.py", line 57, in entry
execute_from_command_line(sys.argv)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 261, in fetch_command
commands = get_commands()
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 107, in get_commands
apps = settings.INSTALLED_APPS
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 54, in __getattr__
self._setup(name)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 49, in _setup
self._wrapped = Settings(settings_module)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 128, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/importlib.py", line 40, in import_module
__import__(name)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/settings.py", line 391, in <module>
from libsaml.saml_settings import *
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/libs/libsaml/src/libsaml/saml_settings.py", line 105, in <module>
SAML_ATTRIBUTE_MAPPING = libsaml.conf.USER_ATTRIBUTE_MAPPING.get()
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 147, in get
return self.config.get_value(data, present=present, prefix=self.prefix, coerce_type=True)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 263, in get_value
return self._coerce_type(raw_val, prefix)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 283, in _coerce_type
return self.type(raw)
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/libs/libsaml/src/libsaml/conf.py", line 46, in dict_list_map
for k, v in json.loads(value).iteritems():
File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 381, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 1 column 2 (char 1) Here is my complete saftey valve for [libsam] in hue.ini. [libsaml] xmlsec_binary=/usr/bin/xmlsec1 create_users_on_login=true metadata_file=/opt/hue/saml/metadata.xml key_file=/opt/hue/saml/my_key.pem cert_file=/opt/hue/saml/my_cert.pem logout_requests_signed=true required_attributes=emailAddress,givenName,surname,nameId username_source=attributes logout_enabled=true user_attribute_mapping="{'nameId': ('username',), 'emailAddress': ('email',), 'givenName' : ('first_name',), 'surname': ('last_name', )}" authn_requests_signed=false Thank you for the issue report on the create users! /Jesper
... View more
09-01-2016
06:23 AM
Hi. I'm trying to configure HUE for saml login with CDH5.7.1 parcels. First I can't get the user_attribure_mapping directive in hue.ini to work. We use Cloudera Manager so configured it in "hue/config/service-wide/Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini" [libsaml] required_attributes=emailAddress,givenName,surname,nameId username_source=attributes user_attribute_mapping={'nameId': ('username',), 'emailAddress': ('email',), 'givenName' : ('first_name',), 'surname': ('last_name', )} Here is the error msg from hue.error log in debug. [01/Sep/2016 12:08:33 +0000] backends ERROR Session info or attribute mapping are None [01/Sep/2016 12:08:33 +0000] views ERROR The user is None Session info is not empty, it's the attribute_mapping that's not updated with any value from hue.ini. When hard coding it as the default value in $HUE_HOME/desktop/libs/libsaml/src/libsaml/conf.py it works. USER_ATTRIBUTE_MAPPING = Config( key="user_attribute_mapping", #default={'uid': ('username', )}, default={'nameId': ('username',), 'emailAddress': ('email',), 'givenName' : ('first_name',), 'surname': ('last_name', )}, type=dict_list_map, help=_t("A mapping from attributes in the response from the IdP to django user attributes.")) Second problem is that previously when using ldap login the hdfs user folder was created during login, this isn't done when using libsaml. I have added the "create_users_on_login=true" for libsaml in hue.ini but not sure if this directive just creates the users in the hue DB or if it should create the hdfs home folder for the username that login? If not, is there a way to get HUE/libsaml to create the hdfs user folder as when using ldap login for hue? We could script it but if there is a way to get hue/libsaml to do it I would prefer that. /Jesper
... View more
06-20-2016
01:38 AM
Hi. We get this warning after a upgrade to CDH5.7.1. "WARN [main] mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it." Have tried to copy all <parcel>/jars/hbase-* to the folder configured in hive for the " Hive Auxiliary JARs Directory" setting in CMS.
... View more
Labels:
06-16-2016
01:44 AM
Hi. I'm fairly new to how hive works, how does one actually add the jar to the hive classpath in Cloudera Manager?
... View more
06-14-2016
06:53 AM
Hi. Thank you for your support on this issue! We will proceed to upgrade our production cluster to 5.7.1 now that we know the details for this log entry.
... View more
06-14-2016
01:35 AM
Another thing, looking at the "VERSION" file in the DatanNode directories they all have "layoutVersion=-56". And the NameNode/JournalNode directory "VERSION" file report layoutVersion=-60. Is this normal, thought they should report the same version?
... View more
06-14-2016
12:49 AM
Hi. These logs repeat over the day, upgraded to "5.7.1" 4 days ago and the Warning still show when the any of the nodes do a blockreport. Didn't do a rolling upgrade, followed the steps in this article. http://www.cloudera.com/documentation/enterprise/latest/topics/install_upgrade_to_cdh57_parcels.html Looking at the NameNode Web UI / live nodes all of them report version " 2.6.0-cdh5.7.1". I'm running NameNode HA.
... View more
06-13-2016
05:10 AM
After a upgrade using Cloudera Manager and pracels I get these events in the active NameNode log. WARN BlockStateChange: BLOCK* processReport: Report from the DataNode (<datanodeUuid>) is unsorted. This will cause overhead on the NameNode which needs to sort the Full BR. Please update the DataNode to the same version of Hadoop HDFS as the NameNode (2.6.0-cdh5.7.1). When checking the version of HDFS on the datanode in the event I can see that it reports the correct version. # hdfs version Hadoop 2.6.0-cdh5.7.1 Subversion http://github.com/cloudera/hadoop -r ae44a8970a3f0da58d82e0fc65275fff8deabffd Compiled by jenkins on 2016-06-01T23:26Z Compiled with protoc 2.5.0 From source with checksum 298b68dc3b308983f04cb37e8416f13 This command was run using /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/jars/hadoop-common-2.6.0-cdh5.7.1.jar Has anybody seen this warning before? You can reproduce the error with "hdfs dfsadmin -triggerBlockReport". We upgraded from CDH5.5.1 -> CDH5.7.1 using parcels.
... View more
Labels: