About JesperCelander

JesperCelander · ‎10-06-2017

Hi. Worked! Thank you for your help.

JesperCelander · ‎09-22-2017

Hi. We have configured HUE Load Balancer with cloudera manager. And we also have an additional iptables NAT rule that forward 443 -> HUE Load Balancer port for a user friendly experience. We run HUE LB on port tcp/7180. iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 7180 End users access HUE with a CNAME in the browser, let’s stay https://internal-hue.company.net. The actual HUE instance is on AWS and we have a VPN from our internal network so the server is using the standard VPC hostname ip-X-X-X-X.<aws region>.compute.internal. We now get the following warning banner from HUE saying that we are accessing non-optimized hue, even if we are going through the HUE LB. "You are accessing a non-optimized Hue, please switch to one of the available addresses: https://ip-X-X-X-X.<region>..compute.internal:7180" Has anyone seen this problem before when using CNAMES instead of hostname with HUE LB?

JesperCelander · ‎01-12-2017

Thanks!

JesperCelander · ‎12-19-2016

We are using cloudera manager with CDH 5.8.3. Thanks for the update! I can see now that YARN and HDFS can be refreshed without restarting the cluster. However some services like SPARK2, HIVE, HUE and OOZIE still have the old topology.map file and cloudera manager wants to restart stale services. Is it safe to wait until we have server maintenance or something and do the restart?

JesperCelander · ‎12-15-2016

Hi. Downloaded spark 2.0 from http://www.cloudera.com/downloads/spark2/2-0.html and deployed it according to the instructions for cloudera manager. Installed latest livy server from github with options "mvn clean package -DskipTests -Dspark-2.0 -Dscala-2.11". During startup livy server can't figure out the spark-submit version due to a regex problem in livy server LivySparkUtils.scala file. "Fail to parse Spark version from 2.0.0.cloudera1" Here is a similar issue, https://issues.cloudera.org/browse/LIVY-229. Is there any official guidelines from Cloudera on what version of Livy server to use for Spark 2.0? And will livy server be packaged in future CDH releases?

JesperCelander · ‎10-24-2016

Hi. When adding a new worker node with Datanode, NodeManager and Impalad cloudera manager wants to restart stale services due to the topology.map file. We are deploying a single DC / Rack cluster so there isn't any need for us to divide into racks, all of them are in /default. Is there any way to avoid this behavior? And are there any negative effects ( if only using /default rack ) to not restarting the stale services when asked to update topology.map file? The node seems to be in both YARN and HDFS node list and we can see that jobs and blocks are being copied to the host. /Jesper

JesperCelander · ‎09-08-2016

Hi. Thanks for your reply! Yes, I tried that. When I add user_attribute_mapping="{'nameId': ('username',), 'emailAddress': ('email',), 'givenName' : ('first_name',), 'surname': ('last_name', )}" to hue_safety_valve_server.ini hue server throws this exemption. + /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue migrate --merge [08/Sep/2016 12:26:08 +0000] settings DEBUG DESKTOP_DB_TEST_NAME SET: /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/desktop-test.db [08/Sep/2016 12:26:08 +0000] settings DEBUG DESKTOP_DB_TEST_USER SET: hue_test Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue", line 12, in <module> load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')() File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/manage_entry.py", line 57, in entry execute_from_command_line(sys.argv) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line utility.execute() File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 392, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 261, in fetch_command commands = get_commands() File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 107, in get_commands apps = settings.INSTALLED_APPS File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 54, in __getattr__ self._setup(name) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 49, in _setup self._wrapped = Settings(settings_module) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 128, in __init__ mod = importlib.import_module(self.SETTINGS_MODULE) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/importlib.py", line 40, in import_module __import__(name) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/settings.py", line 391, in <module> from libsaml.saml_settings import * File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/libs/libsaml/src/libsaml/saml_settings.py", line 105, in <module> SAML_ATTRIBUTE_MAPPING = libsaml.conf.USER_ATTRIBUTE_MAPPING.get() File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 147, in get return self.config.get_value(data, present=present, prefix=self.prefix, coerce_type=True) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 263, in get_value return self._coerce_type(raw_val, prefix) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 283, in _coerce_type return self.type(raw) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/libs/libsaml/src/libsaml/conf.py", line 46, in dict_list_map for k, v in json.loads(value).iteritems(): File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python2.7/json/decoder.py", line 381, in raw_decode obj, end = self.scan_once(s, idx) ValueError: Expecting property name: line 1 column 2 (char 1) + '[' dumpdata = runcpserver ']' + '[' syncdb = runcpserver ']' + exec /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue runcpserver [08/Sep/2016 12:26:09 +0000] settings DEBUG DESKTOP_DB_TEST_NAME SET: /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/desktop-test.db [08/Sep/2016 12:26:09 +0000] settings DEBUG DESKTOP_DB_TEST_USER SET: hue_test Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue", line 12, in <module> load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')() File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/manage_entry.py", line 57, in entry execute_from_command_line(sys.argv) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line utility.execute() File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 392, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 261, in fetch_command commands = get_commands() File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 107, in get_commands apps = settings.INSTALLED_APPS File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 54, in __getattr__ self._setup(name) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 49, in _setup self._wrapped = Settings(settings_module) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/conf/__init__.py", line 128, in __init__ mod = importlib.import_module(self.SETTINGS_MODULE) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/utils/importlib.py", line 40, in import_module __import__(name) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/settings.py", line 391, in <module> from libsaml.saml_settings import * File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/libs/libsaml/src/libsaml/saml_settings.py", line 105, in <module> SAML_ATTRIBUTE_MAPPING = libsaml.conf.USER_ATTRIBUTE_MAPPING.get() File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 147, in get return self.config.get_value(data, present=present, prefix=self.prefix, coerce_type=True) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 263, in get_value return self._coerce_type(raw_val, prefix) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/core/src/desktop/lib/conf.py", line 283, in _coerce_type return self.type(raw) File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/libs/libsaml/src/libsaml/conf.py", line 46, in dict_list_map for k, v in json.loads(value).iteritems(): File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python2.7/json/decoder.py", line 381, in raw_decode obj, end = self.scan_once(s, idx) ValueError: Expecting property name: line 1 column 2 (char 1) Here is my complete saftey valve for [libsam] in hue.ini. [libsaml] xmlsec_binary=/usr/bin/xmlsec1 create_users_on_login=true metadata_file=/opt/hue/saml/metadata.xml key_file=/opt/hue/saml/my_key.pem cert_file=/opt/hue/saml/my_cert.pem logout_requests_signed=true required_attributes=emailAddress,givenName,surname,nameId username_source=attributes logout_enabled=true user_attribute_mapping="{'nameId': ('username',), 'emailAddress': ('email',), 'givenName' : ('first_name',), 'surname': ('last_name', )}" authn_requests_signed=false Thank you for the issue report on the create users! /Jesper

JesperCelander · ‎09-01-2016

Hi. I'm trying to configure HUE for saml login with CDH5.7.1 parcels. First I can't get the user_attribure_mapping directive in hue.ini to work. We use Cloudera Manager so configured it in "hue/config/service-wide/Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini" [libsaml] required_attributes=emailAddress,givenName,surname,nameId username_source=attributes user_attribute_mapping={'nameId': ('username',), 'emailAddress': ('email',), 'givenName' : ('first_name',), 'surname': ('last_name', )} Here is the error msg from hue.error log in debug. [01/Sep/2016 12:08:33 +0000] backends ERROR Session info or attribute mapping are None [01/Sep/2016 12:08:33 +0000] views ERROR The user is None Session info is not empty, it's the attribute_mapping that's not updated with any value from hue.ini. When hard coding it as the default value in $HUE_HOME/desktop/libs/libsaml/src/libsaml/conf.py it works. USER_ATTRIBUTE_MAPPING = Config( key="user_attribute_mapping", #default={'uid': ('username', )}, default={'nameId': ('username',), 'emailAddress': ('email',), 'givenName' : ('first_name',), 'surname': ('last_name', )}, type=dict_list_map, help=_t("A mapping from attributes in the response from the IdP to django user attributes.")) Second problem is that previously when using ldap login the hdfs user folder was created during login, this isn't done when using libsaml. I have added the "create_users_on_login=true" for libsaml in hue.ini but not sure if this directive just creates the users in the hue DB or if it should create the hdfs home folder for the username that login? If not, is there a way to get HUE/libsaml to create the hdfs user folder as when using ldap login for hue? We could script it but if there is a way to get hue/libsaml to do it I would prefer that. /Jesper

JesperCelander · ‎06-20-2016

Hi. We get this warning after a upgrade to CDH5.7.1. "WARN [main] mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it." Have tried to copy all <parcel>/jars/hbase-* to the folder configured in hive for the " Hive Auxiliary JARs Directory" setting in CMS.

JesperCelander · ‎06-16-2016

Hi. I'm fairly new to how hive works, how does one actually add the jar to the hive classpath in Cloudera Manager?

JesperCelander · ‎06-14-2016

Hi. Thank you for your support on this issue! We will proceed to upgrade our production cluster to 5.7.1 now that we know the details for this log entry.

JesperCelander · ‎06-14-2016

Another thing, looking at the "VERSION" file in the DatanNode directories they all have "layoutVersion=-56". And the NameNode/JournalNode directory "VERSION" file report layoutVersion=-60. Is this normal, thought they should report the same version?

JesperCelander · ‎06-14-2016

Hi. These logs repeat over the day, upgraded to "5.7.1" 4 days ago and the Warning still show when the any of the nodes do a blockreport. Didn't do a rolling upgrade, followed the steps in this article. http://www.cloudera.com/documentation/enterprise/latest/topics/install_upgrade_to_cdh57_parcels.html Looking at the NameNode Web UI / live nodes all of them report version " 2.6.0-cdh5.7.1". I'm running NameNode HA.

JesperCelander · ‎06-13-2016

After a upgrade using Cloudera Manager and pracels I get these events in the active NameNode log. WARN BlockStateChange: BLOCK* processReport: Report from the DataNode (<datanodeUuid>) is unsorted. This will cause overhead on the NameNode which needs to sort the Full BR. Please update the DataNode to the same version of Hadoop HDFS as the NameNode (2.6.0-cdh5.7.1). When checking the version of HDFS on the datanode in the event I can see that it reports the correct version. # hdfs version Hadoop 2.6.0-cdh5.7.1 Subversion http://github.com/cloudera/hadoop -r ae44a8970a3f0da58d82e0fc65275fff8deabffd Compiled by jenkins on 2016-06-01T23:26Z Compiled with protoc 2.5.0 From source with checksum 298b68dc3b308983f04cb37e8416f13 This command was run using /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/jars/hadoop-common-2.6.0-cdh5.7.1.jar Has anybody seen this warning before? You can reproduce the error with "hdfs dfsadmin -triggerBlockReport". We upgraded from CDH5.5.1 -> CDH5.7.1 using parcels.

Online	Offline
Last Visited	‎11-23-2017 04:41 AM

Date Registered	‎06-13-2016 04:50 AM
Last Visited	‎11-23-2017 04:41 AM
Total Messages Posted	14
Total Kudos Received	1

Re: HUE Load Balancer warning banner

HUE Load Balancer warning banner

Re: Spark 2.0 / livy server 3

Re: topology.map - Restart stale services necessar...

Spark 2.0 / livy server 3

topology.map - Restart stale services necessary?

Re: HUE 3.9 saml login.

HUE 3.9 saml login.

CDH5.7.1 The hbase-prefix-tree module jar containi...

Re: Hive reports manifest security exception after...

Re: "Report from the DataNode (datanodeUuid) is un...

Re: "Report from the DataNode (datanodeUuid) is un...

Re: "Report from the DataNode (datanodeUuid) is un...

"Report from the DataNode (datanodeUuid) is unsort...