Member since
10-02-2017
116
Posts
3
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1021 | 07-18-2020 12:04 PM | |
1773 | 09-11-2019 01:14 PM | |
2533 | 08-16-2019 08:17 AM | |
6429 | 08-15-2019 12:23 PM | |
4268 | 05-14-2019 08:48 AM |
07-08-2019
04:58 PM
Using Altus Director 6.2.1 to deploy CDH cluster to Azure via command line using bootstrap-remote.
Per the Cloudera documentation, I added the following in my cluster bootstrap file in the base instance template:
image: /subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55
useCustomManagedImage: Yes
customImagePlan: ""
managedDisks: Yes
There is no purchase plan associated with the image, so I left blank per your documentation.
Deployment fails with the following error:
ValidationExceptionCondition{scope='azure.com.cloudera.director.azure.compute.provider.AzureComputeProvider.template', key='Optional.of(image)', exceptionInfo='ErrorInfo{code=PROVIDER_EXCEPTION, properties={message=Image '/subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55' is referencing a Custom Image but not all required fields are set. To use Custom Images set the 'useCustomManagedImage' and 'customImagePlan' fields.}, causes=[]}'}
Is this actually supported?
... View more
Labels:
05-14-2019
08:48 AM
Update: I rebuilt our image and no longer see this issue. I still don't know why we didn't see this issue with Altus 2.8 using the same image, but I'll mark as resolved since we no longer see it with the new image.
... View more
04-19-2019
07:49 AM
Thank you for your response. We base our image on the stock CentOS AMI provided by AWS, with some security policies and tuning applied. Interestingly enough, we've deployed several 5.16.1 clusters using the same AMI. This appears to be specific to CDH6.2. I encounter this issue whether or not I try to include Hue in the Altus cluster config, or add the service post deployment from Cloudera Manager. Wrapt is just the first module it complains about. There are several others. I ended up simply changing directory / file permissions for all dist-info, afterwhich I could add Hue service. chmod 755 /usr/lib64/python2.7/site-packages/*.dist-info
chmod -R 644 /usr/lib64/python2.7/site-packages/*.dist-info/
chmod 755 /usr/lib/python2.7/site-packages/*.dist-info
chmod -R 644 /usr/lib/python2.7/site-packages/*.dist-info/ I am using the latest version of Altus Director (6.2).
... View more
04-18-2019
09:30 AM
I tried another deployment and hit the same errors. All other cluster services start fine during first run. Hue continues to fail to start due to insufficient permissions to load Python modules. Pretty sure I could fix this if someone pointed me to where you handle the install of these Python modules. Suprised no one else is reporting it.
... View more
04-17-2019
01:15 PM
I've read the known issues for Hue in the Altus Director 6.2 docs, however according to the information in the docs, this shouldn't be applicable to CM6.2 and CentOS 7.6. I'm seeing the following error when Hue attempts to start during the cluster's firstrun phase: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue makemigrations --noinput
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue", line 9, in <module>
from pkg_resources import load_entry_point
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
@_call_aside
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
f(*args, **kwargs)
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 574, in _build_master
ws = cls()
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 567, in __init__
self.add_entry(entry)
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 623, in add_entry
for dist in find_distributions(entry, True):
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2033, in find_on_path
for dist in factory(fullpath):
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2095, in distributions_from_metadata
if len(os.listdir(path)) == 0:
OSError: [Errno 13] Permission denied: '/usr/lib64/python2.7/site-packages/wrapt-1.11.1.dist-info'
+ /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue migrate --fake-initial
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/bin/hue", line 9, in <module>
from pkg_resources import load_entry_point
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
@_call_aside
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
f(*args, **kwargs)
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 574, in _build_master
ws = cls()
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 567, in __init__
self.add_entry(entry)
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 623, in add_entry
for dist in find_distributions(entry, True):
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2033, in find_on_path
for dist in factory(fullpath):
File "/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2095, in distributions_from_metadata
if len(os.listdir(path)) == 0:
OSError: [Errno 13] Permission denied: '/usr/lib64/python2.7/site-packages/wrapt-1.11.1.dist-info' Update: I "solved" this by updating the site-package permissions under /usr/lib /usr/lib64, as they were locked down to root. If I were to guess, some modules were pip installed without the -u switch, locking down access to root only. I assume this should be handled by the deployment scripts when preparing the cluster nodes.
... View more
Labels:
- Labels:
-
Cloudera Hue
04-04-2019
10:55 AM
The list of supported CentOS versions for CDH clusters managed by Cloudera Altus Director 2.8 and 6.0 do not include CentOS 7.6, however we have several clusters managed by Director 2.8 running CentOS 7.6. Are the docs out of date?
... View more
Labels:
09-04-2018
08:43 AM
Bill, Thank you for the clarification.
... View more
09-03-2018
03:42 PM
I would like some clarification on the reported deprecation of the Director client's "standalone" functionality in version 2.8 and beyond. We currently use Cloudera Director 2.7 to bootstrap client environments. To do this, we use the standalone client on Jenkins to execute the bootstrap-remote command, which makes a call out to our Director server. We would like to continue using Jenkins because our Cloudera deployments are simply part of the multi-stage client deployment job. Will the bootstrap-remote option be removed in future releases as part of this deprecation?
... View more
05-31-2018
08:15 AM
Thanks for your reply. The following link leads me to believe that what I'm seeing is expected: https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Spark-gateway-not-starting/m-p/49021
... View more
05-18-2018
08:50 AM
Ah. I see. In the Cloudera Director bootstrap config file for this deployment, I have the Spark_On_Yarn Gateway role enabled for all cluster hosts. The hosts are associated with the role, however the role didn't start after deployment. I tried starting the gateway role on one of the nodes using Cloudera Manager and receive the following error: "Command Start is not currently available for execution" All other Spark roles (and remaining cluster services / roles) are functioning normally. We are running CDH 5.14.2
... View more