Cloudera Community
kernel8liang
user rank
Explorer
My Accepted Solutions
Show All

Enabling auto-TLS, failed to receive heartbeat fro...

by Explorer in Support Questions
‎06-26-2019 04:16 AM
‎06-26-2019 04:16 AM
Install CDH step by step followed the document.   Enabled auto-TLS, after all agents are installed, get error message:   Installation failed. Failed to receive heartbeat from agent. Ensure that the host's hostname is configured properly. Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules). Ensure that ports 9000 and 9001 are not in use on the host being added. Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details). If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.   And in the log:   Installation failed. Failed to receive heartbeat from agent. Ensure that the host's hostname is configured properly. Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules). Ensure that ports 9000 and 9001 are not in use on the host being added. Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details). If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here. I noticed "An Enterprise license is required to enable Auto-TLS. ",  is that a problem? I don't hive a Enterprise license for CDH. ... View more

Re: ImportError: No module named numpy

by Explorer in Support Questions
‎05-14-2019 08:28 PM
‎05-14-2019 08:28 PM
find solution here https://stackoverflow.com/questions/46857090/adding-pyspark-python-path-in-oozie. ... View more

Re: ImportError: No module named numpy

by Explorer in Support Questions
‎05-14-2019 03:31 AM
‎05-14-2019 03:31 AM
We installed anaconda vir cdh. which is already actived.     In the below file:   /run/cloudera-scm-agent/process/895-spark_on_yarn-SPARK_YARN_HISTORY_SERVER/spark-conf/spark-env.sh we can see:   ... View more

Re: ImportError: No module named numpy

by Explorer in Support Questions
‎05-14-2019 03:18 AM
‎05-14-2019 03:18 AM
use command below, the job can be executed successfully.   export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark export HADOOP_CONF_DIR=/etc/alternatives/hadoop-conf PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python spark-submit --master yarn --deploy-mode cluster test.py   In Hue,  open a spark snippet , select the py file, then run it. And the same code can also be executed in Hue's nodebook with yarn model.   ... View more

ImportError: No module named numpy

by Explorer in Support Questions
‎05-13-2019 02:51 AM
‎05-13-2019 02:51 AM
Befor I post this issue, we have already readed all the same issue's solutions that we can find.   Our cluster is installed with cdh6.2, after install we use HUE to use the cluster. Job is submited via Hue.   When spark code need to import numpy,  got error below:   Traceback (most recent call last): File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_000001/test.py", line 79, in <module> from pyspark.ml.linalg import Vectors File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_000001/python/lib/pyspark.zip/pyspark/ml/__init__.py", line 22, in <module> File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_000001/python/lib/pyspark.zip/pyspark/ml/base.py", line 24, in <module> File "/var/yarn/nm/usercache/admin/appcache/application_1557739482535_0001/container_1557739482535_0001_01_000001/python/lib/pyspark.zip/pyspark/ml/param/__init__.py", line 26, in <module> ImportError: No module named numpy   We followed office guied to install anaconda parcel,  And setup the Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh    export PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python export PYSPARK_DRIVER_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python Setup the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf   spark.yarn.appMasterEnv.PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python Also, setup the YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)   PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python PYSPARK_DRIVER_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python But non of these can help to solve the import issue.   Thanks for any help. ... View more
Contact Me
Online
Offline
Last Visited
‎08-02-2019 12:08 AM
Community Statistics
Member Since ‎05-13-2019 02:30 AM
Last Visited ‎08-02-2019 12:08 AM
Posts 11