Member since
07-06-2017
53
Posts
12
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
13509 | 05-03-2018 08:01 AM | |
8075 | 10-11-2017 08:17 AM | |
8632 | 07-20-2017 07:04 AM | |
608 | 04-05-2017 07:32 AM | |
1840 | 03-09-2017 12:05 PM |
05-04-2018
03:49 AM
Glad I could help.
... View more
05-03-2018
08:01 AM
2 Kudos
Hello, Support fixed it for me : The workaround for this is the following: copy snappy-java-1.1.4.jar to /opt/cloudera/parcels/SPARK2/lib/spark2/jars/ on each node where such executors are running. That can be downloaded from http://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar. Tested - confirmed working
... View more
04-24-2018
07:23 AM
1 Kudo
Hi, A bit trivial.. but 1/ Do you have the folder /var/log/spark2/lineage present on the Gateway instance of Spark2? 2/ is spark:spark listed as owner of said folder if it exist? I have a similar issue (CDH 5.14.1 / Spark 2.3 - Issue appaered after Spark 2.3 got enabled) though Yarn & workbench for which a case is open In the meantime, I disabled Navigator Lineage from cloudera Manager (spark2 Configuration / config.navigator.lineage_enabled) which allowed my colleagues to work.
... View more
10-11-2017
08:17 AM
2 Kudos
The Password policy / password format is indeed guilty : 1. Set the policy validation to low 2. ALTER USER 'root'@'localhost' IDENTIFIED BY 'Password1234'; 3. [root@scmtst ~]# /usr/share/cmf/schema/scm_prepare_database.sh -uroot -pPassword1234 --verbose mysql scmdb scmuser PAssword1234
JAVA_HOME=/usr/java/latest/
Verifying that we can write to /etc/cloudera-scm-server
Database type: mysql
Database user: root
Executing: /usr/java/latest//bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbProvisioner --create -h localhost -u root -H localhost -U scmuser -d scmdb -t mysql
Wed Oct 11 17:14:57 CEST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Creating SCM configuration file in /etc/cloudera-scm-server
Created db.properties file:
# Auto-generated by scm_prepare_database.sh on Wed 11 Oct 17:14:58 CEST 2017
#
# For information describing how to configure the Cloudera Manager Server
# to connect to databases, see the "Cloudera Manager Installation Guide."
#
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=localhost
com.cloudera.cmf.db.name=scmdb
com.cloudera.cmf.db.user=scmuser
com.cloudera.cmf.db.setupType=EXTERNAL
com.cloudera.cmf.db.password=PAssword1234
Executing: /usr/java/latest//bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Wed Oct 11 17:14:59 CEST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
... View more
10-11-2017
08:10 AM
I actually install a MySql intance on the SCM server directly, and It exposed the exact same behaviour: [root@cdhscmtst ~]# /usr/share/cmf/schema/scm_prepare_database.sh -uroot -p'<root password>' --verbose mysql scmdb scmuser 'scmuser password>'
JAVA_HOME=/usr/java/latest/
Verifying that we can write to /etc/cloudera-scm-server
Database type: mysql
Database user: root
Executing: /usr/java/latest//bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbProvisioner --create -h localhost -u root -H localhost -U scmuser -d scmdb -t mysql
Wed Oct 11 17:06:41 CEST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[ main] DbProvisioner ERROR Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://localhost/?useUnicode=true&characterEncoding=UTF-8'
java.sql.SQLException: Access denied for user 'root'@'localhost' (using password: YES) I kinda wonder if it might related to some special character mis=-intepreted in the passwords. I will disable the password policy in MySql & run more tests
... View more
10-11-2017
07:37 AM
Same behaviour with CDH 5.10.1 installer
... View more
- Tags:
- e b
10-11-2017
07:18 AM
Hello, I'm working of installing a brand new cluster, on a fresh dedicated MySql Instance. Servernames : SCM : scmtst Mysql : mysqltst001 CDH 5.12.1 freshly downalod & Installed MySQL 5.7.19 MySql connector 5.1.44 CentOs 7.3, configured according to Cloudera specs When i ran scm_prepare_database.sh on SCM, i keep getting "Access denied for user 'admin@'scmtst'" /usr/share/cmf/schema/scm_prepare_database.sh -u admin -h mysqltst001 -P 3306 -p mysql scmdb scmuser [root@scmtst ~]# /usr/share/cmf/schema/scm_prepare_database.sh -u admin -h mysqltst001 -P 3306 -p mysql scmdb scmuser
Enter database password:
Enter SCM password:
JAVA_HOME=/usr/java/latest/
Verifying that we can write to /etc/cloudera-scm-server
Wed Oct 11 14:50:28 CEST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[ main] DbProvisioner ERROR Exception when creating/dropping database with user 'admin' and jdbc url 'jdbc:mysql://mysqltst001:3306/?useUnicode=true&characterEncoding=UTF-8'
java.sql.SQLException: Access denied for user 'admin'@'scmtst' (using password: YES) In order to check that admin user was indeed set properly, I install the MySql client on scmtst, and I can indeed create and drop database at will if logged in as admin While investigating, I extracted the command-line actually called by scm_prepare_database.sh, and once i explicitly add the password (-p & -P) : it worked. The database got created, as well as the user & GRANTS [root@scmtst ~]# /usr/java/latest//bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbProvisioner --create -h mysqltst001.lab.ams:3306 -u admin -H scmtst.lab.ams -U scmuser -d scmdb -t mysql -p 'admin password' -P 'scm password'
Wed Oct 11 15:51:52 CEST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
=> Works However the script will fail later on on command com.cloudera.enterprise.dbutil.DbCommandExecutor. The file /etc/cloudera-scm-server/db.properties looks good to me, yet it does not connect [ main] DbCommandExecutor INFO Unable to login using supplied username/password.
[ main] DbCommandExecutor ERROR Error when connecting to database. It's lookng like the scripts scm_prepare_database.sh is not properly processing the passwords when attempting the logging to the MySql server. I coud not find any similar error, hence this post Thanks Chris
... View more
- Tags:
- MySQL
Labels:
- Labels:
-
Cloudera Manager
07-20-2017
07:04 AM
Hello, I fixed it. In the Spark2 Configuration Screen (In Cloudera Manager for the CDH cluster), Hiver Service was set to none I set it to Hive and CDSW is now working as expected. Thanks!
... View more
07-13-2017
07:14 AM
Hi Peter, See the roles installed. (In case of : Hive GW is the second listed role) From Spark2-shell (Workbench host) Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_67)
Type in expressions to have them evaluated.
Type :help for more information.
scala> spark.catalog.listTables.show()
+----+--------+-----------+---------+-----------+
|name|database|description|tableType|isTemporary|
+----+--------+-----------+---------+-----------+
+----+--------+-----------+---------+-----------+ From HUE running in the CDH cluster (table datascience): the table exist, and can be queried from HUE, the returned data is correct Spark2-shell from a CDH node scala> spark.catalog.listTables.show()
// Detected repl transcript. Paste more, or ctrl-D to finish.
+----+--------+-----------+---------+-----------+
|name|database|description|tableType|isTemporary|
+----+--------+-----------+---------+-----------+
+----+--------+-----------+---------+-----------+
scala> spark.sql("describe database default").show
+-------------------------+--------------------------+
|database_description_item|database_description_value|
+-------------------------+--------------------------+
| Database Name| default|
| Description| default database|
| Location| file:/log/clouder...|
+-------------------------+--------------------------+
scala> spark.sql("describe formatted default.datascience").show
org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'datascience' not found in database 'default';
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.requireTableExists(SessionCatalog.scala:138)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableMetadata(SessionCatalog.scala:289)
at org.apache.spark.sql.execution.command.DescribeTableCommand.run(tables.scala:437)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
... 48 elided
Thanks!
... View more
07-13-2017
04:54 AM
Hi Peter, I updated my test code as follow import org.apache.spark.sql.hive.HiveContext
spark.catalog.listTables.show()
val sqlContext = new HiveContext(sc)
sqlContext.sql("describe database default").show
sqlContext.sql("describe formatted default.mytable").show
sc.version the results for SparkSession: spark.catalog.listTables.show()
+----+--------+-----------+---------+-----------+
|name|database|description|tableType|isTemporary|
+----+--------+-----------+---------+-----------+
+----+--------+-----------+---------+-----------+ (There should be one table listed) The result for HiveContext sqlContext.sql("describe database default").show
+-------------------------+--------------------------+
|database_description_item|database_description_value|
+-------------------------+--------------------------+
| Database Name| default|
| Description| default database|
| Location| /user/hive/warehouse|
+-------------------------+--------------------------+
sqlContext.sql("describe formatted default.mytable").show
Name: org.apache.spark.sql.catalyst.analysis.NoSuchTableException
Message: Table or view 'mytable' not found in database 'default';
StackTrace: at org.apache.spark.sql.catalyst.catalog.SessionCatalog.requireTableExists(SessionCatalog.scala:138)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableMetadata(SessionCatalog.scala:289)
at org.apache.spark.sql.execution.command.DescribeTableCommand.run(tables.scala:437)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:699) while I do get an result for Describe Table, the description returned does not match the setting in Hive.
... View more
07-13-2017
04:33 AM
Hi Peter, The goal is to run Data Analysis using Spark where part of the data is stored in hive, The idea is to use the whole cluster to distribute the workload. I missed the fact that I was running in local mode (As said it's a pilot, i'm totally new to Cloudera stack). I assumed that the workbench was by default in yarn mode. I'll dig in the doc again. Trying to access hive using Sparksession object in my test was returning nothing. I'll try using yarn & report back thanks! Regards. Chris
... View more
07-13-2017
03:40 AM
Hello, While running a Pilot for CDSW & CDH, i'm struggling running queries on hive table. Scala Workbench Test code: import org.apache.spark.sql.Row
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.Column
val sparkSession = SparkSession.builder.master("local").appName("mapexample").
enableHiveSupport().getOrCreate()
sparkSession.catalog.listTables.show()
val sqlContext = new HiveContext(sc)
sqlContext.sql("describe database default").show
sqlContext.sql("describe formatted default.mytable").show
sc.version Test return import org.apache.spark.sql.Row
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.Column
val sparkSession = SparkSession.builder.master("local").appName("mapexample").
enableHiveSupport().getOrCreate()
sparkSession.catalog.listTables.show()
+----+--------+-----------+---------+-----------+
|name|database|description|tableType|isTemporary|
+----+--------+-----------+---------+-----------+
+----+--------+-----------+---------+-----------+
val sqlContext = new HiveContext(sc)
sqlContext.sql("describe database default").show
+-------------------------+--------------------------+
|database_description_item|database_description_value|
+-------------------------+--------------------------+
| Database Name| default|
| Description| default database|
| Location| /user/hive/warehouse|
+-------------------------+--------------------------+
sqlContext.sql("describe formatted default.mytable").show
Name: org.apache.spark.sql.catalyst.analysis.NoSuchTableException
Message: Table or view 'mytable' not found in database 'default';
StackTrace: at org.apache.spark.sql.catalyst.catalog.SessionCatalog.requireTableExists(SessionCatalog.scala:138)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableMetadata(SessionCatalog.scala:289)
at org.apache.spark.sql.execution.command.DescribeTableCommand.run(tables.scala:437)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:699) Interesing to note: Access hive though SparkSession returns nothing, while HiveContext does return a description of the default DB that does not match the actual default DB From Hue default
Default Hive database
hdfs://<masked>/user/hive/warehouse
public
ROLE The CDSW host has the right gateway roles installed, hive-site.xml is present with the spark config files. Hive Metastore log does not register any access when the workbench is trying to access Hive No Kerberos involved I ran out of option to check, hence this post Thanks Chris
... View more
Labels:
07-07-2017
01:40 AM
Found it. I interpreted "MASTER" as being Master node of the CDH cluster 😉 Unsing the right IP did fix the issue Thanks
... View more
07-07-2017
01:18 AM
Hello, The 3 last pods are not starting due some issue mounting volumes Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
15h 33s 224 {kubelet workbench} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/nfs/bee36b58-6247-11e7-9372-000d3a29b7ab-projects-share" (spec.Name: "projects-share") pod "bee36b58-6247-11e7-9372-000d3a29b7ab" (UID: "bee36b58-6247-11e7-9372-000d3a29b7ab") with: mount failed: exit status 32
Mounting arguments: 10.0.0.4:/var/lib/cdsw/current/projects /var/lib/kubelet/pods/bee36b58-6247-11e7-9372-000d3a29b7ab/volumes/kubernetes.io~nfs/projects-share nfs []
Output: mount.nfs: Connection timed out
19h 4s 502 {kubelet workbench} Warning FailedMount Unable to mount volumes for pod "web-3826671331-7xchm_default(bee36b58-6247-11e7-9372-000d3a29b7ab)": timeout expired waiting for volumes to attach/mount for pod "web-3826671331-7xchm"/"default". list of unattached/unmounted volumes=[projects-claim]
19h 4s 502 {kubelet workbench} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "web-3826671331-7xchm"/"default". list of unattached/unmounted volumes=[projects-claim] Google research did not really help me pointing waht could be the cause. Any pointers where is should look at ? Thanks!
... View more
07-06-2017
09:17 AM
Hi Peter, It actually did the trick $ sudo docker pull "docker.repository.cloudera.com/cdsw/1.0.1/web:052787a"
052787a: Pulling from cdsw/1.0.1/web
b6f892c0043b: Already exists
55010f332b04: Already exists
2955fb827c94: Already exists
3deef3fcbd30: Already exists
cf9722e506aa: Already exists
72923da64564: Already exists
3101e33a625d: Already exists
c03d5fa4b8e5: Already exists
35c1e4a8663c: Already exists
a1b3940356ad: Already exists
62370be47aba: Already exists
ddb5566a99f9: Already exists
8b5b82cdf853: Already exists
0c1a28ba377b: Already exists
5911a6a3d3db: Already exists
eb2b63f33d61: Already exists
3af8b8e8dc75: Already exists
19d9e7bce45d: Pull complete
396039e72b5e: Pull complete
b1fa7de66580: Pull complete
c15cd2ff85a4: Pull complete
87916a3ab13a: Pull complete
6c2fbb95a61e: Pull complete
938edf86928e: Pull complete
e0889d759edc: Extracting [==================================================>] 526.4 MB/526.4 MB
e0889d759edc: Pull complete
319dc7c60d62: Pull complete
dd1001380640: Pull complete
Digest: sha256:ecb807b8758acdfd1c6b0ff5acb1dad947cded312b47b60012c7478a0fcd9232
Status: Downloaded newer image for docker.repository.cloudera.com/cdsw/1.0.1/web:052787a Still a problem with 3 pods - will check that later Cloudera Data Science Workbench Status
Service Status
docker: active
kubelet: active
nfs: active
Checking kernel parameters...
Node Status
NAME STATUS AGE STATEFUL
workbench Ready 3h true
System Pod status
NAME READY STATUS RESTARTS AGE IP NODE
dummy-2088944543-uev12 1/1 Running 0 3h 10.0.0.5 workbench
etcd-workbench 1/1 Running 0 3h 10.0.0.5 workbench
kube-apiserver-workbench 1/1 Running 0 3h 10.0.0.5 workbench
kube-controller-manager-workbench 1/1 Running 3 3h 10.0.0.5 workbench
kube-discovery-1150918428-v7vu8 1/1 Running 0 3h 10.0.0.5 workbench
kube-dns-3873593988-vos07 3/3 Running 0 3h 100.66.0.2 workbench
kube-proxy-7qq63 1/1 Running 0 3h 10.0.0.5 workbench
kube-scheduler-workbench 1/1 Running 3 3h 10.0.0.5 workbench
node-problem-detector-v0.1-kngbh 1/1 Running 0 3h 10.0.0.5 workbench
weave-net-clu7s 2/2 Running 0 3h 10.0.0.5 workbench
Cloudera Data Science Workbench Pod Status
NAME READY STATUS RESTARTS AGE IP NODE ROLE
cron-2934152315-56p1n 1/1 Running 0 3h 100.66.0.8 workbench cron
db-39862959-icvq9 1/1 Running 1 3h 100.66.0.5 workbench db
db-migrate-052787a-mvb40 1/1 Running 0 3h 100.66.0.4 workbench db-migrate
engine-deps-du8cx 1/1 Running 0 3h 100.66.0.3 workbench engine-deps
ingress-controller-3138093376-l5z46 1/1 Running 0 3h 10.0.0.5 workbench ingress-controller
livelog-1900214889-qppq2 1/1 Running 0 3h 100.66.0.6 workbench livelog
reconciler-459456250-wgems 1/1 Running 0 3h 100.66.0.7 workbench reconciler
spark-port-forwarder-a31as 1/1 Running 0 3h 10.0.0.5 workbench spark-port-forwarder
web-3826671331-7xchm 0/1 ContainerCreating 0 3h <none> workbench web
web-3826671331-h3gkd 0/1 ContainerCreating 0 3h <none> workbench web
web-3826671331-vtbdh 0/1 ContainerCreating 0 3h <none> workbench web
Cloudera Data Science Workbench is not ready yet: some application pods are not ready Thanks!
... View more
07-06-2017
07:46 AM
Hello, New to Cloudera, I'm deploying the CDSW in Azure (on a Cloudera Centos7.2 template). The installation went ok, the Init started well, but eventually not all the pod will start: Cloudera Data Science Workbench Status
Service Status
docker: active
kubelet: active
nfs: active
Checking kernel parameters...
Node Status
NAME STATUS AGE STATEFUL
workbench Ready 2h true
System Pod status
NAME READY STATUS RESTARTS AGE IP NODE
dummy-2088944543-uev12 1/1 Running 0 2h 10.0.0.5 workbench
etcd-workbench 1/1 Running 0 2h 10.0.0.5 workbench
kube-apiserver-workbench 1/1 Running 0 2h 10.0.0.5 workbench
kube-controller-manager-workbench 1/1 Running 2 2h 10.0.0.5 workbench
kube-discovery-1150918428-v7vu8 1/1 Running 0 2h 10.0.0.5 workbench
kube-dns-3873593988-vos07 3/3 Running 0 2h 100.66.0.2 workbench
kube-proxy-7qq63 1/1 Running 0 2h 10.0.0.5 workbench
kube-scheduler-workbench 1/1 Running 2 2h 10.0.0.5 workbench
node-problem-detector-v0.1-kngbh 1/1 Running 0 2h 10.0.0.5 workbench
weave-net-clu7s 2/2 Running 0 2h 10.0.0.5 workbench
Cloudera Data Science Workbench Pod Status
NAME READY STATUS RESTARTS AGE IP NODE ROLE
cron-2934152315-56p1n 1/1 Running 0 2h 100.66.0.8 workbench cron
db-39862959-icvq9 1/1 Running 1 2h 100.66.0.5 workbench db
db-migrate-052787a-mvb40 0/1 ImagePullBackOff 0 2h 100.66.0.4 workbench db-migrate
engine-deps-du8cx 1/1 Running 0 2h 100.66.0.3 workbench engine-deps
ingress-controller-3138093376-l5z46 1/1 Running 0 2h 10.0.0.5 workbench ingress-controller
livelog-1900214889-qppq2 1/1 Running 0 2h 100.66.0.6 workbench livelog
reconciler-459456250-wgems 1/1 Running 0 2h 100.66.0.7 workbench reconciler
spark-port-forwarder-a31as 1/1 Running 0 2h 10.0.0.5 workbench spark-port-forwarder
web-3826671331-7xchm 0/1 ContainerCreating 0 2h <none> workbench web
web-3826671331-h3gkd 0/1 ContainerCreating 0 2h <none> workbench web
web-3826671331-vtbdh 0/1 ContainerCreating 0 2h <none> workbench web
Cloudera Data Science Workbench is not ready yet: some application pods are not ready $sudo journalctl -u docker
Jul 06 13:42:03 workbench docker[6669]: time="2017-07-06T13:42:03.996814534Z" level=error msg="Handler for GET /images/docker.repository.cloudera.com/cdsw/1.0.1/web:052787a/json returned error: No such image: docker.repository.cloudera.com/cdsw/1.0.1/we The access to Internet is available (As most of the other pods are started) Any ideas ? Thanks
... View more
- Tags:
- Installation
Labels:
05-29-2017
07:25 AM
1 Kudo
Hi, Sqoop will add (using --incremental) the new data added to the source RDBMS, but will not update the data already ingested. You'll need manual process, or third-party tooling to keep your Hadoop copy in sync with the RDBMS.
... View more
05-24-2017
02:16 PM
1 Kudo
Hello, After upgrading HDP to 2.6 and getting the GA release of Spark 2.1, i'm trying (to no luck so far) to add Spark2 Interpreter in Zeppelin (if possible at all). I did create a new interpreter spark2 in Zeppelin which will be instantiate properly (%spark2), however sc.version indicates that i'm still running Spark 1.6.3. Digging in the config & the doc, I found out that SPARK_HOME is defined in zeppelin-env.sh, pointing by default to Spark 1.6.63. Editing the config & restart Zeppelin will "work" in the sense that I can now successfully instantiate Spark2, but Spark 1.6.3 is not available anymore from the notebook (and livy is still configured to Spark 1.6.3). Is there any way to create interpreters to allow using both Spark 1.6.3 & Spark 2 from Zeppelin 0.7 ? Thanks Christophe
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache Zeppelin
05-23-2017
06:32 PM
Hello Can you post the job.properties & workflow.xml? I'd say you miss hive-site.xml in hdfs://<your path to workflow.xml>/lib
... View more
05-23-2017
06:24 PM
2 Kudos
Hi, As far as I know, you will not be able to compress "on-wire" with MS SQL. This will not directly answer your question, but there might few track to investigate: - Use SQOOP with incremental ingestion. I manage to ingest 600GB in <20min (be careful with the number of threads, Sqoop can be very hard on the SQL Server). Sqoop can wirte straight to Hive - Assuming this is a bulk, you can try to BCP the data to (SQL) disk, compress the BCP, transmit to HDFS before de-compress & ingestion.
... View more
04-06-2017
12:45 PM
@rguruvannagari I did regenerate the cert, check the FQDN etc against the certificate used on the precedent machine: all seems good to me. Still the same problem, ONLY for Ambari. I'm really puzzled 🙂 What I forgot to mention I realize: Oozie before was on the Ambari server.
... View more
04-05-2017
07:32 AM
Fixed! The HDFS client was not installed on the new Oozie server.
... View more
04-05-2017
06:45 AM
@rguruvannagari I checked and the truststore was already set as it should. The problem seems to be located @the agent side, not on the server itself. See logs & report below. I have re-checked the truststores on both sides, check the ownership & permissions to no luck. The ambari-agent does not list truststore in its config, so I'm not sure where to look here. I'm still puzzled by the fact that oozie command works just fine when run form the prompt. Thanks! Ambari-agent log (on the nodes running Oozie server)
ERROR 2017-04-05 08:26:47,657 script_alert.py:119 - [Alert][oozie_server_status] Failed with result CRITICAL: ["Execution of 'source /usr/hdp/current/oozie-server/conf/oozie-env.sh ; oozie admin -oozie https://FQDNoozie:port/oozie -status' returned 255. Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"]
ERROR 2017-04-05 08:26:47,657 script_alert.py:119 - [Alert][oozie_server_status] Failed with result CRITICAL: ["Execution of 'source /usr/hdp/current/oozie-server/conf/oozie-env.sh ; oozie admin -oozie https://FQDNoozie:port/oozie -status' returned 255. Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"]
INFO 2017-04-05 08:26:59,681 Controller.py:277 - Heartbeat with server is running...
Oozie call from the command line
[user@server conf]$ oozie admin -oozie https://FQDNoozie:port/oozie -status
System mode: NORMAL
Ambari-alert stack
Execution of 'source /usr/hdp/current/oozie-server/conf/oozie-env.sh ; oozie admin -oozie https://FQDNoozie:port/oozie -status' returned 255. Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
... View more
04-04-2017
09:11 PM
@Swapan Shridhar The Oozie logs are clean indeed so n regard with this issue (and running Oozie admin command from the Linux prompt does return a valid healthy status) The PKI chain error is indeed in the Oozie alert (and matched in the ambari agent log file). I'll post the stack tomorrow. (I just saw I made a typo in original post : I could NOT locate what for truststore ambari agent relies on ) Thanks
... View more
04-04-2017
06:16 PM
Hello, after i moved Oozie between server, ambari flags the server in Error as "Oozie admin -Oozie <URL> -status " returns a PKI check errors. I have check all the key stores and trust store to be ok, the root cert is added to the Linux trusted CA (server was rebooted in case). The same command from Linux Prompt works flawless. Ambari-agent temp and cache have been cleared, I could find where ambari-agent is retrieving his trust store. Any ideas.? Thanks Christoohe
... View more
Labels:
- Labels:
-
Apache Ambari
04-04-2017
06:10 PM
Hello I moved my Oozie server between nodes today and while the move was successful for Ambari, and in the great lines, all the Coordinator jobs are failing since with Error 10005, java Exception UnknownHost, 'ha name of the name nodes' Apparently Oozie server is struggling located The name node which is in HA mode The hdfs- and core-site.xml look good and in sync with the rest of the cluster, the proxy config in Hdfs and other services are properly set. I'm running out of idea on where to look. Any suggestions? Thanks Christophe
... View more
- Tags:
- Oozie
Labels:
- Labels:
-
Apache Oozie
03-23-2017
10:27 AM
Hi @Binu Mathew Thanks for your answer. I'll dive into this approach & post further if/when required. Thanks! Christoohe
... View more
03-21-2017
09:18 AM
Hello, I'm still new to Hadoop technology and I'm struggling defining the best approach for the following 2 similar challenges: In first instance, trying to ingest the files -> Hive. InferAVROSchema in Nifi is limited as it does not always recognize the right data type, generating a fair amount of error when the files are ingested. Switching to specifiy the schema manually bring the following problems: - Ingesting CSV files that have schema updates over the year, I have a versioning documentation giving me the schema changes, however the date in the versioning document do not match the date of effective change. - Ingesting hourly CSV files with a schema depending of the business activity (a set of columns is mandatory, a large set is optionnal and will only be seen when the underling options have been used) . The schema of the files is different from hours to hours, and I can't predict which one is to expect. My feelings are that I have to move to NoSql type of DB / storage, but I'm not exactly sure how to tackle this in the best way. Has anyone faced similar problematic? Thanks Christophe
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
-
Apache NiFi
03-16-2017
10:39 AM
Hi, This bug can have consequences on Spark / Yarn as well. We were encountering Out of Memory conditions running Spark job, not matter how much memory we assigned, we kept ending up exhausting it completely. This behaviour actually disappeared when we applied the fix listed here. I'll post back when I know more about the root cause & link between issues. Regards, Christophe
... View more
03-14-2017
04:21 PM
Hi @Matt Clarke You nailed it again. The two "old" hosts have Owner : CN=FQDN, etc The new host has Owner: CN=<Shortname>, etc Seems the 2 initial hosts cert have been created by a different script that the one I got. It's not clear where is the problem. I'll check and update the cert where needed. Thanks!
... View more