Re: Unable to launch spark using spark-shell on li...

aswinkv28 · ‎03-14-2022

Hi everyone, I am a student in the UK doing MSc in Data Science. I would like to know which are the workbenches offered in Open Source. The HDP, Data Science Workbench and Quickstart VM seem to be unavailable. Would someone help me guide me in the right way?

spiceagent · ‎12-07-2021

yes you can update

MilesYao · ‎10-12-2020

I imported our existing v5.12 workflows via command-line loaddata. They show up in Hue 3 Oozie Editor, but not Hue 4. We are using CDH 5.16. I find the new "everything is document" paradigm confusing and misleading - Oozie workflows, Hive queries, Spark jobs etc. are not physical documents - in the Unix/HDFS sense that normal users would expect, with absolute paths that can be accessed and manipulated directly. The traditional-style Hue 3 UI lets one focus on working with the technology at hand, instead of imposing The Grand Unifying Design on the user.

vkarthikeyn · ‎08-12-2019

TLS version mismatch is the issue. The below commands, fixed this issue for me, you can try this as well: echo 'export JAVA_TOOL_OPTIONS="-Dhttps.protocols=TLSv1.2"' >> ~/.bashrc source ~/.bashrc

wiwengweng · ‎05-15-2019

OK, leave me and I will do a clean installation with parcels. Thanks Harsh. 🙂

DanielDuan · ‎11-14-2018

Hi, I have the same problem, could you please give me some detail guid? Thanks a lot

deepak_ahirwar · ‎10-24-2018

@Huriye: What does that mean ? Please can you explain....

sim6 · ‎10-09-2018

Setting the cron job will take this particular error away but eventually, you are bound to run into a lot of other issues. Feel free to try though. Also, let me know your experience after trying that 🙂

cflanagan · ‎09-27-2018

@ramarov Thank you for the suggestion! We'll keep it in mind for our future sprint updates.

xBigDatax · ‎09-20-2018

Here is output. Wondering if we can connect over webex? [root@myumyhost25 cloudera-scm-agent]# grep -v "^#" /etc/cloudera-scm-agent/config.ini | grep -v "^$" [General] server_host=myumyhost03 server_port=7182 [Security] use_tls=1 verify_cert_file=/opt/cloudera/security/x509/agents.pem monitored_nodev_filesystem_types=nfs,nfs4,tmpfs [root@myumyhost25 cloudera-scm-agent]# mount -l /dev/mapper/vg_myumyhost-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/xvdb1 on /boot type ext4 (rw) /dev/mapper/vg_data-lv_data on /data type ext4 (rw) /dev/mapper/vg_data-lv_home on /home type ext4 (rw) /dev/mapper/vg_data-lv_opt on /opt type ext4 (rw) /dev/mapper/vg_data-lv_var on /var type ext4 (rw) /dev/mapper/vg_myumyhost-lv_var_crash on /var/crash type ext4 (rw) /dev/mapper/vg_myumyhost-lv_var_log on /var/log type ext4 (rw) /iso/OEL6.9/V860937-01.iso on /var/OSimage/OL6.9_x86_64 type iso9660 (ro,loop=/dev/loop0) [OL6.9 x86_64 Disc 1 20170324] none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/blkio type cgroup (rw,blkio) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpuacct type cgroup (rw,cpuacct) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpu type cgroup (rw,cpu) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/memory type cgroup (rw,memory) cm_processes on /var/run/cloudera-scm-agent/process type tmpfs (rw,mode=0751,rootcontext="unconfined_u:object_r:var_run_t:s0") [root@myumyhost25 cloudera-scm-agent]#

bgooley · ‎09-17-2018

@HEWITT, Dang... that's odd. I don't see how we could be setting that automatically. What worries me here is that if you were able to make the configuration change and then restart, that would imply a successful write of the configuration change to the CM database. For the configuration to "revert" indicates the save was not successful after all. I would suggest trying again by restoring the value to the default (no value) and restarting CM. If the value comes back this time, try updating the value with the fully-qualified domain name of the Cloudera Manager host (the one used by the agents to heartbeat). Save/restart CM again. If that still reverts to "dev1" please share a screen shot with us and also look at the Cloudera Manager log file to see if there are any messages (errors, warnings) that indicate a problem writing to the database when you are making the configuration changes.

Harsh J · ‎09-06-2018

This is related to the JobHistoryServer log reported earlier. Please ensure/perform the following items for JHS and job completions to thoroughly work: First: Ensure that 'mapred' and 'yarn' are part of the 'hadoop' group in common: ~> hdfs groups mapred ~> hdfs groups yarn Both command must include 'hadoop' in their outputs. If not, ensure they are added to that group name. Second, all files and directories under HDFS /tmp/logs aggregation dir (or whatever you've reconfigured it to use) and /user/history/* have their group set to 'hadoop' and not anything else: ~> hadoop fs -chgrp -R hadoop /user/history /tmp/logs ~> hadoop fs -chmod -R g+rwx /user/history /tmp/logs Note: ACLs suggested earlier are not required to resolve this problem. The group used on these dirs is what matters in the default state, and the group setup described above is how YARN and JHS daemon users share information and responsibilities with each other. You may remove any ACLs set, or leave them be as they are still permissive.

BTibetanMastiff · ‎08-30-2018

After my research, remove the cluster node to operate up to two at a time, otherwise the data is at risk of being lost. And if the number of copies is insufficient, the system will not complete the removal operation, and finally have to retrieve the assigned role again.

saikrishnamante · ‎08-24-2018

@bgooley and @GeKas Your responses helped me According to @bgooley I have checked /var/log/cloudera-scm-server/cloudera-scm-server.log I found following can't connect to mysql server on 'localhost' (10061) Then I checked https://www.tecmint.com/fix-error-2003-hy000-cant-connect-to-mysql-server-on-127-0-0-1-111/ link to knowmon which IP address mysql-server is running by typing mysql -u root -p -h <hostname -I> That didn't work I tried with mysql -u root -p -h 127.0.0.1 now I got mysql promt on my terminal Then I have edited some config files where we define IP address 1) sudo vim /etc/cloudera-scm-agent/config.ini server_host=127.0.0.1 2) sudo vim /etc/hosts 127.0.0.1 Fully Qualified Domain Name sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf bind-address = 127.0.0.1 Now I have restarted the following processes sudo service ssh restart sudo service mysql restart sudo service cloudera-scm-server restart sudo service cloudera-scm-agent restart According to @GeKas link I have changed the permissions on /var/lib/mysql folder to 755 Then I tried to do database set up by giving credentials for hue, hive, activity monitor, reports manager, oozie. Then the database setup was completed successfully. Thank you !

GeKas · ‎08-22-2018

If I understand the question, you want to remove a node from your Cluster and install Apache Hadoop directly. No requirement to preserve data. You can got to "Hosts-> All Hosts" in Cloudera Manager, select the node and from Actions menu you can do sequentially: Stop Roles on Hosts Remove from Cluster Remove from Cloudera Manager Before you do the last step, stop and disable cloudera-scm-agent # systemctl stop cloudera-scm-agent # systemctl disable cloudera-scm-agent Then, you can uninstall cloudera-scm-agent from this node, and install the Apache Hadoop.

yassine24 · ‎07-27-2018

I am using cloudera manager to handle my cluster. I found my problem. It was that I wanted to update a parameter that had already been configured by Cloudera manager team and that is a constant value. Cloudera manager doesn't allow to update some parameter like : io.storefile.bloom.block.size and the others constant parameters you cand find here : https://www.cloudera.com/documentation/other/shared/CDH5-Beta-2-RNs/hbase_jdiff_report-p-cdh4.5-c-cdh5b2/cdh4.5/constant-values.html So my problem is solved. Thank you very much for your help.

Huriye · ‎07-26-2018

Hello, I changed the jar file name as mysql-connector-java.jar then I get error which password is not suitable for the policy. But removed my redhat machines and created Centos. Now I dont have any error to connecting mysql database. I still dont know what was the problem. Thanks, Huriye

amekala · ‎07-20-2018

When enabling YARN Cluster Utilization Report. Please make sure that User and Pool are existing. Also make sure that the User is a linux user on all Hadoop nodes.

bgooley · ‎07-18-2018

@yassine24, This shows how to update a service configuration http://cloudera.github.io/cm_api/docs/python-client/#configuring-services-and-roles You need to update the config with the attribute and value. The configuration is JSON format, but the safety valve you want is in XML format. An example of how to update a safety valve (hdfs in this case) via REST API is here: curl -iv -X PUT -H "Content-Type:application/json" -H "Accept:application/json" -d '{"items":[{ "name": "core_site_safety_valve","value": "<property><name>hadoop.proxyuser.ztsps.users</name><value>*</value></property><property><name>hadoop.proxyuser.ztsps.groups</name><value>*</value></property>"}]}' http://admin:admin@10.1.0.1:7180/api/v12/clusters/cluster/services/hdfs/config I am pretty sure you can pass the JSON as shown above in the -d argument hbase.update_config() or whatever

GeKas · ‎07-18-2018

The error is that user impala has not write permissions to "/user/cloudera" directory and its contents (physician.csv). Write permissions are required, because impala will move this file from "/user/cloudera/physician.csv" to "/user/hive/warehouse/test.db/tablename/physician.csv"

GeKas · ‎07-18-2018

According to the error, it is looking for java 7 installed by cloudera. You should define JAVA_HOME={path_to_your_jdk8_installation} in bashrc.

JSenzier · ‎07-13-2018

The developer (customer side) who work with me on the cluster try to use Apache Airflow, and after one week, he can do what we need (workflow, emailing / alerting, re-run, ...) without the load of files into hdfs, Apache airflow is running in standalone mode and the web UI is better than Oozie UI. It seems a better solution than oozie, what do you think about this ? As it is an incubating project, I don't know if it's a good idea, but the web UI is good, it looks easy to manage, I didn't know this new project but I think Oozie is outdated compare to Airflow. For the moment Oozie is in stand-by, they will make a choice between oozie and airflow, but I must admit that Airflow looks a better solution.

GeKas · ‎07-13-2018

Regarding python 2. If your hive server is configured with SSL, then you should consider installing "sasl" package in python. As about python3, although this is a python question not hive related, usually the issue is on the previous lines, e.g. quotes or parentheses that do not terminate.

GeKas · ‎07-12-2018

Sentry is used as an authorization tool. Define rules what users are allowed to do. Sentry is not a tool to edit data, so no redaction can be done. If you meant "restrict" and that's a typo error, then the answer is positive, you can check https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_sg_sentry_service.html#hive_impala_privilege_model The easiest way is to use HUE. If you need to do it through Beeline, then check syntax in https://www.cloudera.com/documentation/enterprise/5-14-x/topics/sg_hive_sql.html

PeterLuo · ‎06-26-2018

Here are the output: sudo -u hdfs hdfs dfs -ls / Found 3 items drwxr-xr-x - hbase hbase 0 2018-06-26 11:19 /hbase drwxrwxrwx - hdfs supergroup 0 2018-06-26 11:18 /tmp drwxrwxr-x - mapred mapred 0 2018-06-26 11:19 /user I filed another problem. The title is " Could not find yarn-site.xml, make sure to deploy yarn client in UI". I guess these problems may be related. Can you please take a look?

Siv · ‎06-26-2018

Thank you, I had mentioned alias in front of hostname in /etc/sysconfig/network file. i removed alias issue resolved .. hostname -f command helped me identify the issue Regards Siv

GeKas · ‎06-26-2018

If you have enabled "HDFS-Sentry synchronization" then your setfacl actions will have no impact. Sentry rules are translated to ACL. You should use either HUE (Security/Sentry Roles) to fix the group or connect to beeline and use the grant/revoke commands.

GeKas · ‎06-26-2018

Ok, from the log it is obvioues that issue for spark is the old jdk. When you tried to upgrade java have you defined the java home in "/etc/default/cloudera-scm-server" e.g.: export JAVA_HOME="/usr/lib/jvm/java-8-oracle/" Can you send the relevant "/var/log/cloudera-scm-server/cloudera-scm-server.out" ?

GeKas · ‎06-21-2018

You should not worry for compatibility between KTS and CDH If you check https://www.cloudera.com/documentation/enterprise/latest/topics/encryption_ref_arch.html#concept_npk_rxh_1v CDH connects to KMS. KMS will connect to KTS So you have to check whether the KMS which is compatible to KTS3.8, is compatible with CDH5.14.2.

subbu · ‎06-09-2018

While running a wordcount program i am getting the following error. cloudera@localhost ~]$ hadoop jar WordCount.jar WordCount /inputnew2/inputfile.txt /output_new 18/06/09 00:29:06 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:07 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:08 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:09 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:10 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:11 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:12 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:13 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:14 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:15 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:15 ERROR security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost.localdomain:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused Exception in thread "main" java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost.localdomain:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729) at org.apache.hadoop.ipc.Client.call(Client.java:1241) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225) at org.apache.hadoop.mapred.$Proxy10.getStagingAreaDir(Unknown Source) at org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1324) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:951) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945) at org.apache.hadoop.mapreduce.Job.submit(Job.java:566) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596) at WordCount.main(WordCount.java:132) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:509) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:603) at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1290) at org.apache.hadoop.ipc.Client.call(Client.java:1208) ... 18 more

Online	Offline
Last Visited	‎10-27-2021 07:02 AM

Date Registered	‎01-08-2018 08:01 AM
Last Visited	‎10-27-2021 07:02 AM
Total Messages Posted	133
Total Kudos Received	31

Re: Unable to launch spark using spark-shell on li...

Re: Failed to start Cloudera Manager Agent

Re: Spark 2 Installation Failed

Re: KTS and CDH Compatability

Re: Disabling Kafka Sentry

Re: INTRODUCE YOURSELF: Please Reply here to tell ...

Re: Can i upgrade my Java version to Java SE 8u171...

Re: Cannot find Oozie Workflows Editor in Hue 4

Re: issue with spark-shell package dependency from...

Re: not able to activate KAFKA

Re: impala-shell -k with ha proxy not working

Re: Unable to connect to database on host 'clouder...

Re: hostname and canonical name for this host are ...

Re: How can we better serve you with this communit...

Re: Failed to add host

Re: NOT ABLE to add new hosts with CDH6

Re: HDFS or HIVE Replication

Re: How to remove a large number of nodes from the...

Re: Problem in database set up in cloudera manager

Re: How can I stop and uninstall only one node wit...

Re: Cloudera manager API modifiy service HBASE

Re: scm_prepare_database.sh failure

Re: I don't see any YARN reports in Cloudera Manag...

Re: Hbase configuration hbase-site.xml command lin...

Re: Impala Unable to Load Data

Re: Unable to launch spark using spark-shell on li...

Re: Oozie - Create a Spark workflow

Re: Using python to connect to Hive in cloudera

Re: It is possible to do hive column level redacti...

Re: Failed to become active master org.apache.hado...

Re: Failed to start Cloudera Manager Agent

Re: Not able to delete sentry permission on Hive D...

Re: Spark 2 Installation Failed

Re: KTS and CDH Compatability

Runnigcloudera@localhost ~]$ hadoop jar WordCount....