Member since
01-08-2018
133
Posts
31
Kudos Received
21
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
11735 | 07-18-2018 01:29 AM | |
2144 | 06-26-2018 06:21 AM | |
3747 | 06-26-2018 04:33 AM | |
1932 | 06-21-2018 07:48 AM | |
1361 | 05-04-2018 04:04 AM |
03-14-2022
10:24 AM
1 Kudo
Hi everyone, I am a student in the UK doing MSc in Data Science. I would like to know which are the workbenches offered in Open Source. The HDP, Data Science Workbench and Quickstart VM seem to be unavailable. Would someone help me guide me in the right way?
... View more
12-07-2021
03:57 AM
yes you can update
... View more
10-12-2020
10:08 PM
I imported our existing v5.12 workflows via command-line loaddata. They show up in Hue 3 Oozie Editor, but not Hue 4. We are using CDH 5.16. I find the new "everything is document" paradigm confusing and misleading - Oozie workflows, Hive queries, Spark jobs etc. are not physical documents - in the Unix/HDFS sense that normal users would expect, with absolute paths that can be accessed and manipulated directly. The traditional-style Hue 3 UI lets one focus on working with the technology at hand, instead of imposing The Grand Unifying Design on the user.
... View more
08-12-2019
12:12 AM
TLS version mismatch is the issue. The below commands, fixed this issue for me, you can try this as well: echo 'export JAVA_TOOL_OPTIONS="-Dhttps.protocols=TLSv1.2"' >> ~/.bashrc source ~/.bashrc
... View more
05-15-2019
07:27 PM
OK, leave me and I will do a clean installation with parcels. Thanks Harsh. 🙂
... View more
11-14-2018
01:57 AM
Hi, I have the same problem, could you please give me some detail guid? Thanks a lot
... View more
10-24-2018
10:17 PM
@Huriye: What does that mean ? Please can you explain....
... View more
10-09-2018
09:04 AM
1 Kudo
Setting the cron job will take this particular error away but eventually, you are bound to run into a lot of other issues. Feel free to try though. Also, let me know your experience after trying that 🙂
... View more
09-27-2018
06:55 AM
@ramarov Thank you for the suggestion! We'll keep it in mind for our future sprint updates.
... View more
09-20-2018
10:26 PM
Here is output. Wondering if we can connect over webex? [root@myumyhost25 cloudera-scm-agent]# grep -v "^#" /etc/cloudera-scm-agent/config.ini | grep -v "^$" [General] server_host=myumyhost03 server_port=7182 [Security] use_tls=1 verify_cert_file=/opt/cloudera/security/x509/agents.pem monitored_nodev_filesystem_types=nfs,nfs4,tmpfs [root@myumyhost25 cloudera-scm-agent]# mount -l /dev/mapper/vg_myumyhost-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/xvdb1 on /boot type ext4 (rw) /dev/mapper/vg_data-lv_data on /data type ext4 (rw) /dev/mapper/vg_data-lv_home on /home type ext4 (rw) /dev/mapper/vg_data-lv_opt on /opt type ext4 (rw) /dev/mapper/vg_data-lv_var on /var type ext4 (rw) /dev/mapper/vg_myumyhost-lv_var_crash on /var/crash type ext4 (rw) /dev/mapper/vg_myumyhost-lv_var_log on /var/log type ext4 (rw) /iso/OEL6.9/V860937-01.iso on /var/OSimage/OL6.9_x86_64 type iso9660 (ro,loop=/dev/loop0) [OL6.9 x86_64 Disc 1 20170324] none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/blkio type cgroup (rw,blkio) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpuacct type cgroup (rw,cpuacct) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/cpu type cgroup (rw,cpu) cm_cgroups on /var/run/cloudera-scm-agent/cgroups/memory type cgroup (rw,memory) cm_processes on /var/run/cloudera-scm-agent/process type tmpfs (rw,mode=0751,rootcontext="unconfined_u:object_r:var_run_t:s0") [root@myumyhost25 cloudera-scm-agent]#
... View more
09-17-2018
10:41 AM
@HEWITT, Dang... that's odd. I don't see how we could be setting that automatically. What worries me here is that if you were able to make the configuration change and then restart, that would imply a successful write of the configuration change to the CM database. For the configuration to "revert" indicates the save was not successful after all. I would suggest trying again by restoring the value to the default (no value) and restarting CM. If the value comes back this time, try updating the value with the fully-qualified domain name of the Cloudera Manager host (the one used by the agents to heartbeat). Save/restart CM again. If that still reverts to "dev1" please share a screen shot with us and also look at the Cloudera Manager log file to see if there are any messages (errors, warnings) that indicate a problem writing to the database when you are making the configuration changes.
... View more
09-06-2018
01:53 AM
This is related to the JobHistoryServer log reported earlier. Please ensure/perform the following items for JHS and job completions to thoroughly work: First: Ensure that 'mapred' and 'yarn' are part of the 'hadoop' group in common: ~> hdfs groups mapred ~> hdfs groups yarn Both command must include 'hadoop' in their outputs. If not, ensure they are added to that group name. Second, all files and directories under HDFS /tmp/logs aggregation dir (or whatever you've reconfigured it to use) and /user/history/* have their group set to 'hadoop' and not anything else: ~> hadoop fs -chgrp -R hadoop /user/history /tmp/logs ~> hadoop fs -chmod -R g+rwx /user/history /tmp/logs Note: ACLs suggested earlier are not required to resolve this problem. The group used on these dirs is what matters in the default state, and the group setup described above is how YARN and JHS daemon users share information and responsibilities with each other. You may remove any ACLs set, or leave them be as they are still permissive.
... View more
08-30-2018
02:00 AM
After my research, remove the cluster node to operate up to two at a time, otherwise the data is at risk of being lost. And if the number of copies is insufficient, the system will not complete the removal operation, and finally have to retrieve the assigned role again.
... View more
08-24-2018
11:38 PM
@bgooley and @GeKas Your responses helped me According to @bgooley I have checked /var/log/cloudera-scm-server/cloudera-scm-server.log I found following can't connect to mysql server on 'localhost' (10061) Then I checked https://www.tecmint.com/fix-error-2003-hy000-cant-connect-to-mysql-server-on-127-0-0-1-111/ link to knowmon which IP address mysql-server is running by typing mysql -u root -p -h <hostname -I> That didn't work I tried with mysql -u root -p -h 127.0.0.1 now I got mysql promt on my terminal Then I have edited some config files where we define IP address 1) sudo vim /etc/cloudera-scm-agent/config.ini server_host=127.0.0.1 2) sudo vim /etc/hosts 127.0.0.1 Fully Qualified Domain Name sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf bind-address = 127.0.0.1 Now I have restarted the following processes sudo service ssh restart sudo service mysql restart sudo service cloudera-scm-server restart sudo service cloudera-scm-agent restart According to @GeKas link I have changed the permissions on /var/lib/mysql folder to 755 Then I tried to do database set up by giving credentials for hue, hive, activity monitor, reports manager, oozie. Then the database setup was completed successfully. Thank you !
... View more
08-22-2018
07:54 AM
If I understand the question, you want to remove a node from your Cluster and install Apache Hadoop directly. No requirement to preserve data. You can got to "Hosts-> All Hosts" in Cloudera Manager, select the node and from Actions menu you can do sequentially: Stop Roles on Hosts Remove from Cluster Remove from Cloudera Manager Before you do the last step, stop and disable cloudera-scm-agent # systemctl stop cloudera-scm-agent # systemctl disable cloudera-scm-agent Then, you can uninstall cloudera-scm-agent from this node, and install the Apache Hadoop.
... View more
07-27-2018
01:53 AM
I am using cloudera manager to handle my cluster. I found my problem. It was that I wanted to update a parameter that had already been configured by Cloudera manager team and that is a constant value. Cloudera manager doesn't allow to update some parameter like : io.storefile.bloom.block.size and the others constant parameters you cand find here : https://www.cloudera.com/documentation/other/shared/CDH5-Beta-2-RNs/hbase_jdiff_report-p-cdh4.5-c-cdh5b2/cdh4.5/constant-values.html So my problem is solved. Thank you very much for your help.
... View more
07-26-2018
11:44 PM
Hello, I changed the jar file name as mysql-connector-java.jar then I get error which password is not suitable for the policy. But removed my redhat machines and created Centos. Now I dont have any error to connecting mysql database. I still dont know what was the problem. Thanks, Huriye
... View more
07-20-2018
06:20 AM
When enabling YARN Cluster Utilization Report. Please make sure that User and Pool are existing. Also make sure that the User is a linux user on all Hadoop nodes.
... View more
07-18-2018
02:40 PM
1 Kudo
@yassine24, This shows how to update a service configuration http://cloudera.github.io/cm_api/docs/python-client/#configuring-services-and-roles You need to update the config with the attribute and value. The configuration is JSON format, but the safety valve you want is in XML format. An example of how to update a safety valve (hdfs in this case) via REST API is here: curl -iv -X PUT -H "Content-Type:application/json" -H "Accept:application/json" -d '{"items":[{ "name": "core_site_safety_valve","value": "<property><name>hadoop.proxyuser.ztsps.users</name><value>*</value></property><property><name>hadoop.proxyuser.ztsps.groups</name><value>*</value></property>"}]}' http://admin:admin@10.1.0.1:7180/api/v12/clusters/cluster/services/hdfs/config I am pretty sure you can pass the JSON as shown above in the -d argument hbase.update_config() or whatever
... View more
07-18-2018
01:44 AM
The error is that user impala has not write permissions to "/user/cloudera" directory and its contents (physician.csv). Write permissions are required, because impala will move this file from "/user/cloudera/physician.csv" to "/user/hive/warehouse/test.db/tablename/physician.csv"
... View more
07-18-2018
01:29 AM
1 Kudo
According to the error, it is looking for java 7 installed by cloudera. You should define JAVA_HOME={path_to_your_jdk8_installation} in bashrc.
... View more
07-13-2018
02:05 AM
The developer (customer side) who work with me on the cluster try to use Apache Airflow, and after one week, he can do what we need (workflow, emailing / alerting, re-run, ...) without the load of files into hdfs, Apache airflow is running in standalone mode and the web UI is better than Oozie UI. It seems a better solution than oozie, what do you think about this ? As it is an incubating project, I don't know if it's a good idea, but the web UI is good, it looks easy to manage, I didn't know this new project but I think Oozie is outdated compare to Airflow. For the moment Oozie is in stand-by, they will make a choice between oozie and airflow, but I must admit that Airflow looks a better solution.
... View more
07-13-2018
12:20 AM
Regarding python 2. If your hive server is configured with SSL, then you should consider installing "sasl" package in python. As about python3, although this is a python question not hive related, usually the issue is on the previous lines, e.g. quotes or parentheses that do not terminate.
... View more
07-12-2018
06:53 AM
Sentry is used as an authorization tool. Define rules what users are allowed to do. Sentry is not a tool to edit data, so no redaction can be done. If you meant "restrict" and that's a typo error, then the answer is positive, you can check https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_sg_sentry_service.html#hive_impala_privilege_model The easiest way is to use HUE. If you need to do it through Beeline, then check syntax in https://www.cloudera.com/documentation/enterprise/5-14-x/topics/sg_hive_sql.html
... View more
06-26-2018
12:00 PM
Here are the output: sudo -u hdfs hdfs dfs -ls / Found 3 items drwxr-xr-x - hbase hbase 0 2018-06-26 11:19 /hbase drwxrwxrwx - hdfs supergroup 0 2018-06-26 11:18 /tmp drwxrwxr-x - mapred mapred 0 2018-06-26 11:19 /user I filed another problem. The title is " Could not find yarn-site.xml, make sure to deploy yarn client in UI". I guess these problems may be related. Can you please take a look?
... View more
06-26-2018
09:46 AM
Thank you, I had mentioned alias in front of hostname in /etc/sysconfig/network file. i removed alias issue resolved .. hostname -f command helped me identify the issue Regards Siv
... View more
06-26-2018
06:27 AM
If you have enabled "HDFS-Sentry synchronization" then your setfacl actions will have no impact. Sentry rules are translated to ACL. You should use either HUE (Security/Sentry Roles) to fix the group or connect to beeline and use the grant/revoke commands.
... View more
06-26-2018
04:33 AM
Ok, from the log it is obvioues that issue for spark is the old jdk. When you tried to upgrade java have you defined the java home in "/etc/default/cloudera-scm-server" e.g.: export JAVA_HOME="/usr/lib/jvm/java-8-oracle/" Can you send the relevant "/var/log/cloudera-scm-server/cloudera-scm-server.out" ?
... View more
06-21-2018
07:48 AM
You should not worry for compatibility between KTS and CDH If you check https://www.cloudera.com/documentation/enterprise/latest/topics/encryption_ref_arch.html#concept_npk_rxh_1v CDH connects to KMS. KMS will connect to KTS So you have to check whether the KMS which is compatible to KTS3.8, is compatible with CDH5.14.2.
... View more
06-09-2018
03:00 AM
While running a wordcount program i am getting the following error. cloudera@localhost ~]$ hadoop jar WordCount.jar WordCount /inputnew2/inputfile.txt /output_new 18/06/09 00:29:06 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:07 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:08 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:09 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:10 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:11 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:12 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:13 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:14 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:15 INFO ipc.Client: Retrying connect to server: localhost.localdomain/127.0.0.1:8021. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 18/06/09 00:29:15 ERROR security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost.localdomain:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused Exception in thread "main" java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost.localdomain:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729) at org.apache.hadoop.ipc.Client.call(Client.java:1241) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225) at org.apache.hadoop.mapred.$Proxy10.getStagingAreaDir(Unknown Source) at org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1324) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:951) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945) at org.apache.hadoop.mapreduce.Job.submit(Job.java:566) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596) at WordCount.main(WordCount.java:132) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:509) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:603) at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1290) at org.apache.hadoop.ipc.Client.call(Client.java:1208) ... 18 more
... View more