Member since
07-06-2018
24
Posts
4
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3009 | 11-06-2019 04:11 AM | |
2158 | 10-31-2018 04:56 AM | |
6431 | 10-14-2018 11:04 AM | |
7550 | 07-18-2018 12:26 AM |
03-25-2019
12:42 PM
1 Kudo
Hello, is it possible for one single user to allocate more than 50% of memory on cdsw because I'm not able to start a session with more than 50% of memory? Actually this makes sense if this is a limitation from cdsw but most of the time I use cdsw alone in our company and I wanted to ask if there is a parameter which I can set to allocate more memory. Thanks.
... View more
Labels:
10-31-2018
04:56 AM
The problem could be solved with setting up the Wildcard DNS Subdomain for CDSW properly. (CDSW 1.4.X documentation page 51) Thanks.
... View more
10-30-2018
02:14 AM
I could successfully deploy CDSW 1.4.2 on my CDH 5.14 with the Cloudera parcel. If I create a new project and open the workbench I get this error message on the right side of the workbench: "The IP-Address of consoles.hadoop-workbench... could not be found" All the initial templates (Python, Scala, R..) gave me the same result. How can I resolve this problem? Grateful for any help!
... View more
Labels:
10-15-2018
08:59 AM
EDIT: I did a copy/paste mistake! Please ignore my full-log given in my post above. Here is the correct Error-Log: /usr/share/cmf/bin/gen_credentials.sh failed with exit code 1 and output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ CMF_REALM=MYCOMPANY.REALM
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf2548823212650177196.keytab
+ PRINC=sdc/hostname.FQDN@MYCOMPANY.REALM
+ MAX_RENEW_LIFE=604800
+ KADMIN='kadmin -k -t /var/run/cloudera-scm-server/cmf6838080336847771087.keytab -p admin/admin@MYCOMPANY.REALM -r MYCOMPANY.REALM'
+ RENEW_ARG=
+ '[' 604800 -gt 0 ']'
+ RENEW_ARG='-maxrenewlife "604800 sec"'
+ '[' -z /var/run/cloudera-scm-server/krb52847952611766397096.conf ']'
+ echo 'Using custom config path '\''/var/run/cloudera-scm-server/krb52847952611766397096.conf'\'', contents below:'
+ cat /var/run/cloudera-scm-server/krb52847952611766397096.conf
+ kadmin -k -t /var/run/cloudera-scm-server/cmf6838080336847771087.keytab -p admin/admin@MYCOMPANY.REALM -r MYCOMPANY.REALM -q 'addprinc -maxrenewlife "604800 sec" -randkey sdc/hostname.FQDN@MYCOMPANY.REALM'
WARNING: no policy specified for sdc/hostname.FQDN@MYCOMPANY.REALM; defaulting to no policy
add_principal: Operation requires ``add'' privilege while creating "sdc/hostname.FQDN@MYCOMPANY.REALM".
+ '[' 604800 -gt 0 ']'
++ kadmin -k -t /var/run/cloudera-scm-server/cmf6838080336847771087.keytab -p admin/admin@MYCOMPANY.REALM -r MYCOMPANY.REALM -q 'getprinc -terse sdc/hostname.FQDN@MYCOMPANY.REALM'
++ tail -1
++ cut -f 12
get_principal: Operation requires ``get'' privilege while retrieving "sdc/hostname.FQDN@MYCOMPANY.REALM".
+ RENEW_LIFETIME='Authenticating as principal admin/admin@MYCOMPANY.REALM with keytab /var/run/cloudera-scm-server/cmf6838080336847771087.keytab.'
+ '[' Authenticating as principal admin/admin@MYCOMPANY.REALM with keytab /var/run/cloudera-scm-server/cmf6838080336847771087.keytab. -eq 0 ']'
/usr/share/cmf/bin/gen_credentials.sh: line 35: [: too many arguments
+ kadmin -k -t /var/run/cloudera-scm-server/cmf6838080336847771087.keytab -p admin/admin@MYCOMPANY.REALM -r MYCOMPANY.REALM -q 'xst -k /var/run/cloudera-scm-server/cmf2548823212650177196.keytab sdc/hostname.FQDN@MYCOMPANY.REALM'
kadmin: Operation requires ``change-password'' privilege while changing sdc/hostname.FQDN@MYCOMPANY.REALM's key
+ chmod 600 /var/run/cloudera-scm-server/cmf2548823212650177196.keytab
chmod: cannot access ‘/var/run/cloudera-scm-server/cmf2548823212650177196.keytab’: No such file or directory
>>
... View more
10-14-2018
12:02 PM
Hello, this is maybe an old post but I'm struggling with the same problem and didn't wanted to open a new thread. I'm working with CentOS 7.5 and added two new gateway hosts for StreamSets. After I installed successfully StreamSets with the Parcel I created the principals for Kerberos with kadmin.local for both hosts like this: add_principal sdc/hostname1.FQDN@MYCOMPANY.REALM add_principal sdc/hostname2.FQDN@MYCOMPANY.REALM After this step I wanted to create the missing Kerberos credentials over Cloudera Manager which fails. I'm not sure with this line. Is this maybe the problem? /usr/share/cmf/bin/gen_credentials.sh: line 35: [: too many arguments My full log file: /usr/share/cmf/bin/gen_credentials.sh failed with exit code 1 and output of <<
+ export PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ PATH=/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin:/sbin:/usr/sbin:/bin:/usr/bin
+ CMF_REALM=MYCOMPANY.REALM
+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf3411206514354101952.keytab
+ PRINC=sdc/hostname1.FQDN@MYCOMPANY.REALM
+ MAX_RENEW_LIFE=604800
+ KADMIN='kadmin -k -t /var/run/cloudera-scm-server/cmf5906055974897109624.keytab -p sdc/hostname2.FQDN@MYCOMPANY.REALM -r MYCOMPANY.REALM'
+ RENEW_ARG=
+ '[' 604800 -gt 0 ']'
+ RENEW_ARG='-maxrenewlife "604800 sec"'
+ '[' -z /var/run/cloudera-scm-server/krb57389542731171685362.conf ']'
+ echo 'Using custom config path '\''/var/run/cloudera-scm-server/krb57389542731171685362.conf'\'', contents below:'
+ cat /var/run/cloudera-scm-server/krb57389542731171685362.conf
+ kadmin -k -t /var/run/cloudera-scm-server/cmf5906055974897109624.keytab -p sdc/hostname2.FQDN@MYCOMPANY.REALM -r MYCOMPANY.REALM -q 'addprinc -maxrenewlife "604800 sec" -randkey sdc/hostname1.FQDN@MYCOMPANY.REALM'
WARNING: no policy specified for sdc/hostname1.FQDN@MYCOMPANY.REALM; defaulting to no policy
add_principal: Operation requires ``add'' privilege while creating "sdc/hostname1.FQDN@MYCOMPANY.REALM".
+ '[' 604800 -gt 0 ']'
++ kadmin -k -t /var/run/cloudera-scm-server/cmf5906055974897109624.keytab -p sdc/hostname2.FQDN@MYCOMPANY.REALM -r MYCOMPANY.REALM -q 'getprinc -terse sdc/hostname1.FQDN@MYCOMPANY.REALM'
++ tail -1
++ cut -f 12
get_principal: Operation requires ``get'' privilege while retrieving "sdc/hostname1.FQDN@MYCOMPANY.REALM".
+ RENEW_LIFETIME='Authenticating as principal sdc/hostname2.FQDN@MYCOMPANY.REALM with keytab /var/run/cloudera-scm-server/cmf5906055974897109624.keytab.'
+ '[' Authenticating as principal sdc/hostname2.FQDN@MYCOMPANY.REALM with keytab /var/run/cloudera-scm-server/cmf5906055974897109624.keytab. -eq 0 ']'
/usr/share/cmf/bin/gen_credentials.sh: line 35: [: too many arguments
+ kadmin -k -t /var/run/cloudera-scm-server/cmf5906055974897109624.keytab -p sdc/hostname2.FQDN@MYCOMPANY.REALM -r MYCOMPANY.REALM -q 'xst -k /var/run/cloudera-scm-server/cmf3411206514354101952.keytab sdc/hostname1.FQDN@MYCOMPANY.REALM'
kadmin: Operation requires ``change-password'' privilege while changing sdc/hostname1.FQDN@MYCOMPANY.REALM's key
+ chmod 600 /var/run/cloudera-scm-server/cmf3411206514354101952.keytab
chmod: cannot access ‘/var/run/cloudera-scm-server/cmf3411206514354101952.keytab’: No such file or directory
>> Grateful for any help.
... View more
10-14-2018
11:04 AM
1 Kudo
I could solve the problem myself. As written in the documentation you have to have a block device for Docker. This can be either a second hard drive or a file which works as a block device. Since I have only one hard drive on every CDSW-Host I had to create a block device as a file. Following steps might be helpful for someone: Creating a file (300GB) out of our aivalable block device (hard drive) under /home/myBlockDevice : dd if=/dev/sda of=/home/myBlockDevice bs=1M count=300000 Format file with ext4: mkfs.ext4 /home/myBlockDevice Bind file with loop device loop0: losetup /dev/loop0 /home/myBlockDevice Do the following if you wish to bind your newely created block-device after a reboot of your host with the loop device: nano /etc/rc.local Paste: losetup /dev/loop0 /home/myBlockDevice Safe file and close Run following comment on shell to ensure that the script will be executed during boot: chmod +x /etc/rc.d/rc.local In this example the path of DOCKER_BLOCK_DEVICES during the installation of CDSW will be /dev/loop0
... View more
10-10-2018
07:57 AM
Hello guys,
I have a problem by adding CDS 1.4 as service on my CDH 5.14. The error-message is:
ERROR:: Entries in DOCKER_BLOCK_DEVICES must only be block devices: [/dev/xvde]: 1
ERROR:: Unable to create storage for docker.: 1
What I did so far or tried out:
Added two new Hosts to my Cluster (Master and Worder for CDSW)
Installed CDSW successfully with Cloudera Manager
Formatted and mounted one block device to /var/lib/cdsw for the Application Block Device (CDSW Documentation Page 45)
At this point I am not sure what to paste in during the installation for DOCKER_BLOCK_DEVICES I tried here severl paths without success. As I understood I can choose myself here a path for instance /dev/xvde (Example can be found on CDSW documentation page 48) and the installer will do the work (format and mount Docker on each gateway host)
Full-Log from Cloudera Manager after the installation aborts:
Mi 10. Okt 11:26:24 CEST 2018
Current working directory: /run/cloudera-scm-agent/process/2762-cdsw-CDSW_DOCKER-prepare_node
Launching one-off process: /usr/lib64/cmf/service/csd/csd.sh prepare_node true
Mi 10. Okt 11:26:24 CEST 2018
JAVA_HOME=/usr/java/jdk1.8.0_181
Using /run/cloudera-scm-agent/process/2762-cdsw-CDSW_DOCKER-prepare_node as conf dir
Using scripts/control.sh as process script
CONF_DIR=/run/cloudera-scm-agent/process/2762-cdsw-CDSW_DOCKER-prepare_node
CMF_CONF_DIR=/etc/cloudera-scm-agent
Checking if [nfs-utils.x86_64] is installed...OK
Checking if [redhat-lsb-core.x86_64] is installed...OK
Checking if [nmap-ncat.x86_64] is installed...OK
Checking if [selinux-policy-targeted.noarch] is installed...OK
Checking if [libtool-ltdl.x86_64] is installed...OK
Checking if [libcgroup.x86_64] is installed...OK
Checking if [libseccomp.x86_64] is installed...OK
Checking if [lvm2.x86_64] is installed...OK
Checking if [bridge-utils.x86_64] is installed...OK
Checking if [policycoreutils-python.x86_64] is installed...OK
Checking if [ntp.x86_64] is installed...OK
Checking if [ebtables.x86_64] is installed...OK
Checking if [bind-utils.x86_64] is installed...OK
Checking if [openssl.x86_64] is installed...OK
Checking if [e2fsprogs.x86_64] is installed...OK
Checking if [rsync.x86_64] is installed...OK
Checking if [selinux-policy.noarch] is installed...OK
Checking if [gawk.x86_64] is installed...OK
Checking if [sed.x86_64] is installed...OK
Checking if [coreutils.x86_64] is installed...OK
Checking if [util-linux.x86_64] is installed...OK
Checking if [ncurses.x86_64] is installed...OK
Checking if [socat.x86_64] is installed...OK
Setting up docker storage.
ERROR:: Entries in DOCKER_BLOCK_DEVICES must only be block devices: [/dev/xvde]: 1
ERROR:: Unable to create storage for docker.: 1
Exit code: 1
Grateful for any help!
... View more
Labels:
10-08-2018
06:15 AM
My fault. I forgot to add the Custom Service Descriptor (CSD) to /opt/cloudera/csd/. After restarting scm with: service cloudera-scm-server restart I was able to find StreamSets on the AddService-Page.
... View more
10-08-2018
04:43 AM
Hello @aarman, could you give me please further assistance regarding the config-files you mentioned? I have a similar issue like @SAKTIPADA. I installed and distributed the StreamSets Data Collector version 3.5.0 Parcel with Cloudera Manager on our CDH 5.14.1 with success but I am not able to add the services with CM because there is no StreamSets option. Thanks.
... View more
07-18-2018
12:26 AM
After opening ports 1004 and 1006 we are now able to write data into our Cluster. Many thanks to weichiu, without his hint regarding enabling debug log for class org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology I would not be able to see the problem.
... View more
- « Previous
-
- 1
- 2
- Next »