Mandatory Pre-requisite Upgrade JDK from 1.7 to 1.8 If the step is missed it will result in to below errors. Nodemanager will fail to start if spark shuffle service is enabled. "java.lang.UnsupportedClassVersionError: org/apache/spark/network/yarn/YarnShuffleService : Unsupported major.minor version 52.0" Hive Metastore will fail to start. "Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/hadoop/hive/ql/log/NullAppender : Unsupported major.minor version 52.0" Service checks will fail. Yarn Hive Finalize upgrade will fail and cluster have to be downgraded. ... View more

Hive Compare Tables and Databases before and after Upgrades. Run below script to collect all databases. hive -e "show databases" | sed "s/ *$//g" | sed "s/^ *//g" | sort 1>databaselist Run below script to collect all tables in database. for dbname in `cat databaselist` ; do hive -e "use ${dbname} ; show tables ;" | sed "s/ *$//g" | sed "s/^ *//g" | sort 1> tablelist; done ; Execute Scripts above before upgrade and once after upgrade . Simply do vimdiff afterwards to identify the difference if any. Happy Migrating or Upgrading. ... View more

[users] # To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections admin = admin [main] ldapRealm = org.apache.zeppelin.server.LdapGroupRealm ldapRealm.contextFactory.environment[ldap.searchBase] = dc=mgmt,dc=example,dc=net ldapRealm.contextFactory.url = ldaps://me.abc.example.net:636 ldapRealm.userDnTemplate = uid={0},cn=group,dc=mgmt,dc=example,dc=net ldapRealm.contextFactory.authenticationMechanism = SIMPLE ldapRealm.contextFactory.systemUsername=uid=example,ou=example,dc=mgmt,dc=example,dc=net ldapRealm.contextFactory.systemPassword=example sessionManager=org.apache.shiro.web.session.mgt.DefaultWebSessionManager securityManager.sessionManager = $sessionManager securityManager.sessionManager.globalSessionTimeout = 86400000 shiro.loginUrl = /api/login [urls] /** = authc ... View more

Logical Disk Encryption The approach is to avoid using HDFS encryption and use Disk LUKS encryption for data at rest encryption requirement specially when using public cloud IAAS. To build manually encrypted volumes or drives ,use following steps on a d2*8xLarge instance flavor. lsblk cryptsetup --verbose --verify-passphrase luksFormat /dev/xvdb cryptsetup --verbose --verify-passphrase luksFormat /dev/xvdc cryptsetup --verbose --verify-passphrase luksFormat /dev/xvdd cryptsetup luksOpen /dev/xvdb vol1 cryptsetup luksOpen /dev/xvdc vol2 cryptsetup luksOpen /dev/xvdd vol3 dd if=/dev/urandom of=/root/keyfile1 bs=1024 count=4 chmod 0400 /root/keyfile1 cryptsetup luksAddKey /dev/xvdb /root/keyfile1 cryptsetup luksAddKey /dev/xvdc /root/keyfile1 cryptsetup luksAddKey /dev/xvdd /root/keyfile1 mkfs.ext4 /dev/mapper/vol1 mkfs.ext4 /dev/mapper/vol2 mkfs.ext4 /dev/mapper/vol3 echo "/dev/mapper/vol1 /data/vol1 ext4 defaults,nofail,nodev 0 2" >> /etc/fstab echo "/dev/mapper/vol2 /data/vol2 ext4 defaults,nofail,nodev 0 2" >> /etc/fstab echo "/dev/mapper/vol3 /data/vol3 defaults,nofail,nodev 0 2" >> /etc/fstab echo "vol1 /dev/xvdb /root/keyfile1 luks" >>/etc/crypttab echo "vol1 /dev/xvdc /root/keyfile1 luks" >>/etc/crypttab echo "vol1 /dev/xvdd /root/keyfile1 luks" >>/etc/crypttab mount -a Automated Shell Script #!/bin/bash set -x set: ${PLATFORM_DISK_PREFIX:? required}: ${START_LABEL:? required} format_disks_encrypted() { mkdir /hadoopfs openssl rand -base64 32 > /root/encrypt cat /root/encrypt > /root/encrypt1 cat /root/encrypt1 > /root/encrypt2 yum -y install cryptsetup-luks for (( i=1; i<=24;i++ )); do LABEL=$(printf "\x$(printf %x $((START_LABEL+i)))") DEVICE=/dev/${PLATFORM_DISK_PREFIX}${LABEL} if [ -e $DEVICE ]; then MOUNTPOINT=$(grep $DEVICE /etc/fstab | tr -s ' \t' ' ' | cut -d' ' -f 2) if [ -n "$MOUNTPOINT" ]; then umount "$MOUNTPOINT" sed -i "\|^$DEVICE|d" /etc/fstab fi mkdir /hadoopfs/fs${i} cryptsetup --verbose luksFormat $DEVICE -yrq --key-file=/root/encrypt cryptsetup luksOpen $DEVICE vol${i} --key-file=/root/encrypt cryptsetup luksAddKey $DEVICE /root/encrypt --key-file=/root/encrypt mkfs.ext4 /dev/mapper/vol${i} echo UUID=$(blkid -o value /dev/mapper/vol${i} | head -1) /hadoopfs/fs${i} ext4 inode_readahead_blks=128,data=writeback,noatime,nodiratime 0 2 >> /etc/fstab echo "vol${i} `UUID=$(blkid -o value /dev/mapper/vol${i} | head -1)` /root/encrypt luks" >> /etc/crypttab mount /hadoopfs/fs${i} chmod 777 /hadoopfs/fs${i} fi done } main() { format_disks_encrypted } Reference: https://wiki.archlinux.org/index.php/Dm-crypt/Device_encryption ... View more

Enabling SMTP in Cloudbreak --------------------------- 1. The Profile file 2. Bug in mailer.js and a workaround     2.1 The Problem     2.2 The Cause     2.3 Details     2.4 A Workaround 3. Fix postfix config --- 1. The Profile file ------------------- In the Profile file, set the following CLOUDBREAK_SMTP_* variables and set CBD_FORCE_START to enable starting Cloudbreak containers with a modified file docker-composer.yml (see section 2.4):   cloudbreak $ cd $CBD_ROOT   cloudbreak $ more Profile   export PUBLIC_IP=example.compute.amazonaws.com   export AWS_SECRET_ACCESS_KEY=***   export AWS_ACCESS_KEY_ID=***   export CBD_FORCE_START=true   export CLOUDBREAK_SMTP_SENDER_HOST="172.17.0.1"   export CLOUDBREAK_SMTP_SENDER_FROM="cloudbreak@compute.amazonaws.com"   export CLOUDBREAK_SMTP_AUTH=false   export CLOUDBREAK_SMTP_STARTTLS_ENABLE=false --- 2. Bug in mailer.js and a workaround ------------------------------------ See   https://github.com/sequenceiq/cloudbreak/issues/1492 --- 2.1 The Problem --------------- Can not receive mail from the cloufd UI, e.g., to reset the password:    http://example.compute.amazonaws.com:3000 I found the cause of the problem and a workaround. --- 2.2 The Cause ------------- When the environment variables     SL_SMTP_SENDER_USERNAME     SL_SMTP_SENDER_PASSWORD are defined in the sultans container -- and they are derived from the environment variables    CLOUDBREAK_SMTP_SENDER_USERNAME    CLOUDBREAK_SMTP_SENDER_PASSWORD on the cloudbreak deployer -- even if they are set to the empty string, the Javascript code in /sultans/mailer.js in he sultans container tries to do authentication with the SMTP server. --- 2.3 Details ----------- 1. Because on the Cloudbreak deployer the env varts are defined     cloudbreak $ cbd env show | egrep SMTP     CLOUDBREAK_SMTP_SENDER_USERNAME   =       CLOUDBREAK_SMTP_SENDER_PASSWORD   =       CLOUDBREAK_SMTP_SENDER_HOST       = 172.17.0.1     CLOUDBREAK_SMTP_SENDER_PORT       = 25     CLOUDBREAK_SMTP_SENDER_FROM       = cloudbreak@compute.amazonaws.com     CLOUDBREAK_SMTP_AUTH              = false     CLOUDBREAK_SMTP_STARTTLS_ENABLE   = false     CLOUDBREAK_SMTP_TYPE              = smtp   the cbd start command will inser in docker-compose.yml    cloudbreak $ egrep -A 10 sultans: /var/lib/cloudbreak-deployment/docker-compose.yml    sultans:     environment:         - SL_CLIENT_ID=sultans         - SL_CLIENT_SECRET=cbsecret2015         - SERVICE_NAME=sultans           #- SERVICE_CHECK_HTTP=/         - SL_PORT=3000         - SL_SMTP_SENDER_HOST=172.17.0.1         - SL_SMTP_SENDER_PORT=25         - SL_SMTP_SENDER_USERNAME=         - SL_SMTP_SENDER_PASSWORD= 2. The above settings in docker-compose.yml will in turn cause the    sultans container to have      SL_SMTP_SENDER_USERNAME      SL_SMTP_SENDER_PASSWORD   Indeed:     bash-4.3# cat /proc/5/environ | sed 's/\0/\n/' | egrep SMTP | sort     SL_SMTP_SENDER_FROM=cloudbreak@compute.amazonaws.com     SL_SMTP_SENDER_HOST=172.17.0.1     SL_SMTP_SENDER_PASSWORD=     SL_SMTP_SENDER_PORT=25     SL_SMTP_SENDER_USERNAME= 3. The code in /sultans/mailer.js will do auth if these are defined, even    if they are the empry string:      SL_SMTP_SENDER_USERNAME      SL_SMTP_SENDER_PASSWORD   Indeed:    bash-4.3# egrep -A10 ^sendSimple   /sultans/mailer.js    sendSimpleEmail = function(to, subject, content) {     var transport = null;     if (process.env.SL_SMTP_SENDER_USERNAME == null && process.env.SL_SMTP_SENDER_PASSWORD == null) {         transport = nodemailer.createTransport(smtpTransport({             host: process.env.SL_SMTP_SENDER_HOST,             port: process.env.SL_SMTP_SENDER_PORT,             secure: false,             tls: {                 rejectUnauthorized: false             }         })); --- 2.4 A Workaround ----------------- Make sure that     SL_SMTP_SENDER_USERNAME     SL_SMTP_SENDER_PASSWORD are not defined in the cbreak_sultans_1 bash container. Steps: 1. Hack the file docker-compose.yml:     cloudbreak $ diff /var/lib/cloudbreak-deployment/docker-compose.yml \                       /var/lib/cloudbreak-deployment/docker-compose.yml.sav     149a150,151     >         - SL_SMTP_SENDER_USERNAME=     >         - SL_SMTP_SENDER_PASSWORD= 2. Restart the containers but not with cbd start, because that will overwrite    docker-compose.yml.sav:      cloudbreak $ cbd kill      cloudbreak $ cd /var/lib/cloudbreak-deployment/      cloudbreak $ ./.deps/bin/docker-compose -p cbreak up -d 3. Check that     SL_SMTP_SENDER_USERNAME     SL_SMTP_SENDER_PASSWORD   are not defined on the container:     cloudbreak $ alias sultans     alias sultans='docker exec -it  cbreak_sultans_1 bash'     cloudbreak $ sultans     bash-4.3# ps                                                                                                                       PID   USER     TIME   COMMAND       1 root       0:00 {start-docker.sh} /bin/bash /sultans/start-docker.sh       5 root       0:03 node main.js     bash-4.3# cat /proc/5/environ | sed 's/\0/\n/' | egrep SMTP                                                                            SL_SMTP_SENDER_FROM=cloudbreak@compute.amazonaws.com     SL_SMTP_SENDER_PORT=25     SL_SMTP_SENDER_HOST=172.17.0.1 --- 3. Fix postfix config --------------------- Change /etc/postfix/main.cf   cloudbreak # egrep "inet_.*=" /etc/postfix/main.cf.orig   #inet_interfaces = all   #inet_interfaces = $myhostname   #inet_interfaces = $myhostname, localhost   inet_interfaces = localhost   inet_protocols = all Set inet_interfaces = all   cloudbreak # diff /etc/postfix/main.cf /etc/postfix/main.cf.orig   113c113   < inet_interfaces = all   ---   > #inet_interfaces = all   116c116   < #inet_interfaces = localhost   ---   > inet_interfaces = localhost Restart   cloudbreak # systemctl stop postfix.service   cloudbreak # systemctl start postfix.service ... View more

Please refer to below forked project for details of benchmark commands ,configuration and results, was able to achieve 3 Million records/sec Read , 1.5 Million records/sec and 5 msec 99.9 Percentile for End-to-end Latency. https://gist.github.com/saurabhmishra/6ed428be4c00003dd926 ... View more