About Joey_Krabacher

Joey_Krabacher · ‎07-06-2020

After testing this, we are now seeing these messages for any type of Spark/Scala session in CDSW Spark history logs show java.io.FileNotFoundException: /tmp/spark-driver.log (Permission denied) Removed the overrides for spark-env.sh and now those failing sessions are working again. More testing needed.

Joey_Krabacher · ‎07-04-2020

HADOOP_CONF_DIR=$SPARK_CONF_DIR/yarn-conf/* HIVE_CONF_DIR=${HIVE_CONF_DIR:-/etc/hive/conf} if [ -d "$HIVE_CONF_DIR" ]; then HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HIVE_CONF_DIR" fi I'm currently testing the above setting. It is essentially the same as what was already in the original spark-env.sh , I just modified the first line to not use the :- HADOOP_CONF_DIR default . This approach is an alternative to using static values.

Joey_Krabacher · ‎04-22-2020

Good article and is still relevant for me. I would also add the following for Kudu tables create table hive.TABLE_PARAMS_BKUP as select * from hive.TABLE_PARAMS ; UPDATE hive.TABLE_PARAMS SET PARAM_VALUE = 'masterhostname01.fqdn,masterhostname02.fqdn,masterhostname03.fqdn' WHERE PARAM_KEY = 'kudu.master_addresses' and PARAM_VALUE like '%oldmasterhostname%'; And also for Sentry URI's that contain the HDFS namespace create table sentry.SENTRY_DB_PRIVILEGE_BKUP as select * from sentry.SENTRY_DB_PRIVILEGE ; UPDATE sentry.SENTRY_DB_PRIVILEGE SET URI = REPLACE(URI, 'hdfs://oldcluster-ns', 'hdfs://newcluster-ns/') WHERE URI like '%oldcluster-ns%';

Joey_Krabacher · ‎03-12-2020

old question, but it's gotten a lot of views. the solution is to create the cmjobuser at the OS level - all nodes or at least the NodeManager nodes. If you create the user with an id below 1000, you have to also add the cmjobuser user to the NodeManager allowed.system.users config. Ansible playbook example: --- - name: Create missing YARN user hosts: all,!ilo gather_facts: True tasks: - name: Create user 'cmjobuser' user: name: cmjobuser groups: - hadoop home: /var/lib/cmjobuser shell: /sbin/nologin system: yes

Joey_Krabacher · ‎10-18-2018

For SIEM purposes, it would be greatly beneficial to capture the originating _users_ IP address instead of the IP address of the host, which is not very helpful. I understand that this information is further back in the request and is not available at the web browser level. It should be fairly easy to modify the web application's server config to include this information in the event stream. For instance, when running queries through Impala or Hive while inside of Hue, the Hue service IP is recorded instead of the IP of the user that is logged into Hue.

Joey_Krabacher · ‎07-11-2018

Thank you for the response. Any idea on roadmap dates or targeted quarters?

Joey_Krabacher · ‎06-29-2018

Considering that Hive is more mature than Impala and many of your customers(including me) are most likely considering switching from Hive to Impala, wouldn't it make sense for you to support this interoperability? We have over 7000 tables in our data lake, so just coming back with "well you'll just have to convert those tables to use strings or timestamps instead" is not solution that works for us. Your sales force and product managers are pushing Impala as a solution to Fast BI and Analytics needs. With this hurdle in place, it's pretty much a non-starter. Folks will just continue to pump their data into a traditional database for BI and Analytics. With 3.0 just released, how in the world do you not support the DATE data type?

Joey_Krabacher · ‎04-10-2018

I now have CDSW up and running. I'm not sure which one of these did the trick or if there was some other force at play. We found a bug in ip6tables.service (RHEL 7.4) that was producing error messages like this: Apr 10 10:06:56 [redacted] systemd[1]: [/usr/lib/systemd/system/ip6tables.service:3] Failed to add dependency on syslog.target,iptables.service, ignoring: Invalid argument so we changed the After parameter from comma delimited to space delimited. before change: After=syslog.target,iptables.service after change: After=syslog.target iptables.service Bug link: https://bugzilla.redhat.com/show_bug.cgi?id=1499367 Here are the commands that were run edit /usr/lib/systemd/system/ip6tables.service systemctl stop iptables systemctl disable iptables systemctl stop ip6tables systemctl disable ip6tables /usr/bin/cdsw reset /usr/bin/cdsw init

Joey_Krabacher · ‎04-09-2018

All, I'm still facing the same issue. If any of you have the kube-dns pod running with all 3 containers running successfully (kubedns,dnsmasq and sidecar), can you run the following and reply back with the output...it would be greatly appreciated. Get the pod names from the output of this command kubectl get pods --all-namespaces then get the CLUSTER-IP from this command kubectl get services --sort-by=.metadata.name then execute nslookup commands on the running pods e.g. kubectl exec <kube-dns-pod-name> -c sidecar --namespace=kube-system -- nslookup <CLUSTER-IP> kubectl exec <kube-dns-pod-name> -c dnsmasq --namespace=kube-system -- nslookup <CLUSTER-IP> kubectl exec <kube-dns-pod-name> -c kubedns --namespace=kube-system -- nslookup <CLUSTER-IP> e.g. kubectl exec kube-dns-3911048160-lhtvm -c kubedns --namespace=kube-system -- nslookup 100.77.0.1 I may be barking up the wrong tree, but I'm trying to figure out why my containers timeout when trying to connect to https:// 100 . 77 . 0 . 1 : 443 Also, if you could post a copy of your /etc/cdsw/config/cdsw.conf (with sensitive information redacted or masked) that would be great.

Joey_Krabacher · ‎03-29-2018

kubedns and dnsmasq both appear to be failing sudo /usr/bin/cdsw init ... Waiting for kube-system cluster to come up. This could take a few minutes... ERROR:: Unable to bring up kube-system cluster.: 1 ERROR:: Unable to start kubernetes system pods.: 1 ... $ sudo kubectl --namespace=kube-system get pods NAME READY STATUS RESTARTS AGE etcd-udodapp05 1/1 Running 0 16m kube-apiserver-udodapp05 1/1 Running 0 16m kube-controller-manager-udodapp05 1/1 Running 0 16m kube-dns-3911048160-99klb 2/3 CrashLoopBackOff 13 15m kube-proxy-02z9b 1/1 Running 0 15m kube-scheduler-udodapp05 1/1 Running 0 15m weave-net-4fzw6 2/2 Running 0 15m $ cat cdsw.conf JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera/ MASTER_IP=[redacted] DOMAIN=[redacted] DOCKER_BLOCK_DEVICES=/dev/mapper/imgvg-imglv APPLICATION_BLOCK_DEVICE=/dev/mapper/appvg-applv NO_PROXY="127.0.0.1,localhost,[redacted],100.66.0.1,100.66.0.2,100.66.0.3,100.66.0.4,100.66.0.5,100.66.0.6,100.66.0.7,100.66.0.8,100.66.0.9,100.66.0.10,100.66.0.11,100.66.0.12,100.66.0.13,100.66.0.14,100.66.0.15,100.66.0.16,100.66.0.17,100.66.0.18,100.66.0.19,100.66.0.20,100.66.0.21,100.66.0.22,100.66.0.23,100.66.0.24,100.66.0.25,100.66.0.26,100.66.0.27,100.66.0.28,100.66.0.29,100.66.0.30,100.66.0.31,100.66.0.32,100.66.0.33,100.66.0.34,100.66.0.35,100.66.0.36,100.66.0.37,100.66.0.38,100.66.0.39,100.66.0.40,100.66.0.41,100.66.0.42,100.66.0.43,100.66.0.44,100.66.0.45,100.66.0.46,100.66.0.47,100.66.0.48,100.66.0.49,100.66.0.50,100.77.0.129,100.77.0.130,100.77.0.1,100.77.0.10" $ sudo kubectl logs -f --since=1h po/kube-dns-3911048160-99klb dnsmasq --namespace=kube-system I0320 22:03:25.264188 1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000} I0320 22:03:25.265432 1 nanny.go:86] Starting dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] I0320 22:03:25.298956 1 nanny.go:111] I0320 22:03:25.298956 1 nanny.go:108] dnsmasq[25]: started, version 2.78-security-prerelease cachesize 1000 W0320 22:03:25.299025 1 nanny.go:112] Got EOF from stdout I0320 22:03:25.299031 1 nanny.go:108] dnsmasq[25]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify I0320 22:03:25.299044 1 nanny.go:108] dnsmasq[25]: using nameserver 127.0.0.1#10053 for domain ip6.arpa I0320 22:03:25.299052 1 nanny.go:108] dnsmasq[25]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa I0320 22:03:25.299055 1 nanny.go:108] dnsmasq[25]: using nameserver 127.0.0.1#10053 for domain cluster.local I0320 22:03:25.299065 1 nanny.go:108] dnsmasq[25]: reading /etc/resolv.conf I0320 22:03:25.299068 1 nanny.go:108] dnsmasq[25]: using nameserver 127.0.0.1#10053 for domain ip6.arpa I0320 22:03:25.299072 1 nanny.go:108] dnsmasq[25]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa I0320 22:03:25.299076 1 nanny.go:108] dnsmasq[25]: using nameserver 127.0.0.1#10053 for domain cluster.local I0320 22:03:25.299079 1 nanny.go:108] dnsmasq[25]: using nameserver [redacted]#53 I0320 22:03:25.299082 1 nanny.go:108] dnsmasq[25]: using nameserver [redacted]#53 I0320 22:03:25.299085 1 nanny.go:108] dnsmasq[25]: using nameserver [redacted]#53 I0320 22:03:25.299089 1 nanny.go:108] dnsmasq[25]: using nameserver [redacted]#53 I0320 22:03:25.299092 1 nanny.go:108] dnsmasq[25]: read /etc/hosts - 7 addresses $ sudo kubectl logs -f --since=1h po/kube-dns-3911048160-99klb kubedns --namespace=kube-system I0320 21:58:22.617903 1 dns.go:48] version: 1.14.4-2-g5584e04 I0320 21:58:22.619053 1 server.go:70] Using configuration read from directory: /kube-dns-config with period 10s I0320 21:58:22.619096 1 server.go:113] FLAG: --alsologtostderr="false" I0320 21:58:22.619108 1 server.go:113] FLAG: --config-dir="/kube-dns-config" I0320 21:58:22.619114 1 server.go:113] FLAG: --config-map="" I0320 21:58:22.619118 1 server.go:113] FLAG: --config-map-namespace="kube-system" I0320 21:58:22.619121 1 server.go:113] FLAG: --config-period="10s" I0320 21:58:22.619129 1 server.go:113] FLAG: --dns-bind-address="0.0.0.0" I0320 21:58:22.619132 1 server.go:113] FLAG: --dns-port="10053" I0320 21:58:22.619137 1 server.go:113] FLAG: --domain="cluster.local." I0320 21:58:22.619142 1 server.go:113] FLAG: --federations="" I0320 21:58:22.619148 1 server.go:113] FLAG: --healthz-port="8081" I0320 21:58:22.619151 1 server.go:113] FLAG: --initial-sync-timeout="1m0s" I0320 21:58:22.619155 1 server.go:113] FLAG: --kube-master-url="" I0320 21:58:22.619162 1 server.go:113] FLAG: --kubecfg-file="" I0320 21:58:22.619165 1 server.go:113] FLAG: --log-backtrace-at=":0" I0320 21:58:22.619171 1 server.go:113] FLAG: --log-dir="" I0320 21:58:22.619175 1 server.go:113] FLAG: --log-flush-frequency="5s" I0320 21:58:22.619180 1 server.go:113] FLAG: --logtostderr="true" I0320 21:58:22.619183 1 server.go:113] FLAG: --nameservers="" I0320 21:58:22.619186 1 server.go:113] FLAG: --stderrthreshold="2" I0320 21:58:22.619189 1 server.go:113] FLAG: --v="2" I0320 21:58:22.619192 1 server.go:113] FLAG: --version="false" I0320 21:58:22.619202 1 server.go:113] FLAG: --vmodule="" I0320 21:58:22.619292 1 server.go:176] Starting SkyDNS server (0.0.0.0:10053) I0320 21:58:22.619587 1 server.go:198] Skydns metrics enabled (/metrics:10055) I0320 21:58:22.619599 1 dns.go:147] Starting endpointsController I0320 21:58:22.619603 1 dns.go:150] Starting serviceController I0320 21:58:22.619713 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0] I0320 21:58:22.619737 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0] I0320 21:58:23.119838 1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver... I0320 21:58:23.619844 1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver... E0320 21:58:23.623059 1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://100.77.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 100.77.0.1:443: getsockopt: connection refused E0320 21:58:23.623077 1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://100.77.0.1:443/api/v1/services?resourceVersion=0: dial tcp 100.77.0.1:443: getsockopt: connection refused I0320 21:58:24.119875 1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver... I0320 21:58:24.619805 1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver... I0320 21:58:25.119883 1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver... I0320 21:58:25.619870 1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver.. .............. I0320 21:59:22.119836 1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver... F0320 21:59:22.619832 1 dns.go:168] Timeout waiting for initialization

Joey_Krabacher · ‎02-23-2018

echo "use navigator;" >> alter_audit_tables.sql mysql -h [hostname] -u root -p -e "use navigator;SELECT DISTINCT TABLE_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE COLUMN_NAME IN ('OPERATION_TEXT') AND TABLE_SCHEMA='navigator';" | grep 'AUDIT_EVENTS' | awk '{print "ALTER TABLE " $1 " modify column OPERATION_TEXT longtext CHARACTER SET utf8;"}' >> alter_audit_tables.sql cat alter_audit_tables.sql | mysql -h [hostname] -u root -p echo "commit;" >> alter_audit_tables.sql This fixed the issue for me and subsequently got rid of the Audit Pipeline test failures

Joey_Krabacher · ‎08-24-2016

Thank you for the quick response Ben. We will be upgrading soon, so it is good to know that this will be fixed then. For now, I will just keep using native mysql commands for syncing. Thanks, /* Joey */

Joey_Krabacher · ‎08-24-2016

Could not find anything related to this or others that are experiencing the same issue. java.lang.IllegalArgumentException: Pathname /user/hdfs/.cm/hive/2016-08-24-16-52-30+00:00-8590 from /user/hdfs/.cm/hive/2016-08-24-16-52-30+00:00-8590 is not a valid DFS filename. I am using CDH 5.4.7. I can't find a way to change the settings on how this task is generating the log name to remote the "+" from the name. For now I am just going to use mysql function to backup and restore the Hive Metastore database on another cluster, to sync them. Thanks, /* Joey */

Joey_Krabacher · ‎07-20-2016

We have thousands of tables and hundreds of partitions for each of those tables. After enabling Sentry and HDFS ACL sync, it now take 2 full hours before Hive is usable again after restarting. Metastore logs are filled with log entries that look like this: Jul 20, 3:00:02.313 PM INFO org.apache.hadoop.hive.metastore.HiveMetaStore 9: get_partitions_by_names : db=**** tbl=***** Jul 20, 3:00:02.313 PM INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit ugi=hive ip=unknown-ip-addr cmd=get_partitions_by_names : db=**** tbl=**** It looks like it is calling the get_partitions_by_names 10 times per table. Is there any way to get the HMS to load faster? /* Joey */

Joey_Krabacher · ‎08-21-2015

If anyone is curious about a solution to this. Unless you have two or more racks that have the same number of nodes, the solution is to not use a rack topology script or bypass it by assigning all nodes to one rack. The only downside is that you run the risk of having missing blocks/data if one of your racks were to get smashed by a truck, vaporized, beamed into outer-space, melts, <insert catastrophic event that only affects one rack here>, etc. 2 racks uneven, rack topology should be 1 rack 2 racks even, rack topology should be 2 racks 2 racks even, 1 rack not eve, rack topology should be 2 racks with the 3rd rack split between rack 1 and rack 2. etc

Joey_Krabacher · ‎08-20-2015

Let's say you have two racks, one with 18 nodes and one with 6 nodes. If most of your work is done from a specific node on rack one, then 2/3rds of the blocks in your cluster would go to the 2nd rack. What is the best solution for keeping these two racks balanced? I think one option is to change the rack topology to think you only have 1 rack. You would lose the certainty that at least 1 block would make it to the 2nd rack though.

Joey_Krabacher · ‎08-14-2014

Directory does not exist but can be deleted 1. Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://host01-ns/etl/ABC/XYZ/2014-08-13/mytestdirectory already exists 2. hadoop fs -ls /etl/ABC/XYZ/2014-08-13/mytestdirectory ( nothing returned) 3. hadoop fs -test -e /etl/ABC/XYZ/2014-08-13/mytestdirectory ( nothing returned) 4. hadoop fs -rm -r /etl/ABC/XYZ/2014-08-13/mytestdirectory 14/08/14 23:07:49 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 1440 minutes, Emptier interval = 0 minutes. Moved: 'hdfs://host01-ns/etl/ABC/XYZ/2014-08-13/mytestdirectory' to trash at: hdfs://host01-ns/user/testuser/.Trash/Current ( how did it delete a file that did not exist...supposedly ) This is causing all of our scripts that check for existing files to fail. #file 1. hadoop fs -test -e /etl/ABC/XYZ/2014-08-13/mytestfile.BIN ( nothing returned ) 2. hadoop fs -ls /etl/ABC/XYZ/2014-08-13/mytestfile.BIN Found 1 items -rw-r--r-- 3 testuser supergroup 2087844001 2014-08-13 23:52 /etl/ABC/XYZ/2014-08-13/mytestfile.BIN

Joey_Krabacher · ‎08-14-2014

From what I have seen these are IPC ephemeral ports. Is there anyway to control the range on these?

Joey_Krabacher · ‎08-12-2014

org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:136) I have tried setting yarn.app.mapreduce.am.job.client.port-range to a specific range, but it doesn't look like that is the right setting. I have seen it use ports ranging from 32000 - 65000 CDH 5.1

Online	Offline
Last Visited	‎07-06-2020 02:40 PM

Date Registered	‎08-12-2014 06:34 AM
Last Visited	‎07-06-2020 02:40 PM
Total Messages Posted	19
Total Kudos Received	3

Re: cdsw init fails with kube-dns issues

Re: "RuntimeException: core-site.xml not foun...

Re: "RuntimeException: core-site.xml not foun...

Re: Migrate Hive from HDP 2.4.3 to HDP-2.5.3

Re: 'User cmjobuser not found' when trying to aggr...

Navigator Audit Events > IP Address

Re: Why Won't Impala Support The Date Data Type?

Why Won't Impala Support The Date Data Type?

Re: cdsw init fails with kube-dns issues

Re: cdsw init fails with kube-dns issues

cdsw init fails with kube-dns issues

Re: Error persisting events for service hive

Re: Hive Replication Failing with Invalid DFS File...

Hive Replication Failing with Invalid DFS Filename

Hive Metastore Takes Two Hours to Start

Re: How to balance block placement on uneven racks...

How to balance block placement on uneven racks?

Weird HDFS Shell Issues When Testing For Existing ...

Re: Where is the setting for the port-range used b...

Where is the setting for the port-range used by or...