As a general statement this is not right by any means. LDAP provides secure and encrypted authentication (encrypted user password and SSL/TLS communication) , together with user/group management. It's only the Hadoop stack does not support this and the two only autentication methods implemented for all the CDP components are the dummy simple auth (described above) and the Kerberos authentication (used in combination with PAM or LDAP for user/group mappings). As an example, nothing less than Knox (the security gateway to HDP or CDP) implements full authenticacion using only LDAP (with TLS), and it only relies on Kerberos to authenticate a single service/proxy user to communicate with the rest of the cluster. ... View more

Frustrating that that link is hidden behind a 'paywall'.  I have an account but I am not allowed to view without contacting sales ... View more

How did you resolve this issue? ... View more

Facing issues with region availability and it seems to be due to compactions. We are getting below exception when we try to access region org.apache.hadoop.hbase.NotServingRegionException: Region is not online But when we checked corresponding region server logs we can see lot of compactions happening on the table. Does table becomes unaccessible during compaction? Is there a way to reduce number of compactions through some setting? ... View more

Please check below command, here 2> /dev/null will consume all the logs and error. It will now allow standard output to be shown:  beeline -u jdbc:hive2://somehost_ip/ -f 2> /dev/null hive.hql >op.txt if you like this please give me kudos. Thanks!!!  ... View more

hive -e 'select col1,col2 from schema.your_table_name' --hiveconf tez.queue.name=YOUR_QUEUE_NAME > /yourdir/subdir/my_sample_output.csv ... View more

Hello, I'm looking your answer 3 years later because I'm in a similar situation :). In my company (telco) we're planning using 2 hot clusters with dual ingest because our RTO is demanding and we're looking for mechanisms to monitor and keep in sync both clusters. We ingest data in real-time with kafka + spark streaming, loading to HDFS and consuming with Hive/Impala. I'm thinking about a first approach making simple counts with Hive/Impala tables on both clusters each hour/half hour and comparing. If something is missing in one of the clusters, we will have to "manually" re-ingest the missing data (or copy it with cloudera BDR from one cluster to the other) and re-process enriched data. I'm wondering if have you dealt with similar scenarios or suggestions you may have. Thanks in advance! ... View more

Hi @Jeff Watson. You are correct about SAS use of String datatypes. Good catch! One of my customers also had to deal with this. String datatype conversions can perform very poorly in SAS. With SAS/ACCESS to Hadoop you can set the libname option DBMAX_TEXT (added with SAS 9.4m1 release) to globally restrict the character length of all columns read into SAS. However for restricting column size SAS does specifically recommends using the VARCHAR datatype in Hive whenever possible. http://support.sas.com/documentation/cdl/en/acreldb/67473/HTML/default/viewer.htm#n1aqglg4ftdj04n1eyvh2l3367ql.htm Use Case Large Table, All Columns of Type String: Table A stored in Hive has 40 columns, all of type String, with 500M rows. By default, SAS Access converts String to $32K. So, 32K in length for char. The math for this size table yields 1.2MB row length x 500M rows. This causes the system to come to a halt - Too large to store in LASR or WORK. The following techniques can be used to work around the challenge in SAS, and they all work: Use char and varchar in Hive instead of String. Set the libname option DBMAX_TEXT to globally restrict the character length of all columns read in In Hive do "SET TBLPROPERTIES SASFMT" to add formats for SAS on schema in HIVE. Add formatting to SAS code during inbound reads example: Sequence Length 8 Informat 10. format 10. I hope this helps. ... View more

I've seen it in as far back as 2.1 that's why I was surprised it was missing in my install. ... View more

Very neatly explained.! ... View more

I had the same problem and I also solve it adding the user to all controler nodes. Run a script for add them from on_to_all ______________________________________________________________________________________________ #!/bin/bash # Linux/UNIX box with ssh key based login SERVERS=/root/hadoop_hosts # SSH User name USR="root" # Email SUBJECT="Server user login report" EMAIL="your_e-mail@here" EMAILMESSAGE="/tmp/sshpool_`date +%Y%m%d-%H:%M`.txt" # create new file >$EMAILMESSAGE # connect each host and pull up user listing for host in `cat $SERVERS` do echo "--------------------------------" >>$EMAILMESSAGE echo "* HOST: $host " >>$EMAILMESSAGE echo "--------------------------------" >>$EMAILMESSAGE ###ssh $USR@$host w >> $EMAILMESSAGE ssh -tq -o "BatchMode yes" $USR@$host $1 >> $EMAILMESSAGE done # send an email using /bin/mail ######/bin/mailx -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGE echo ">>>>>>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<<<<<<<<<<" echo ">>> check the output file " $EMAILMESSAGE _________________________________________________________________________________________ put DNS servers names into /root/hadoop_hosts Also in linux there is a good command called pssh to run comands in parallel in computer clusters 🙂 ... View more

normally mapper dont fail with OOM and 8192M is pretty good, I suspect that if you have some big records while reading from csv, are you doing some memory intensive operation inside mapper. could you please share the task log for this attempt attempt_1466342436828_0001_m_000008_2 ... View more

Good you fixed it. I would just read a good hadoop book and understand the MapCombinerShuffleReduce process in detail. After that the majority of markers should be pretty self evident. https://www.amazon.com/Hadoop-Definitive-Guide-Tom-White/dp/1449311520/184-6666119-1311365?ie=UTF8&*Version*=1&*entries*=0 ... View more

Rajkumar, Have you tried connecting directly with the hive jdbc driver? I'm suspecting it's a jar conflict somewhere. Here's my hive driver config in IntelliJ, obviously took the shotgun approach and added all client jar but the main required are hive-common, hive-jdbc. ... View more

yes you would need to configure user sync with ldap/ad in the ranger ui. Alternatively use UNIX user sync in Ranger to sync with the local operating system. ( Works as well ) ... View more

Can you give me top 50 min, max and the average. Also did you try the query ? What was the behaviour ? The reason I am asking that if your query is very long using a few number of reducer for example it may imply the skew and so to maximize usage of the cluster one way is too look at surrogate key creation. ... View more

Here is a great writeup on file compression in Hadoop - http://comphadoop.weebly.com/ ... View more

Hi Ed, It would be useful to know if you are aiming for HA or performance. Since it is a small cluster you may use it as a POC and not care much about HA, I don't know. One option not mentioned below is going with 3 masters and 3 slaves in a small HA cluster setup. That allows you to balance services on the masters more and/or dedicate one to be mostly an edge node. If security is a topic that may come in handy. Cheers, Christian ... View more

1) You essentially have two options. Use Sqoop import-all-tables with exclude as you mention. However in that case you have a single sqoop action in oozie and no parallelity in oozie. However sqoop might provide that. You have some limitations though ( only straight imports all columns , ... ) Alternatively you make an oozie flow that uses a fork and then one single table sqoop action per table. In that case you have fine grained control over how much you want to run in parallel. ( You could for example load 4 at a time by doing Start -> Fork -> 4 Sqoop Actions -> Join -> Fork -> 4 Sqoop Actions -> Join -> End 2) If you want incremental load I don't think the Sqoop import-all-tables is possible. So one Sqoop action per table it is. Essentially you can either use Sqoop incremental import functionality ( using a property file ) or use WHERE conditions and give through the date parameter from the coordinator. You can use coord:dateformat to transform your execution date. 3) Run One coord for each table OR have a Decision action in the oozie workflow that skips some sqoop actions Like Start -> Sqoop1 where date = mydate -> Decision if mydate % 3 = 0 then Sqoop2 else end. 4) incremental imports load the new data into a folder in HDFS. If you run it the folder needs to be deleted. If you use append it doesn't delete the old data in HDFS. Now you may ask why would I ever not want append and the reason is that you mostly do something with the data after like importing the new data to a hive partitioned table. If you would use append he would load the same data over and over. ... View more

@Joseph Niemiec How can I do this command " select * from table where date <= '2017-12-08' " in nest partitions form? In case the table is partitioned by day,month,year ... View more