Ambari applies some heuristics. But it can never hurt to double check. Datanode: 1GB works but for normal nodes ( 16 cores/12-14 drives/128-256GB RAM) I normally set it to 4GB Namenode: Depends on HDFS size. A good rule of thumb is that 100TB of data in HDFS need 1GB of RAM. Spark: This totally depends on your spark needs. ( not talking about history server but the defaults for executors ) The more power you need the more executors and more RAM in them ( up to 32GB is good apparently ) Yarn: Ambari does decent heuristics but I like to tune them normally. In the end you should change the sizes until yoiur cluster has a good CPU utilization. But this is something more elaborate and depends on your usecases ... View more

My first question would be why do you want to do that. If you want to manage your cluster you would normally install something like pssh or ansible or puppet and use that to manage the cluster. You can put that on one control node define a list of servers and move data/execute scripts on all of them at the same time. You can do something very simple like that with a one line ssh program To execute a script on all nodes: for i in server1 server2;do echo $i; ssh $i $1;done To copy files to all nodes: for i in server1 server2;do scp $1 $i:$2;done [all need keyless ssh from the control node to the cluster nodes] If on the other hand you want to execute job dependencies, something like the distributed mapreduce cache is normally a good idea. Oozie provides the <file> tag to upload files from hdfs to the execution directory of the job. So honestly if you go into more details what you ACTUALLY want we might be able to help more. ... View more

Hmmm access issue? If its in the root folder only root can access it. Hive error messages can be pretty generic here and not distinguish between access rights and files exists. If you run beeline the read will be executed by the hive server and that one is running under the hive user so only he could access the data. Try it in /tmp if it works there too you would know. ... View more

I think the majority of people do not use ssh fencing at all. The reason for this is that Namenode HA works fine without it. The only issue can be that during a network partitioning old connections to the old standby might still exist and get stale old date during read-only operations. - They cannot do any write transactions since the Journalnode majority prohibits that - Normally if zkfc works correctly an active namenode will not go into zombie mode, he is dead or not. So the chances of a split brain are low and the impact is pretty limited. If you use ssh fencing the important part is that your script cannot block other wise the failover will be stopped, you need to have all scripts return in a sensible amount of time even if the access is not possible. Fencing by definition is always an attempt. Since most of the time the node is simply down. And they need to return success in the end. So you need a fork with a timeout and then return true. https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#Verifying_automatic_failover ... View more

if you have a general location you can only use time based scheduling ( every 15 min for example ) you cannot use retention or do late arrivals etc. All the advanced waiting for data to arrive features in oozie are essentially out. Essentially mimicks the datasets in oozie: https://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a5._Dataset ... View more

You need to look into the logs. Most likely yarn logs of the Map Task of your Oozie launcher. This contains the sqoop command execution and any errors you would normally see on the command line. You can get them from resourcemanager ( click on your oozie launcher job and go through to the map task or use yarn application -logs. You can find any issues in the actual data transfer in the kicked off Mapreduce job which is a separate job ... View more

Pretty sure its a community feature and not supported yet even if it should have found its way into the release. Also its very limited right now. Can you really replace your scheduling with it? It still seems to need a bit of time in the oven. ... View more

It depends a bit what you do with it. Each time you send the data over the network a native datatype will be much faster. 999999999 would be 4bytes as integer but 10 bytes as a string ( characters plus length). That holds for shuffles and any operation that needs to copy data. ( Although some functions like a Map might not be impacted since the Strings will not be necessarily copied in Java if you only assign them for example ) It also costs you more RAM in the Executor which can be a considerable factor in Spark when doing aggregations/joins/sorts etc. And finally when you actually need to do computations on these columns you would have to cast them which will be a pretty costly operation that you could save yourself otherwise. So depending on your usecase the performance and RAM difference can vary between not really significant and considerable. ... View more

I was about to say its not possible. I assume both tables have a column with the same name which is one of the reasons database schemas often prefix column names with a letter from the table. Like TPCH: lineitem.l_key and orders.o_key. In this case you would have to suck it up and name all columns in the join using the AS statement. However it looks like Hive has some cool/dirty tricks up its sleeve. Regular Expressions to specify column names. Here is my setup: hive> describe tb1; id int name string hive> describe tb2; id int age int You then have to disable the use of quotes in identifiers because that interferes with the regex. hive> set hive.support.quoted.identifiers=none; And now you can use Java Regex to select all columns from the right table that is NOT the key. So essentially you get all columns but the duplicate. If you have non-join key columns that are duplicate you can exclude them and rename them with the AS statement after: hive> create table tb3 as select tb1.*, tb2.`(id)?+.+` from tb1, tb2 where tb1.id = tb2.id; You can see that I select all columns from the left table and then use the `` quotes to specify a regular expression for the columns from the right side I want. The regex is a mean trick essentially asking for any string unless it starts with id ( it says id once or not at all and something following. This means if it matches the full string it will not be matched because the remainder of the regex needs to match something. You could also do (id|id2)?+.+ or (id*)?+.+. This gives me a result table with all columns from the left table and all columns but the key column from the right table. hive> describe tb3; id int name string age int Neat. You never stop learning something new. Hive is really cool. Edit: Actually made a little article out of the question because these regex would have made my life much easier multiple times before. So giving a bit more attention to it seems to make sense: https://community.hortonworks.com/articles/43510/excluding-duplicate-key-columns-from-hive-using-re.html ... View more

The oozie installation makes changes to the core-site.xml. Specifically the oozie user. needs to be able to impersonate other users so he can kick off a job as the user who owns the oozie flow. HDFS allows you to configure these powerful users in the proxyuser settings. So to enable that you need to restart HDFS and yarn. hadoop.proxyuser.oozie.groups hadoop.proxyuser.oozie.hosts ... View more

1. where does this splitting of huge file takes place.? A Client is a (mostly) Java program using the HDFS Filesystem API to write a file to HDFS. This can be the hadoop command line client or a program running in the cluster ( like mapreduce, ... ) In this case each mapper/reducer who writes to HDFS will write one file ( you may have seen mapreduce output folders that contain part-0000 part-0001 files, these are the files written by each mapper/reducer. MapReduce takes these folders as if they are one big file. If you write a file with a client ( let's say one TB) the file is written into the API and transparently chunked into 128MB blocks by the API. 2.Does the client forms 3 pipelines for each block to replicate which run in parallel??. No its a chain. The block is committed once it is persisted on the first node, but it is written in parallel to the other two nodes in a chain Client -> Node1 -> Node2(different rack )->node3(same rack as node2) 3. DN1 which received B1 will start sending the data to DN2 before 128 MB of its block is full?? Yes HDFS API writes in buffer fields I think 64kb or so? So every buffer package is written through at the same time "doesn't that contradict the replication principle where "we will get the complete block of data and then start replicating" " Never heard of that replication principle. And it is definitely not true in HDFS. A file doesn't even need three copies to be written successful a put operation is successful if it is persisted on ONE node. The namenode would make sure that the correct replication level is reached eventually. "Can you also provide the possible reasons why the flow is not the otherwise. Because its faster? If you had to wait for three copies sequentially a put would take much longer to succeed. A lot in HDFS is effiiency ... View more

OK for an OLTP requirement and user lookup you are looking at HBase/Phoenix in the HDP distribution ( other possibilities would be Cassandra, Gemfire ... ) HBase is a NoSQL datastore that allows a simple get/put/scan api on flat tables with a unique primary key and arbitrary number of fields. Phoenix is a SQL layer on top of hbase so its very fast for key lookups/inserts of single rows and can also do aggregations on thousands to millions of rows very efficiently. Both scale up very well since you can add Region servers dynamically. Phoenix can maintain secondary indexes as well which might be helpful in your scenario. ( you can obviously directly maintain that in Hbase by adding a translation table and maintaining it yoruself ) ... View more

The reason is that in a distributed transactional system with a Paxos or similar algorithm you need a quorum ( majority ). Essentially a transaction is committed once more than 50% of nodes say that the transaction is committed. You could do 4 journalnodes / zookeeper nodes as well but you would not get any benefit over 3 nodes and you add additional overhead. 4 Nodes can only survive 1 failed node because 3 journal-nodes are a majority. But not 2. Therefore you need an uneven number. 3 nodes can survive 1 failure, 5 nodes can survive 2 failures, 7 nodes can survive 3 failures and so on ... https://en.wikipedia.org/wiki/Paxos_(computer_science) ... View more

You should be able to simply cache your access objects as a class variable and only create it if it has not been created already. You just need to be a bit careful since the functions are sometimes called with empty conf objects first. Thats what the jdbc storage handler does. ... View more

This is something you should discuss with a Sales person: especially licence costs. http://hortonworks.com/contact-us/ The big advantage of HDP is that it is 100% open source and open and always adds the full Apache version of all components. Components are also in Open Hadoop so there is no lock-in like you have with the proprietary components of Mapr ( Map R file system and their proprietary other components ) The second big advantage of Hortonworks is that we have the widest range of committers in the different components so you will get the best support and influence in adding features. Doing a simple feature by feature comparison doesn't really compare these fundamental differences. ... View more