While I am not aware of any formula, there is at least a guide available: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_cluster-planning/index.html In principal it says about 24 - 48 GB per data node. For the name node 64GB are supposed to deal with 100 million files. Otherwise my recommendation would be to go for a real use (not test or demonstration) at least with 3 master nodes and 12 slave nodes, but increase the slave nodes as needed for your use. A typically use is 2GB RAM for one MR task, so that can provide a rule of thumb on how many slave node you should add. To be more precise on the sizing, the expected use should be given, ie. will you just use MR, or maybe HBase, or do you need stream processing etc... The more applications running on the slaves, the more RAM you probably need besides the MR task, so it would result in additional nodes. It is also possible to have separated clusters for the stream processing and the hadoop storage. ... View more

Hi Dominique, yes thats correct. ... View more

@Nilesh within the zookeeper the leader is given at this path: /brokers/topics/<<topicname>> at this path a value named 'partitions' is available, providing key value pairs of <<partition id>> : <<broker id (of leader)>> To identify the connection parameter for the broker id, check at the path /brokers/ids/<<broker id> the attributes "host", "port" and "endpoints", where endpoints is a multivalue and can contain more than one endpoint. ... View more

@Ravikiran Dasari: You can see all parameters from hive with "hive -H". hive -H usage: hive -d,--define <key=value> Variable substitution to apply to Hive commands. e.g. -d A=B or --define A=B -e <quoted-query-string> SQL from command line -f <filename> SQL from files -H,--help Print help information -h <hostname> Connecting to Hive Server on remote host --hiveconf <property=value> Use value for given property --hivevar <key=value> Variable substitution to apply to hive commands. e.g. --hivevar A=B -i <filename> Initialization SQL file -p <port> Connecting to Hive Server on port number -S,--silent Silent mode in interactive shell -v,--verbose Verbose mode (echo executed SQL to the console) You can add two or more tables into the same schema if they have different names (which will be the case if you use the timestamp). If you are running your create script in parallel, you could always just get a new timestamp in case the tablename with the timestamp you have already exists. If needed you can add the date stamp as well by curr_timestamp=`date +%Y%m%d_%s` ... View more

you should use a tool like WinSCP to copy files. you will have to configure the host, which is your sandbox (the name or IP should be known, since you used it to connect via browser).It will be possible to use the user maria_dev as well. If you want to use the shell, you can use a ssh client for Windows, like Putty. Of course there are other ways as well to allow file exchange (like creating a network share used by Windows and the sandbox), but they will all need some deeper Linux/Windows knowhow, so the simple way is WinSCP. If you like to go with network shares, let me know. ... View more

my guess would be that this is a race condition on env-2, leading to the situation that your process doesn't really see the renewed ticket. Can you change the logging, so that the threads are logging the ticket dates when failing? If it is a load issue with the KDC, you would see the error messages on other clients as well, and it should also by chance go away again. My assumption is that you have thread by thread getting the authentication error, and when all your threads are 'down' you see that no file is moved anymore. just to be sure (as mentioned I don't think the root cause is to be found here): check that the hfds name node and your AD are in time sync also. ... View more

the machine names from the error log are the expected ones? So this basically means: env-2 is example1.com and example2.com is your hdfs master node (port 8020 should be hdfs file service from the name node)? Are all the issues related to the communication between env-2 and your name node, or do you have other hosts involved as well? does the process on env-1 start 5 times a day, or is it started once and continues to run (sleeping instead of terminating)? the ticket renewal on env-1 is identical to the ticket renewal on env-2? I am just wondering if it is possible, that your process on env-2 only takes the ticket at start-up, and when the ticket expires, it just doesn't pick the renewed ticket? If after a restart of your processes on env-2 all authentication issues are gone for the next around 20h, this might be the case. And if on env-1 the process is starting 5 times a day instead of continuously running it might be the reason that the issue is not occure on env-1. ... View more

Just tried to check your screen shots, can it be that your virutal machine console is already open in your screen background? I am just wondering as on the top it is displayed "viewing Saravanan Chandran'..." What can you see on that window? ... View more