There are literally a dozen different options here: a) Did you enable SQL Optimization of SPSS ( requires the modeler server licence ) after that it can push tasks into the hive datasource. Not sure if Hive is a supported datasource but I would assume so. You can look into the documentation. https://www.ibm.com/support/knowledgecenter/SS3RA7_15.0.0/com.ibm.spss.modeler.help/sql_overview.htm b) SPSS also supports a set of UDFs for in database scoring but that is not what you want. c) Finally there is the SPSS Analytic Server which can essentially run most functions as an Mapreduce job on the cluster. ftp://public.dhe.ibm.com/software/analytics/spss/documentation/analyticserver/1.0/English/IBM_SPSS_Analytic_Server_1_Users_Guide.pdf Unfortunately if you neither have the Modeler Server licence nor analytic server there is not much you can do besides manually pushing prefilters into the hive database or optimizing your SPSS jobs more. ... View more

@Kartik Vashishta I think you should read an HDFS book :-). Replication is not dedicated to specific discs. HDFS will put 3 copies of the replica on different nodes. He doesn't have to choose a specific disc or node. The only rules are that - All three blocks will be on different nodes - If you have rack topology enabled the second and third copy will be on a different rack from the first copy It does not have to be a specific drive or node. HDFS will search for free space on any node that fits the requirements. The only issue I could imagine would be one huge node and some very small nodes that cannot match the size of the other node in total. ( Have seen this with physical nodes and vm nodes. ) ... View more

If you use a distribution like HDP you cannot individually upgrade a component. They are tested and supported together. So if you want a newer version of Hive you would need to upgrade the whole distribution. HDP 2.5 will have a technical preview of Hive 2.0 for example. If you don't care about support then good luck. You can just try to install Hive manually. The problem is that you will need to upgrade Tez manually as well. The hive.apache.org website will have instructions on getting it to run under "builds". ... View more

Phoenix is installed per default in HDP 2.3 at least in my version. Its just a set of libraries in hbase that are always installed unless I am completely mistaken now. Or do you refer to the Phoenix Query server? Which gets installed as a client on the nodes? ( When you install you can select it in the window where you also select datanodes, nodemanagers etc. If you forgot doing that you can install the PQS later on a host using Ambari on the host page. But the Phoenix libraries should be installed with hbase by default. ... View more

1+2) Its simple the way hadoop works. MapReduce guarantees that the input to the reducers is sorted. There are two reasons for this: a) By definition a reducer is an operation on a key and ALL values that belong to that key regarding from which mapper they come. A reducer simply could read the full input set and create a big hashmap in memory but this would be ridiculously costly so the other option is to sort the input dataset. So it simply reads all values for key1 and the moment it sees key2 it knows that there will be no more values for key1. So you see we have to sort the reducer input to enable the reducer function. b) Sorting the keys gives a lot of nice benefits like the ability to do a global sort more or less for free. 3) Reducers only merge sort the input of the different mappers. so that they have a globally sorted input list, this is low effort since the input sets are sorted. 4) "I have seen like nearly 3 times we are doing sorting and Sorting is too costly operation. No you only sort once. The output of the mappers is sorted and reducers merge sort the inputs from the mappers. It is a single global sort operation. The mappers "local" sort their output and the reducer merges these parts together. And as explained above you HAVE to sort the reducer input for the reducer to work. ... View more

It is a Tez application. They stay around for a while to wait for new dags ( execution graph) otherwise you need to create a new session for every query which adds around 20s to your query time. Its configured here (normally a couple minutes) tez.session.am.dag.submit.timeout.secs ... View more

1. On what basis Application Master decides that it needs more containers? Depends completely on the application master. For example in Pig/Hive he computes the number of mappers based on input splits ( blocks ) so if you have a 1GB file with 10 blocks he will ask for 10 map containers. If you then specified 5 reducers the application master will then ask for 5 containers for the reducers. This is a different configuration for each type of yarn workload. One example of "dynamic" requests are in slider which is a framework you can "flex" up or down based on command line. But again in the end the user tells slider to request more. There is no magic scaling inherent to yarn it depends on your application. 2. Will each mapper have separate containers? In classic MapReduce One map = 1 container. ( Tez on the other hand has container reuse so a container it asked for for a "map" task will then be used for a reducer for example ). And finally we will soon have LLAP which can run multiple map/reduce task in the same container as threads similar to a Spark Executor. So all is possible. 3. Let's say one mapper launched in container and mapper completed 20% of work and if it requires more resources to complete remaining 80% of the task then how the resources will be allocated and who will allocate? If Distribution happens between containers then how it will happen? Again depends. Mapreduce is stupid it knows in advance how much work there is ( number of blocks ) and then asks for the number of mappers/containers it needs and distributes the work between them. For the reducers hive/tez for example can compute the number of reducers it needs based on the output size of the mappers. But once the reducer stage is started it doesnt change that anymore. So your question is not really correct. Summary: In the end you assume Yarn would automatically scale containers on need but that is not what happens. What really happens is that the different workloads in yarn predict how much containers they need based on file sizes/ map output etc. and then ask for the correct amount of containers for a stage. There is no scaling up/down in a single task normally. What is dynamic is yarn providing containers. So if an application master asks for 1000 tasks and there are only 200 slots and some occupied by other tasks. Yarn can provide them to the application master piece by piece. Some application masters like Mapreduce are fine with that. Other application masters like spark will not start processing until they got all the containers running they requested at the same time. Again it depends on the application master. Now there is nothing prohibiting a Application Master to do this if he wanted to do that but its not what happens in reality for most of the workloads like Tez/MapReduce/Spark/ ... The only dynamic scaling I am aware of is in pig/hive between stages as in the application master predicts how many containers it needs for the reducer stage based on the size of the map output. ... View more

First regarding ORC: It is a column store format so it only reads the columns you need. So yes less columns good more columns bad. however its still better than a flat file which reads all columns all the time and is not stored as efficiently ( protobuf, vectorization access ... ) . But its not magic. So the question is if you see big performance hits, is the join correct. Normally the CBO already does a decent job of figuring that out if you have statistics as Constantin says. So that is the first step. The second then is to analyze the explain plan and see if it makes sense. Worst case you could break up the query into multiple pieces with temp tables/with statements to see if a different order results in better performance. I am also a fan of checking the execution with hive.tez.exec.print.summary to see if there is a stage that takes a loong time and doesnt have enough reducers/mappers. I.e. a bottleneck. ... View more

Auxlib works. Its the only thing that works consistently for me. Are you using the hive command line or beeline? Depending on this you need to put the jars into the directory of the hive server or the hive client. You also need to restart hive server. ... View more

Yarn timeline store should cleanup old values. Parameters are: Cleanup cycle ( when he deletes ) yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms Time to live ( what to delete ) yarn.timeline-service.ttl-ms Oh and finally enable the age off yarn.timeline-service.ttl-enable https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_yarn_resource_mgt/content/ref-e54bc3f2-f1bc-4bc6-b6cb-e6337589feb6.1.html I would check those. If you want to clean things you can set that to a low settings and restart. Alternatively you should be able to simply delete the database if you want to its just log information after all. Finally if the parameters are correct and do not work you might want to open a support ticket. ... View more

HDFS: - You need ( per default ) 30GB on the datanodes ( 3x replication ) - On the namenode the space is negligible you have 40 blocks x 3 = 120 blocks ~ 12 kbytes of RAM on the Namenode. ( you need around 100byte of RAM for every block in the Namenode memory. You also need a bit of space on disc but thats even less in the fsimage ( files but not blocks are stored on disc, however namenodes need a bit more since they also store edits and multiple versions of the fsimage. But still very small. HBase: More complicated question. In hbase it depends on the way you store data. Every field in your hbase table is stored in HFiles together with the key, the fieldname, the timestamp ... So if you store it in a single field per row your storage is much less than if you would have hundreds of 2 byte columns. On the other hand you can also enable compression in Hbase so that reduces space. ... View more

The data is a Java class that contains the cluster information. Clusters Centers Statistic ... If you want to work with that you either need to use the spark mlib library to do extraction/scoring etc. OR you can export a lot of these models as PMML which is a XML based standard for Clustering models that is understood by a lot of data mining tools. And can be exported for a lot of the models kmeansModel.toPMML("/path/to/kmeans.xml") https://databricks.com/blog/2015/07/02/pmml-support-in-apache-spark-mllib.html Not all Mlib models support PMML though ... View more