I'm trying to calculate an averange of randomForest predictions in Spark 1.3.1, since the predicted probability of all trees is available only in 1.5.0. The best I could do until now is using the function below: def calculaProbs(dados, modelRF):
trees = modelRF._java_model.trees()
nTrees = modelRF.numTrees()
nPontos = dados.count()
predictions = np.zeros(nPontos)
for i in range(nTrees):
dtm = DecisionTreeModel(trees[i])
predictions+= dtm.predict(dados.map(lambda x: x.features)).collect()
predictions = predictions/nTrees
return predictions This code is running very slow, as expected, since I'm collecting (collect()) predictions from each Tree and adding them up in Driver. I cannot put the dtm.predit() inside a Map operation in this version of Spark. Here is the Note from documentation: "Note: In Python, predict cannot currently be used within an RDD transformation or action. Call predict directly on the RDD instead." Any Idea to improve performance? How can I add values from 2 RDDs without collecting their values to a vector?
... View more
