In theory, design wise, i've been debating with a developer on an architecture using Spark and Kafka for a data processor. In my mind, I prefer a model where data comes into a queue in Kafka, gets picked up by a dispatcher service (spark) and distributed into another Kafka queue based on the content delivered. He prefers a single Kafka queue, where the spark application does all the individual extraction/disassembly of data. My argument for the multiple queues is I can divide up data that is multi-type (emails for example), and distribute them into queues to be more efficient. He feels that a single Spark application can do that more efficiently based on the distributed model. Which one is better? Is one better? ... View more

I've been mulling over a design architecture for a deployment and i'm looking for some input on how to physically layout the environment. My current thought is building a single data storage cluster with HDFS, made up with small machines with large storage and building a separate cluster for my processing layer (spark/yarn/oozie/elastic/etc...) and a DB cluster holding Hive. I don't know if this model is necessarily efficient though or if I should stick with a single cluster, and just manage the services on each individual node. What is everyone's thoughts on these two model options? ... View more

I've successfully created a process in Nifi to send a POST to a web service, but i'm unable to find out how to incorporate the post response into the httpresponse for the caller. Does anyone know if this is possible or did I go the wrong route for a post/modify/response service? EDIT: the flow I have is: HandleHttpRequest --> PutFile --> PostHTTP --> ReplaceText --> PutFile --> HandleHttpResponse the only thing that seems to apply a change to the text is the ReplaceText being submitted. The first PutFile contains the text submitted, the second PutFile only contains the text modified by the replacetext. the PostHTTP result does not seem to be incorporated into the result. ... View more