i use hdf 3.2 and have a stream flow of text that may contain emoji. i need to extract it into array in scale of near real time then store it on elasticsearch. this stream flows in apache nifi with approximate 100 tweets per seconds.
what is the best or better solution/architecture to came on this need? i have couple of idea that listed below.
A) create a web service to extract emoji from input text and then send nifi flows on to it then gather response.
B) same previous step, plus using apache kafka.
C) change architecture to use some feature of Apache Spark or Storm or Flink.
D) Elasticsearch custom mapping?
... View more