Hi, I have a file, which can be used for lookup during a data flow. How do I read the file and put it in the DistributedCache? Thanks, Avijeet ... View more

Hi All, I understand SOLR creates a index file and makes searches faster - however I have a fundamental question - Does SOLR stores the data + index - for example if I have a Table with 100 columns, and I want index on a few columns Will SOLR store all the Table data so that it can show the full row on search match OR The full file can be in HDFS/HBASE and SOLR can look it up and show the full row? So can there be an approach where the Data is in HDFS and the primary/secondary indexes in SOLR - and search can find the full data in HDFS. Not only find , can also update / delete. Thanks, Avijeet ... View more

Hi All, most of the batch processing frameworks (MR, Spark) support a local mode and a distributed mode (standalone, yarn, mesos) of deployment and execution. what about stream processing frameworks such as STORM, Spark-streaming? Do they manage the distributed mode on their own? is it even realistic to expect them to be work on YARN? How to monitor a distributed spark streaming job? And do we need to specify master as yarn to make it distributed? Thanks, Avijeet ... View more

Hi, I have been seeing stream processing use cases where as part of streaming ingest along with HBASE, Cassandra etc. HDFS is also shown. Isn't HDFS write was supposedly only with big files 64MB/128MB +. In Flume this is achieved by hdfs.rollSize configurations. So Flume manages the buffer until it becomes big, then it writes/flushes it out. How does this part is taken care when writing from Spark-streaming or STORM? Thanks, Avijeet ... View more

Hi All, HIVE has been established as an analytics engine (SQL query processing) for large file based data. The new features added to HIVE such as ACID, Streaming, updates etc. how does these features fit into the overall HIVE positioning? Is the idea to create a all-in-one DB on HIVE ? Thanks, Avijeet ... View more