Thanks for the quick reply, Do you mean 10000 rows per second? What's the size of those rows? i.e. whats the rough MB/second ingest rate? Yes about 10,000 to 20,000 records per second, somewhere around 75 MB/sec. A good approach is to stage the ingest data into a row based format (e.g. avro). Periodically, when enough data has been accumulated, the data would be transformed into parquet (This could be done via Impala for example by doing an "insert into <parquet_table> select * from staging_table".) Impala can query tables that are mixed format so the data in the staging format would still be immediately accessible. This makes sense. I'm assuming I would need to create the process to do the "insert into..." and there is not a built in timer to run this task on an interval or watch the staging table? There is very little distinction between external and managed. The big difference is if it is managed, the files in HDFS are deleted if the table is dropped. The automatic partition creation is done in the ingest. For example, if you did the ingest via an insert into a date partitioned table, Impala will automatically create partitions for new date values. If it was external, I would not issue an "insert into..." command since since some other process would be putting the data to HDFS and the table would just be an interface over that data, right?
... View more
