I'm trying to find some document that shows the architecture of Oryx in full details. something like the Class diagram or...
in the main website there is only the explanation and some simple figure that does not show how exactly different components are connecting together ...
components are connecting together ...
any help ?
I am not sure that a class diagram would be so helpful, as it doesn't capture the structure of the project much. However you can look up some free tools that can generate UML from a Java project and those would work. I think the architectural overview here is most useful: http://oryxproject.github.io/oryx/ and module breakdown here: http://oryxproject.github.io/oryx/docs/index.html#Module_Mapping are what you want.
Hello again dear Sowen
in following I write the sequence of what is happening during oryx , according to my understanding. could you please correct me ?
1- there is a HDFS directory which Batch layer has access to it. so it will go there , read the data and compute the model and then it rests
untill next time next time . ( which is based on our configuration. we can set the interval based of our need) --> my question is where it
will store the Model? in the same HDFS directory? OR it send the model instantly to serving layer (using kafka) but anyway the model should
store somewhere that serving be able to use it...
2-then Serving layer will find the model (somehow) , and can answer the queries from the user based on the model.
3- as time passes we may have new data , so this new data will be written again in the HDFS directory (exactly the same as previous ?? ),
and the serving layer will send this data to the speed layer ( or speed layer directly read this data form HDFS ?) and speed layer will compute
the update model ...
4-now the serving layer will find this new model use it model to answer the queries
Well, the batch layer reads input from Kafka. It writes the input to HDFS, but also reads previous input from HDFS and uses all of it to build a model. The model is sent to a different Kafka topic. (It is also written to HDFS actually, in case other apps want to consume it that way.) Serving layer reads the Kafka topic to get the model. Serving layer can also write input to the input topic. No, nothing reads from HDFS except the batch layer.