... | ... | @@ -2,7 +2,7 @@ |
|
|
|
|
|
# Parsing Infrastructure
|
|
|
|
|
|
An overview of the parsing infrastructure, this is what we want to have, not yet what we do have.
|
|
|
An overview of the planned parsing infrastructure.
|
|
|
|
|
|
Every element can be used semi independently, and they are connected through queues when working in a full scale deployment. The queues (kafka, flink) only exchange relatively little information and have been demonstrated to be able to handle very high throughput. Thus we can scale each one of these pieces independently (adding extra workers).
|
|
|
We can also experiment with various pipelines
|
... | ... | |