So, I had to make sure that even if producer is not producing any new messaging, the consumer should be able to reprocess the existing messages to remain fuller when it comes to data.
There are two things which could be inferred from this:
- UI should show consistent data for the various stock exchanges
- People should be able to buy some time to make buy or sell decision
How did I do that?
I had spoken literally in the previous post but here I will be comprehensive about the particular matter.
Concepts used to achieve this:
- seeking to a offset that is filled last 5 minutes ago incase the producer is not producing any new messages. then it would pick up from that offset and start streaming again
- Introducing a cache system which caches each response when the main consumer loop is operating which is supposed to operate when producer is producing
- Then when producer is not producing it would exit the loop and start process the accumulated messages in another subsequent loop
- Each of the loops would call the process_message that send the message to the web socket.
This way we are not a risk of loosing any message incase of any shutdown, because we would manually committing the message after processing and upon returing back from failure it would catch up frm the latest and not the earliest. That should be ofcourse configured in the settings of the kafka.
Here the idea is to capture last 5 minutes data because the latest closing price would be essential to help people make decision either to buy or sell. remember when you wait for the perfect time for buying or selling a stock when you are doing stock marketing. Here its the samee but there is a dashboard showing the buttons rather than you going to the exchanges and checking the rates manually.
I never tried stock marketing but this is under my knowledge.
here this can be seen that the main loop is checking if the message timstamp is within 5 minutes from now then it allows though the loop now here in the loop it is processing the messages and adding the messages to an array which is meant to be used later when the loop is exited. Cache is updated in the loop with the message. This is class level variable hence available through out the class.
Upon exiting the loop the cache messages are processing and we keep on seeing atleast something under 5 minutes on the UI.
