In today applications recommendations became quite necessary for better audience/customer targeting. So, as you could assume, we have our own recommendation engine for that,
using Symfony with a Neo4j database. Communication between the main application and the recommendation engine is done via messages over RabbitMQ.
We are sending interesting events that happened on our website to the recommendation engine and then pulling some recommendations from it.
Over time we noticed something – our message queue got quite clogged, over 2 millions of messages were waiting to be written in the database! So we dug up a bit and found out
that Neo4j is not quite optimized for one by one writing, it is recalculating relations after each insert, and we had large amounts of messages that just kept coming
in. So we needed to fix that, in order to have up to date recommendations for our system.
After reading some articles, and Neo4j official docs, we found several guidelines:
We decided to check how that really impacts the write queries to the database so we made a little experiment. We ran the insert of 5000 entries with ORM, without ORM and using UNWIND:
The write with ORM took 2 minutes and 21 seconds, using CYPHER 43 seconds, using UNWIND only 4 seconds!
When checking the messages stuck on the queue we noticed one message (AdvertServedToUser) was 80% of the total messages. So we decided to move it out of ORM and use UNWIND.
We created another queue for our batch messages, and when consuming them from the original queue – we just re-queued them to another. We were using RabbitMqBundle for consuming messages from the queue and Symfony Messenger inside application.
We had to change routing for the specific message in order to put it on the different queue:
The next step was to get messages from the new queue in a batch of 50 or more (we ended up getting 5000 of them in one batch). In order to do that we had to create consumer class:
and use our consumer as callback for batch RabbitMQ
After we’ve set all up and deployed the code the queue went empty in few days, the app continued running smoothly without clogging of the queue.