Monitoring your cache effectiveness in Magento 2

I’ve recently given a talk on the subject of monitoring your Full Page Cache (FPC) effectiveness for Magento 2, and I was lucky to be given the opportunity to share my thoughts on this at Mage Titans Italy and also at Magento Live UK.

FPC is a layer on top of our site that helps to push performance further. It is not an excuse to mask inefficient code elsewhere in the application. That said, it is an integral part of Magento 2 that should be monitored to ensure it is working effectively and not hurting our potential site performance.

Full Page Cache (FPC) is not an excuse to have slow PHP / Magento code

In the talk I provide a top-level overview of how FPC works in Magento 2 and the integral part that Varnish plays in this. How Varnish sits in front of our web server and therefore Magento, and how it effectively acts as a web server receiving all requests for our domain. Varnish is very fast at replying to these requests if it holds a cached copy of the response (a cache “Hit”), if it doesn’t, it passes the request on to the web server (a cache “Miss”) and therefore Magento to generate.

I then discuss the minimum information required to be logged from the system that can empower us to analyse the FPC effectiveness, this data stems from a Magento 1 module called Mpchadwick_PageCacheHitRate by Max Chadwick that does what we need but for Magento 1.

Once it is understood what we want from Magento 2, I highlight why we cannot solely use a Magento module as per Magento 1, due to the way caching works with Varnish.

So how do we get the data we need?

I explore some alternative options along with how we finally achieved similar logging using the varnishncsa command-line tool. This tool will start logging all requests received by a Varnish server in the same way we have an access log for a standard web server. The cool thing is we can specify a custom format and apply filtering to get close to logging the data we need.

There are two challenges which remain though:

1. Varnish is handling all requests to our domain, therefore the log will include data for requests including static assets. I.e CSS, JS and images.
2. We want the route of the Magento Controller that is responsible for generating the response to be included, but Varnish has no concept of this.

The missing piece of the puzzle is to have a simple Magento 2 module in place that adds a custom HTTP header to each response that is returned to Varnish. This header contains the route/controller that is responsible for generating the requested page content. The header can then be used in our custom log format, it can also be used as a filter to ensure the log only contains page requests rather than all requests to Varnish (therefore excluding all static assets such as CSS, JS and images).

Once you have this log, you can push it into any logging or analysis software that you see fit.

We are using New Relic Insights, and in the talk I conclude by sharing some graphs of problems we have experienced in the wild, I throw this out to the audience to see if anyone can guess what is causing those problems.

If you are interested in this topic, please check out the video below from Mage Titans Italy, and maybe pause the video at the right point to see if you can guess the cause of the issues I share?

Let me know if you have any feedback or questions.