End-to-end tests are usually run on a testing or staging environment, and for good reasons. But there are instances when running your tests on production is not that crazy after all.
There are good reasons for not running your tests on a production environment.
First of all, when your code is on production it’s usually already too late for testing 😓. You want to run your tests so that your customers don’t get a broken product, so usually you should test your code before it arrives into your customers’ hands.
Secondly, you don’t want to mess with production without first messing with staging. This is really a consequence of the point made above. If you haven’t run your tests on a staging environment first and your tests break things up, they’ll break production and that’s - to put it mildly - undesirable. Tests are meant to find bugs, and bugs are unpredictable, you can never be sure of their extent. In short, you may end up in tearing down your production environment or even destroying data. On the positive side, it’s better that your tests find the problem instead of your customers, at least you get notified.
Another reason for not running your tests on production is the complexity of the clean up process. On a testing/staging environment you can just bootstrap the database from scratch every time you deploy, effectively resetting the state of the application after every test. It may not seem a problem at first, but imagine running you tests multiple times a day for years. You will end up having a lot of noise in your production data, which will make it difficult to extract meaningful information from it.
Even if running your tests on production may not seem a good idea at first, it has some important advantages. Of course, as we will see later, you shouldn’t run your tests on production without having a proper plan in place to mitigate the negative aspects we’ve just talked about. That said, let’s see why you should run some tests on production.
Well, first of all, your staging environment is not what your customers use. A staging environment should replicate as closely as possible the production setup, however, your staging environment will never be exactly the same as your production environment. At the very least you will have different URLs, but you can also have different API keys for external services or variables that enable/disable things when on production rather than on staging. These are your dark spots in your QA strategy, and can only be removed by testing your production environment as well.
Also, the good health of your application doesn’t come only from its code, but from a combination of:
All the above factors are necessarily different between your staging and your production environments. If your data becomes corrupt on production because of a bug you missed, your application may become unstable. If your storage provider has a hiccup, your user won’t be able to upload their files. If you have wrongly setup some file permissions on your production server only, you may not be able to run some parts of your application or access some of the files it needs.
At the end, end-to-end testing it’s also a type of integration testing so you will be able to catch all the above issues - hopefully before your customers will notice them - with a properly designed test plan running on production.
So we have assessed that running some end-to-end tests on production is indeed beneficial, but also that there are some caveats we should take into account. So, below is a list of good practices to consider when running some end-to-end tests on production.
This is probably the most important aspect. You should at least run the same set of tests on your application before it reaches production, on a staging or testing environment. This will make sure that your tests won’t break anything on production, but it will also catch any major bug before your customers do.
In an ideal setup, we would have a three-tier testing plan:
What we really want to do here is to monitor the production environment rather than testing it. This means that there should be a very limited number of highly meaningful tests, running continuously at short time intervals. This will ensure that your application is actually running and its major components work as they should.
The interval at which the tests should run may vary and it depends on how long the tests take to run. But a good value falls between 15-30 minutes.
Your production data is important. You can use it to improve your product, your customer experience and nowadays even to build powerful Machine Learning models. So it’s important not to have unnecessary noise created by our tests.
A solution is to remove all the data that we have created during our tests. There are various strategies that we can use - and we’ll have a separate blog post to talk about it in more detail - but probably the easiest approach is to use the same test automation to perform the cleanup.
The practice of regularly checking that a website or web application is running as it should is also called monitoring.
At its most low-level implementation we have uptime monitoring, which simply makes sure that your services are up and running. This is usually done by checking the response to a call to a HTTP endpoint. There are many providers out there to chose from.
If we decide to run our end-to-end tests on production at regular intervals, we are essentially implementing some frontend monitoring (some call it also user flow monitoring).
The scope and purpose of the two techniques is different but it overlaps in many points. Both are intended to make sure that a production application is running as it should. Uptime monitoring usually runs very basic checks that require instants to run, and that’s why the monitoring interval is usually very short, down to 1 minute in many instances. The checks (or tests) performed for frontend monitoring on the other hand require longer to run, but are also more exhaustive as they will take into account also the User Interface subsystem and its integration with the backend.
The words test and production rarely appear on the same sentence and that’s due to very good reasons. However we have seen that with a proper testing strategy, we can safely implement a frontend monitoring solution on production.
As always, if you have any questions or feedback you can send me an email at firstname.lastname@example.org or chat to us on our website.