Smoke testing is the most profitable type of testing in terms of benefits/effort ratio and for this reason it should always be a part of a modern web development pipeline. In this article we'll explore the pros and cons of implementing a DIY solution vs using a pre-packaged web testing tool built for the purpose.
Smoke testing allows us to quickly asses the status of an application by running a set of end-to-end tests targeted at checking the most important (or the most significant) user flows. It should be run just after a fresh deploy and ideally at regular intervals after that.
Smoke testing is different from a full regression testing in mainly two aspects:
For this two reasons, smoke testing are well suited to be run continuously at regular intervals to check the status of an application over time. In fact, running tests just after a fresh deploy will only validate the application while in that particular state, just after a new restart. This is why, running a suite of smoke tests at regular intervals will make sure that the application behaves as it should while in different states at different moments in time.
Now, let’s see what are some frequently asked questions about smoke testing, before moving onto more practical matters.
Ideally on both. Smoke testing is useful on staging because it allows you to be notified about important problems faster. A thorough regression testing suite usually takes a few minutes to run, sometimes even 30 minutes or an hour or even more, depending on the size of the application. This means that after a new commit or release we may have to wait a long time before knowing if our code broke something major. With smoke testing, instead, we have an almost immediate feedback on the status of a new commit or release. Usually, on staging or testing environments a smoke test is followed by a more detailed regression test.
For a smoke test to be effective it should be fast. I’d say that a reasonable upper limit is 5 minutes for a small application, 10 for a medium application and 15 for a large application. But, of course the shortest the better.
There are actually a couple of tricks you can use:
wait
instructions, such as waiting a predefined time (e.g. 10 seconds) for a page to load. Use smart wait instructions instead, such as wait for an element to appear. Even better, use a tool that can do that for you under the hood (implicit waiting).Smoke tests should be run at least after every new deploy to staging or production. Even better, you can run your smoke tests after each commit since they won’t take long to run anyway. Finally, you can also use your smoke tests to check the status of your production or staging environment at regular intervals.
Generally speaking no. Any end-to-end testing framework should just do the job. However, as we will see later, you can also go no-framework and use a pre-packaged cloud testing solution to simplify and reduce the maintenance of your smoke testing solution.
No. Smoke testing won’t be able to cover most of the functionality of an application and its purpose is different than that of regression testing. For a serious product, I recommend having both. That said, if you currently don’t have neither of the two, then having at least a smoke test suite is a giant leap forward because it can potentially catch a good chunk of critical issues before they become a problem.
Now, let’s see what it takes to build a smoke testing solution from scratch. The core requirement that we have, is that the smoke test suite must be able to run after a new deploy and ideally run continuously at regular interval.
The first requirement of our smoke testing solution can be actually simple if you already have an articulated build process or a CI infrastructure in place. In this case, in fact, you could just add another step in your build script to run your smoke tests.
Next, choose a good end-to-end testing framework. If you are using JavaScript for your tests, then you have many good options:
Each of these frameworks has its pros and cons. For example Cypress and Taiko only support Chromium-based browsers. While Selenium is notoriously the most difficult to deal with. At the same time, Cypress, Taiko and Puppeteer have a very good tester experience.
Optionally, if your testing framework already doesn’t integrate one, you have to choose a test runner (make sure it’s compatible with the testing framework you chose). In this case too, there are many options to choose from:
Next, code your tests manually or use a test recorder such as Selenium IDE. Note, however, that if you use Selenium IDE, your framework choices may be limited.
After you have your tests ready, make sure they are checked-in in a repository (you can decide if in the repository of the main application or somewhere else) and make sure they are available to the CI when it runs.
Finally, when a test fails, most CIs will let you know with a notification (an email, Slack message, SMS, etc.), so there is nothing else to add on that compartment.
It’s probably redundant to say that if you don’t have a CI already in place, you probably need one to run your smoke tests. Your options are many in this occasion too:
Now, if you wanted to run your smoke tests at regular intervals, that would be a different kind of beast altogether. In this case, in fact, your CI infrastructure is mostly useless as you would need an always-on infrastructure to be able to continuously run your tests. The options in these case are mainly two:
The second options is probably easier to maintain in the long run, but it requires some prior knowledge of serverless programming and the particular cloud platform in use. In both situations, however, you would have to implement ways to keep the test suites up to date on the remote machine or on the serverless platform, plus you would need to implement a solution for getting notified of failing tests. You can, for example, send an email to yourself or to you entire team, or a send a Slack notification on your preferred channel.
As we have seen in the previous section, implementing a web smoke testing solution from scratch can be a quite laborious matter. If you already have a CI infrastructure in place, then you may get out of it a little bit easier, but still, if you want to run a test suite continuously, then it’s almost impossible to avoid to implement and maintain extra tooling. All those homemade solutions will likely become technical debt, as over time the focus shifts to implementing tests rather than maintaining the infrastructure on which they run.
Based on those assumptions, choosing a tool specifically built for the purpose (without reinventing one) is often the most sensible option, which will repay itself many times even in the short run. To give you an idea of what those tools are capable of, take a look at this list:
In essence, the purpose of those tools is shifting the efforts of testing from setting up and maintaining an infrastructure to just actually writing and maintaining tests. The only disadvantage that I can think of, is that most of these tools are not well suited to satisfy very custom requirements, in which case you may need to go back and do it yourself. But to be honest, this is a common rule in software development.
There are a few tools out there that match the description we’ve given above. Each tool has its pros and cons, some are cheap other more expensive, but in general they all can improve dramatically your smoke testing experience.
One of those tools is Frontend Robot.
Smoke testing ia a crucial part of a software development pipeline, however is often overlooked and misunderstood. Like any other type of tests, setting up and maintaining a custom built solution is often tedious and expensive, with results that most of the time are sub-optimal. Those are exactly the pain points that modern testing tools try to solve. Using a dedicated tools for implementing your smoke testing solution is often the best choice in terms of adoption time, costs and overall developer/tester happiness. The only exception would be if your requirements are so custom that a pre-packaged solution is not an option, but this is usually quite rare occurrence.
PS: For an overview of more testing tools with a good tester experience, refer to the following article: 5 web testing tools with the best Tester Experience (TX).
As always, if you have any questions or feedback you can send me an email at mario@frontendrobot.com.