Implementing smoke testing in production and staging. DIY vs Ad-Hoc Tool

What’s smoke testing?

Smoke testing allows us to quickly asses the status of an application by running a set of end-to-end tests targeted at checking the most important (or the most significant) user flows. It should be run just after a fresh deploy and ideally at regular intervals after that.

Smoke testing is different from a full regression testing in mainly two aspects:

• The coverage of smoke testing is wide and shallow. Its purpose is to roughly test as much functionality as possible in a short time. On the other hand, regression testing is meant to be as thorough as possible to make sure that any existing functionality was not negatively affected by the new changes introduced by a new commit or release.
• Smoke testing should be fast compared to regression testing, as its main purpose is to quickly assess the main user flows within an application.

For this two reasons, smoke testing are well suited to be run continuously at regular intervals to check the status of an application over time. In fact, running tests just after a fresh deploy will only validate the application while in that particular state, just after a new restart. This is why, running a suite of smoke tests at regular intervals will make sure that the application behaves as it should while in different states at different moments in time.

Now, let’s see what are some frequently asked questions about smoke testing, before moving onto more practical matters.

Smoke testing FAQ

Should I run my smoke tests on production or on staging?

Ideally on both. Smoke testing is useful on staging because it allows you to be notified about important problems faster. A thorough regression testing suite usually takes a few minutes to run, sometimes even 30 minutes or an hour or even more, depending on the size of the application. This means that after a new commit or release we may have to wait a long time before knowing if our code broke something major. With smoke testing, instead, we have an almost immediate feedback on the status of a new commit or release. Usually, on staging or testing environments a smoke test is followed by a more detailed regression test.

How long should a smoke test suite take?

For a smoke test to be effective it should be fast. I’d say that a reasonable upper limit is 5 minutes for a small application, 10 for a medium application and 15 for a large application. But, of course the shortest the better.

How can I make sure that my smoke test takes as little time as possible?

There are actually a couple of tricks you can use:

• Limit the use of static wait instructions, such as waiting a predefined time (e.g. 10 seconds) for a page to load. Use smart wait instructions instead, such as wait for an element to appear. Even better, use a tool that can do that for you under the hood (implicit waiting).
• Run the tests in parallel. Make sure that there are no relationships between the various tests in a suite, so that they can be run in parallel. This can usually make a big difference in the total running time.
• Remember to only test the most significant parts of an application. Don’t go too much into the details and make sure to test features that can indicate, with a certain degree of confidence, if other parts of the application are broken too.

When should I run my smoke tests?

Smoke tests should be run at least after every new deploy to staging or production. Even better, you can run your smoke tests after each commit since they won’t take long to run anyway. Finally, you can also use your smoke tests to check the status of your production or staging environment at regular intervals.

Is there a preferred framework for running smoke tests?

Generally speaking no. Any end-to-end testing framework should just do the job. However, as we will see later, you can also go no-framework and use a pre-packaged cloud testing solution to simplify and reduce the maintenance of your smoke testing solution.

Can smoke testing replace regression testing?

No. Smoke testing won’t be able to cover most of the functionality of an application and its purpose is different than that of regression testing. For a serious product, I recommend having both. That said, if you currently don’t have neither of the two, then having at least a smoke test suite is a giant leap forward because it can potentially catch a good chunk of critical issues before they become a problem.

Option 1. Building a smoke testing solution from scratch

Now, let’s see what it takes to build a smoke testing solution from scratch. The core requirement that we have, is that the smoke test suite must be able to run after a new deploy and ideally run continuously at regular interval.

The first requirement of our smoke testing solution can be actually simple if you already have an articulated build process or a CI infrastructure in place. In this case, in fact, you could just add another step in your build script to run your smoke tests.

Choose a testing framework

Next, choose a good end-to-end testing framework. If you are using JavaScript for your tests, then you have many good options:

• Cypress
• Taiko
• Selenium
• WebdriverIO
• Puppeteer
• Testcafe

Each of these frameworks has its pros and cons. For example Cypress and Taiko only support Chromium-based browsers. While Selenium is notoriously the most difficult to deal with. At the same time, Cypress, Taiko and Puppeteer have a very good tester experience.

Choose a test runner (optional)

Optionally, if your testing framework already doesn’t integrate one, you have to choose a test runner (make sure it’s compatible with the testing framework you chose). In this case too, there are many options to choose from:

• Mocha
• Jest
• Jasmine
• Cucumber

Get coding

Next, code your tests manually or use a test recorder such as Selenium IDE. Note, however, that if you use Selenium IDE, your framework choices may be limited.

After you have your tests ready, make sure they are checked-in in a repository (you can decide if in the repository of the main application or somewhere else) and make sure they are available to the CI when it runs.

Finally, when a test fails, most CIs will let you know with a notification (an email, Slack message, SMS, etc.), so there is nothing else to add on that compartment.

Choose a CI platform (if you already don’t have one)

It’s probably redundant to say that if you don’t have a CI already in place, you probably need one to run your smoke tests. Your options are many in this occasion too:

• Travis CI
• Codeship
• CircleCI
• Semaphore
• Gitlab CI
• Github actions

Running a test suite at regular intervals

Now, if you wanted to run your smoke tests at regular intervals, that would be a different kind of beast altogether. In this case, in fact, your CI infrastructure is mostly useless as you would need an always-on infrastructure to be able to continuously run your tests. The options in these case are mainly two:

• Have a dedicated machine in the cloud for running your tests. Setup a cron job to start the smoke test suite at your desired interval.
• Setup your tests to be run in a serverless environment, such as AWS Lambda, Azure functions, Google Cloud Functions. In this case the serverless function can be setup to run at specified intervals natively, without the need of any cronjob.

The second options is probably easier to maintain in the long run, but it requires some prior knowledge of serverless programming and the particular cloud platform in use. In both situations, however, you would have to implement ways to keep the test suites up to date on the remote machine or on the serverless platform, plus you would need to implement a solution for getting notified of failing tests. You can, for example, send an email to yourself or to you entire team, or a send a Slack notification on your preferred channel.

Option 2. Using a web testing tool built for the purpose

As we have seen in the previous section, implementing a web smoke testing solution from scratch can be a quite laborious matter. If you already have a CI infrastructure in place, then you may get out of it a little bit easier, but still, if you want to run a test suite continuously, then it’s almost impossible to avoid to implement and maintain extra tooling. All those homemade solutions will likely become technical debt, as over time the focus shifts to implementing tests rather than maintaining the infrastructure on which they run.

Based on those assumptions, choosing a tool specifically built for the purpose (without reinventing one) is often the most sensible option, which will repay itself many times even in the short run. To give you an idea of what those tools are capable of, take a look at this list:

• With a dedicated tool, usually you just need to create tests and run them. There is no need to choose frameworks, CI platforms, test runners or test recorders. All of them are integrated into one single product and completely hidden to you.
• You can easily run a test suite on-demand, using webhooks.
• You can schedule a test suite to run at regular intervals using a built-in scheduler.
• You can have a full report of the last test run as well as stats about the previous runs.
• All kinds of notifications are taken care of out-of-the-box.

In essence, the purpose of those tools is shifting the efforts of testing from setting up and maintaining an infrastructure to just actually writing and maintaining tests. The only disadvantage that I can think of, is that most of these tools are not well suited to satisfy very custom requirements, in which case you may need to go back and do it yourself. But to be honest, this is a common rule in software development.

There are a few tools out there that match the description we’ve given above. Each tool has its pros and cons, some are cheap other more expensive, but in general they all can improve dramatically your smoke testing experience.

One of those tools is Frontend Robot.

Frontend Robot is a fully cloud-based tool that simplifies the whole testing experience. It provides Smart Triggers to trigger the execution of a test suite on-demand, and the ability to run tests at regular intervals with its Run Scheduler. But this only scratches the surface, take a look the our homepage for more information.

Conclusions

Smoke testing ia a crucial part of a software development pipeline, however is often overlooked and misunderstood. Like any other type of tests, setting up and maintaining a custom built solution is often tedious and expensive, with results that most of the time are sub-optimal. Those are exactly the pain points that modern testing tools try to solve. Using a dedicated tools for implementing your smoke testing solution is often the best choice in terms of adoption time, costs and overall developer/tester happiness. The only exception would be if your requirements are so custom that a pre-packaged solution is not an option, but this is usually quite rare occurrence.

PS: For an overview of more testing tools with a good tester experience, refer to the following article: 5 web testing tools with the best Tester Experience (TX).

As always, if you have any questions or feedback you can send me an email at mario@frontendrobot.com.