End-to-End Testing – Where is the end?
In recent months, several of our clients have been asking for end to end testing. An important aspect of End to End Testing (E2E) is knowing the scenarios regarding how the real users use the product. So not only do you have to know the system itself and the expected behavior with each button or menu item, but you also have to know the system and be able to simulate real user needs. Sometimes, this can be different from typical test cases and scenarios. We’ve been providing end-to-end testing since the beginning of our time in business (wowubuntu was founded in 2006). Mostly, we have referred to it as system integration testing and did not call it End to end testing. Some may argue about specific terminologies, nonetheless, when we begin discussing End to End testing with our clients, we cover many points to ensure we’re all on the same page.
End to End Testing – 11 Steps
- Understand the System: As a first step, it is critical to have a thorough understanding of the software system or application under test. This includes understanding its architecture, components, and interactions with external systems. Because so many web apps are now built with third-party components or layers, knowing how they should and sound not interact is a key element in developing end to end tests.
- Identify End-to-End Scenarios: Button pressing and getting a valid response or discerning an invalid response is certainly necessary. But beyond that, understanding and Identifying the key user workflows or business processes that span multiple components or systems. These scenarios should cover the entire system from start to finish and represent real-world user interactions. This requires significant domain experience of the software under test. When we test accounting software, for example, we have accountants working on the team to make sure we’re covering scenarios that accountants would execute.
- Define Test Cases: While each end to end scenario is unique, there are still some duplicate components. Therefore, each scenario should be decomposed into individual test cases at the level of granularity where modularity will reduce duplication. Of course, each test case should have a clear objective, inputs, expected outputs, and any preconditions including required data, or setup required.
- Categorize Test Cases: For use in specialized regression testing, it’s convenient to categorize or label tests so that you can extract them for targeted functional regression testing. Test cases can be categorized based on different aspects, such as functional areas, business processes, or system components. This categorization helps in organizing and prioritizing the tests.
- Determine Test Data: Identify the necessary test data for each test case or test scenario. This includes both valid and invalid data to cover different scenarios and edge cases. Therefore, different test data sets should be developed for different scenarios, and procedures or scripts should ensure that the same data is always used, thus expecting the same results.
- Design Test Scripts: Those that can be best automated should be tagged, but in general all test scripts or scenarios should be executed a few times manually to ensure that the test data and results are consistent. This is especially true in the accounting or corporate metrics performance data whereby mistakes in aggregation or decimal places can lead to errors later in the process. Always keep in mind that tests should simulate user interactions, with data input as you would expect and not expect a user to input, and validate the expected outputs.
- Set Up Test Environment: The test environment should replicate the production environment as closely as possible. This may involve configuring servers, databases, network connections, and any external systems or APIs to third-party modules or components. Setting up and taking down multiple test environments dependably and consistently is critical to any testing effort whether it be end-to-end, acceptance testing, or unit testing. Yet, this is where many end-to-end and automated test efforts stumble as organizations don’t sufficiently emphasize that without a valid and consistent test environment (and data) results will be inconsistent and unreliable.
- Execute Test Cases: Once things are set up, the end to end test cases need to be executed in a controlled and systematic manner. When executing in a controlled or systematic manner, you need to track all inputs and outputs including the environment, software build, and test data set used. Inconsistent execution will still get you results, but your analysis will suffer as you won’t be able to effectively correlate and compare across test runs. Follow the defined test scripts and record the results, including any deviations or defects encountered.
- Analyze Test Results: Analyze the test results to identify any failures, defects, or deviations from expected behavior. Notice that analyzing results includes more than defects. Failures or deviations are important as test cases may fail to run and be blocked. Categorizing and understanding why the test script failed, and doing a root cause analysis, is the only way to keep it from happening again. We all know the typical analysis of pass/fail/did not run. Of course, once a defect has been verified, then you should log any defects found and provide detailed information for developers to investigate and resolve including any details they need to replicate the defect including data, steps, screenshots, etc.
- Report and Track Defects: Report the defects to the development team and track their resolution. Provide all necessary information, such as steps to reproduce, environment details, and any supporting evidence. Keep in mind that if the development team cannot reproduce then that’s a defect on your part! Use a defect management process to ensure prioritization and that defects don’t fall through the cracks. Track some basic metrics to understand how you can improve your development process. Use these metrics to conduct root cause analysis to prevent defects from happening in the first place, and understand where more testing is needed.
- Repeat and Iterate: Continuously refine and improve the end-to-end tests based on feedback, changes in the system, or new requirements. As new software features are released, you’ll have to review and revise your end to end tests. Regularly review and update the test cases to ensure they remain relevant and effective. Selectively integrate your end-to-end tests into your CI/CD pipeline for consistent and regular execution.
By following these tips, you can get a head start on your end to end testing and ensure comprehensive coverage of your software application. With thorough end-to-end testing, issues or defects that may arise during real-world usage can be found and corrected before your technical support becomes inundated after a new release.