Using a relatively agile, iterative approach to development means that a more traditional looking “big bang” approach to developing and documenting test cases won’t even begin to work. While we have a high level list of what User Stories will be done on the project, these are definitely changing over time, with some ending up as Epic Stories and others being added, deleted, or modified with each passing day. Even if I were to define the tests up front, it is highly unlikely that the implemented functionality would match the tests by the time we get a test-ready release at the end of an increment. So what to do? I have ended up with four layers of tests, two of which are going smoothly as noted in my last post. They include:
- exploratory / session-based tests every two weeks with the BAs who wrote the stories,
- time-boxed user acceptance test sessions with a group of 8-10 client users every 4-6 weeks,
- automated acceptance / integration tests at least every two weeks,
- automated regression suite(s) based on a subset of the full acceptance / integration tests that may be incorporated in to the regular builds in the future.
The exploratory and UAT tests are structured around the User Stories in order to keep the test contents parallel with implemented functionality. For the exploratory tests, I am defining Charters (a la J & J Bach ) for each User Story in a 2 week iteration, without the automated structure since the small size of the project hasn’t warranted that level of tracking.
For the UAT sessions, we send out the stories to the “in the field” users a week in advance, asking them to make notes on test ideas and special cases they run in to during their regular work week. Using one of the large training rooms at the client site, we bring the users together from around the state to do individual time boxed, informal, exploratory sessions and to create lists of comments that are sent via e-mail to a general mailbox for the project. The group then does a BA-facilitated debriefing session to review their comments and questions to generate a group level set of “fix requests”. This list of requests is then reviewed by the business owners for the project to derive the bugs that go in to the project bug tracking system with their associated business priorities. There is a high level round of triage with the lead developer to decide which are candidates for the next iteration and to identify anything that is technically complex enough to warrant further discussion as an open issue.
So far, this process seems to be providing a great mechanism to get a better understanding of the needs of vastly distributed users early enough to readily incorporate the feedback, while allowing those responsible for approval, budget, and design their chances to participate in “just big enough” discussions that fine tune what will actually be changed. For example, if we get 100 comments from a review session, we end up with about 25 bugs, and 5-7 issues.
The bugs are generally being well received and readily fixed, with a fairly low rate of resubmits to development (~ 5-7%). The rate of verified fixes from the first 6 weeks of this process has been running from 75% for the lowest level bugs to 100% for the High & Critical bugs, with an overall fix rate of 88%. Most of the bugs that did not get fixed were due to bigger design implications that need additional negotiation to be resolved. Overall, the users were delighted with the turnaround and being able to see their concerns being addressed so “quickly” and completely compared to their usual development processes.
The biggest win for me has been that these two portions were able to be put in place very quickly (less than 2 weeks to implement and get the first round of testing done), with a high degree of responsiveness to changes in scheduled work and quick turnarounds, to get a decent level of testing prior to getting the full automated suites in place. These are buying us the time to choose the right automated tools and get the architecture right for this particular project as well as building credibility with both the users and the developers very quickly.
