Getting early UAT feedback

When I first considered getting UAT feedback early in the life of a project, it seemed like a great idea in all ways. As frequently happens, though, “seemed” is the key word. There are unintended consequences that I now know to expect on future projects that include both behavioral and process impacts.

Behaviorally, I have noticed that conditioning from non-agile projects seems to be hard to shake for those who are new to this style of project. Normally, by the time UAT comes around for a non-agile project, the time available for fixes has become quite limited and prioratization sessions can get almost cutthroat. Users want changes to the interface that will make their lives easier while programmers may be scrambling just to get all the needed functionality working correctly enough to satisfy release requirements. The project manager is all primed to cry “change control” for anything resembling material effort since time is in very short supply and critical trade-offs loom. These reactions and biases are quite reasonable if UAT only occurs at the end of the project, right before implementation. However, when the first round of UAT happens 6 weeks in to a 6 month project, these time constraints are different, but the emotional reactions don’t automatically adjust.

My first thought was that “They are asking for feedback to find out what the users want – of course they should adapt and incorporate things that make sense. Isn’t that what being agile is all about?” I realize that the wounds of any heated design discussions are still fresh, so any criticism is likely to sting a little worse. However, the length of time and pain involved in the original decision don’t mean that the user input is any less valid, so they should just get over it, right? If the user feedback is saying that there were erroneous assumptions, then that should be considered. Otherwise, why are we bothering to ask for their feedback? We can’t just listen to the kudos. If it turns out that the feedback is based on wrong assumptions, then we incorporate that in to training and future sessions. Case closed, right?

However, the question of time trade-offs does remain valid, even though it becomes more tricky to evaluate. It is true that time spent refactoring and changing approaches may take away time that will later be needed for functionality implementation. The problem is that it is far harder to convince people of what the exact opportunity cost is since you haven’t yet defined those later portions in enough detail to do a precise estimate. It may be a problem, or it may save a lot of time and effort later and end up a wash for time invested. Not only can you never be 100% sure what the future cost will be, it is quite likely that you will never be able to prove the exact cost after the fact either since there will be many other factors that impact the final delivered code. You can say that you just need to manage client expectations about what can be delivered, but that doesn’t make you any more able to definitively estimate the cost trade-offs. You may also be in a situation (as we are) where the end-date has been set with an outside (waterfall based) PMO group.

The other challenge is that you may decide to defer those low priority bugs for quite good reasons until you know better what the end-of-project priorities are. However, the users have given you their feedback now, and the clock is ticking while they wait to find out whether you are going to do anything about their requests. Every time they see that “bug” when they do subsequent UAT sessions for later functionality, they will be reminded that their concern still hasn’t been addressed. Even if they know that it is just a question of time, it won’t change the emotional reaction to seeing the problem again and again. They also may or may not remember which things they reported before and which they haven’t. If you have any sizable number of low priority bugs sitting out there, it will be harder to keep track of what is on the list. If those low priority bugs wouldn’t be hard to fix (such as some UI layout changes), it becomes even harder for the users to understand why they are being delayed.

The problem is definitely not intractable, so there are steps you can take to find the right balance between user requests and project schedule priorities. It just isn’t quite as simple as it looked at first blush. I hate to think that creating a “change control” process is a good idea for UAT-based requests, but there is a real risk of opportunity costs down the road that needs to be considered. This seems to be making communication skills and perception management even more critical for successful QA efforts.

Evolving the test strategy

Using a relatively agile, iterative approach to development means that a more traditional looking “big bang” approach to developing and documenting test cases won’t even begin to work. While we have a high level list of what User Stories will be done on the project, these are definitely changing over time, with some ending up as Epic Stories and others being added, deleted, or modified with each passing day. Even if I were to define the tests up front, it is highly unlikely that the implemented functionality would match the tests by the time we get a test-ready release at the end of an increment. So what to do? I have ended up with four layers of tests, two of which are going smoothly as noted in my last post. They include:

- exploratory / session-based tests every two weeks with the BAs who wrote the stories,

- time-boxed user acceptance test sessions with a group of 8-10 client users every 4-6 weeks,

- automated acceptance / integration tests at least every two weeks,

- automated regression suite(s) based on a subset of the full acceptance / integration tests that may be incorporated in to the regular builds in the future.

The exploratory and UAT tests are structured around the User Stories in order to keep the test contents parallel with implemented functionality. For the exploratory tests, I am defining Charters (a la J & J Bach ) for each User Story in a 2 week iteration, without the automated structure since the small size of the project hasn’t warranted that level of tracking.

For the UAT sessions, we send out the stories to the “in the field” users a week in advance, asking them to make notes on test ideas and special cases they run in to during their regular work week. Using one of the large training rooms at the client site, we bring the users together from around the state to do individual time boxed, informal, exploratory sessions and to create lists of comments that are sent via e-mail to a general mailbox for the project. The group then does a BA-facilitated debriefing session to review their comments and questions to generate a group level set of “fix requests”. This list of requests is then reviewed by the business owners for the project to derive the bugs that go in to the project bug tracking system with their associated business priorities. There is a high level round of triage with the lead developer to decide which are candidates for the next iteration and to identify anything that is technically complex enough to warrant further discussion as an open issue.

So far, this process seems to be providing a great mechanism to get a better understanding of the needs of vastly distributed users early enough to readily incorporate the feedback, while allowing those responsible for approval, budget, and design their chances to participate in “just big enough” discussions that fine tune what will actually be changed. For example, if we get 100 comments from a review session, we end up with about 25 bugs, and 5-7 issues.

The bugs are generally being well received and readily fixed, with a fairly low rate of resubmits to development (~ 5-7%). The rate of verified fixes from the first 6 weeks of this process has been running from 75% for the lowest level bugs to 100% for the High & Critical bugs, with an overall fix rate of 88%. Most of the bugs that did not get fixed were due to bigger design implications that need additional negotiation to be resolved. Overall, the users were delighted with the turnaround and being able to see their concerns being addressed so “quickly” and completely compared to their usual development processes.

The biggest win for me has been that these two portions were able to be put in place very quickly (less than 2 weeks to implement and get the first round of testing done), with a high degree of responsiveness to changes in scheduled work and quick turnarounds, to get a decent level of testing prior to getting the full automated suites in place. These are buying us the time to choose the right automated tools and get the architecture right for this particular project as well as building credibility with both the users and the developers very quickly.

Getting back in the saddle

I’m now back working in QA with a real development team on a great project (as opposed to the business side development I was doing, for a group that lived for the “down and dirty” approach to everything). The new project has all the usual issues with managing client expectations, trying to catch up when you are brought in late, and working with new technologies.  The wonderful opportunity, though, is that it is a relatively agile project team, working for a dedicatedly waterfall client, and I am the first QA person most of the developers have worked with on anything other than a regular waterfall project. I have a lot of room to try new things to bridge this gap between agile and waterfall approaches, so it is a terrific challenge.

So far, I have come up with a UAT process that maps to the 6 week incremental releases and a session based testing process for the 2 week iterations that brings the BAs in to the loop. So far, the client and team responses have been terrific. My next big hurdle is to come up with a way the client will be comfortable with of organizing and documenting the automated integration tests, without dedicating large amounts of time for maintenance. I’m limited to open source or otherwise free tools due to the client’s government budget, and have not yet found a tool to support test case management that has the right level of flexibility vs. ease of use. Since it looks like these tests will be the most detailed documentation we will be providing the client, they need to be in a format that is organized and approachable, so large buckets of code will not work. Suggestions are welcome!