Getting early UAT feedback

When I first considered getting UAT feedback early in the life of a project, it seemed like a great idea in all ways. As frequently happens, though, “seemed” is the key word. There are unintended consequences that I now know to expect on future projects that include both behavioral and process impacts.

Behaviorally, I have noticed that conditioning from non-agile projects seems to be hard to shake for those who are new to this style of project. Normally, by the time UAT comes around for a non-agile project, the time available for fixes has become quite limited and prioratization sessions can get almost cutthroat. Users want changes to the interface that will make their lives easier while programmers may be scrambling just to get all the needed functionality working correctly enough to satisfy release requirements. The project manager is all primed to cry “change control” for anything resembling material effort since time is in very short supply and critical trade-offs loom. These reactions and biases are quite reasonable if UAT only occurs at the end of the project, right before implementation. However, when the first round of UAT happens 6 weeks in to a 6 month project, these time constraints are different, but the emotional reactions don’t automatically adjust.

My first thought was that “They are asking for feedback to find out what the users want – of course they should adapt and incorporate things that make sense. Isn’t that what being agile is all about?” I realize that the wounds of any heated design discussions are still fresh, so any criticism is likely to sting a little worse. However, the length of time and pain involved in the original decision don’t mean that the user input is any less valid, so they should just get over it, right? If the user feedback is saying that there were erroneous assumptions, then that should be considered. Otherwise, why are we bothering to ask for their feedback? We can’t just listen to the kudos. If it turns out that the feedback is based on wrong assumptions, then we incorporate that in to training and future sessions. Case closed, right?

However, the question of time trade-offs does remain valid, even though it becomes more tricky to evaluate. It is true that time spent refactoring and changing approaches may take away time that will later be needed for functionality implementation. The problem is that it is far harder to convince people of what the exact opportunity cost is since you haven’t yet defined those later portions in enough detail to do a precise estimate. It may be a problem, or it may save a lot of time and effort later and end up a wash for time invested. Not only can you never be 100% sure what the future cost will be, it is quite likely that you will never be able to prove the exact cost after the fact either since there will be many other factors that impact the final delivered code. You can say that you just need to manage client expectations about what can be delivered, but that doesn’t make you any more able to definitively estimate the cost trade-offs. You may also be in a situation (as we are) where the end-date has been set with an outside (waterfall based) PMO group.

The other challenge is that you may decide to defer those low priority bugs for quite good reasons until you know better what the end-of-project priorities are. However, the users have given you their feedback now, and the clock is ticking while they wait to find out whether you are going to do anything about their requests. Every time they see that “bug” when they do subsequent UAT sessions for later functionality, they will be reminded that their concern still hasn’t been addressed. Even if they know that it is just a question of time, it won’t change the emotional reaction to seeing the problem again and again. They also may or may not remember which things they reported before and which they haven’t. If you have any sizable number of low priority bugs sitting out there, it will be harder to keep track of what is on the list. If those low priority bugs wouldn’t be hard to fix (such as some UI layout changes), it becomes even harder for the users to understand why they are being delayed.

The problem is definitely not intractable, so there are steps you can take to find the right balance between user requests and project schedule priorities. It just isn’t quite as simple as it looked at first blush. I hate to think that creating a “change control” process is a good idea for UAT-based requests, but there is a real risk of opportunity costs down the road that needs to be considered. This seems to be making communication skills and perception management even more critical for successful QA efforts.

Evolving the test strategy

Using a relatively agile, iterative approach to development means that a more traditional looking “big bang” approach to developing and documenting test cases won’t even begin to work. While we have a high level list of what User Stories will be done on the project, these are definitely changing over time, with some ending up as Epic Stories and others being added, deleted, or modified with each passing day. Even if I were to define the tests up front, it is highly unlikely that the implemented functionality would match the tests by the time we get a test-ready release at the end of an increment. So what to do? I have ended up with four layers of tests, two of which are going smoothly as noted in my last post. They include:

- exploratory / session-based tests every two weeks with the BAs who wrote the stories,

- time-boxed user acceptance test sessions with a group of 8-10 client users every 4-6 weeks,

- automated acceptance / integration tests at least every two weeks,

- automated regression suite(s) based on a subset of the full acceptance / integration tests that may be incorporated in to the regular builds in the future.

The exploratory and UAT tests are structured around the User Stories in order to keep the test contents parallel with implemented functionality. For the exploratory tests, I am defining Charters (a la J & J Bach ) for each User Story in a 2 week iteration, without the automated structure since the small size of the project hasn’t warranted that level of tracking.

For the UAT sessions, we send out the stories to the “in the field” users a week in advance, asking them to make notes on test ideas and special cases they run in to during their regular work week. Using one of the large training rooms at the client site, we bring the users together from around the state to do individual time boxed, informal, exploratory sessions and to create lists of comments that are sent via e-mail to a general mailbox for the project. The group then does a BA-facilitated debriefing session to review their comments and questions to generate a group level set of “fix requests”. This list of requests is then reviewed by the business owners for the project to derive the bugs that go in to the project bug tracking system with their associated business priorities. There is a high level round of triage with the lead developer to decide which are candidates for the next iteration and to identify anything that is technically complex enough to warrant further discussion as an open issue.

So far, this process seems to be providing a great mechanism to get a better understanding of the needs of vastly distributed users early enough to readily incorporate the feedback, while allowing those responsible for approval, budget, and design their chances to participate in “just big enough” discussions that fine tune what will actually be changed. For example, if we get 100 comments from a review session, we end up with about 25 bugs, and 5-7 issues.

The bugs are generally being well received and readily fixed, with a fairly low rate of resubmits to development (~ 5-7%). The rate of verified fixes from the first 6 weeks of this process has been running from 75% for the lowest level bugs to 100% for the High & Critical bugs, with an overall fix rate of 88%. Most of the bugs that did not get fixed were due to bigger design implications that need additional negotiation to be resolved. Overall, the users were delighted with the turnaround and being able to see their concerns being addressed so “quickly” and completely compared to their usual development processes.

The biggest win for me has been that these two portions were able to be put in place very quickly (less than 2 weeks to implement and get the first round of testing done), with a high degree of responsiveness to changes in scheduled work and quick turnarounds, to get a decent level of testing prior to getting the full automated suites in place. These are buying us the time to choose the right automated tools and get the architecture right for this particular project as well as building credibility with both the users and the developers very quickly.

Getting back in the saddle

I’m now back working in QA with a real development team on a great project (as opposed to the business side development I was doing, for a group that lived for the “down and dirty” approach to everything). The new project has all the usual issues with managing client expectations, trying to catch up when you are brought in late, and working with new technologies.  The wonderful opportunity, though, is that it is a relatively agile project team, working for a dedicatedly waterfall client, and I am the first QA person most of the developers have worked with on anything other than a regular waterfall project. I have a lot of room to try new things to bridge this gap between agile and waterfall approaches, so it is a terrific challenge.

So far, I have come up with a UAT process that maps to the 6 week incremental releases and a session based testing process for the 2 week iterations that brings the BAs in to the loop. So far, the client and team responses have been terrific. My next big hurdle is to come up with a way the client will be comfortable with of organizing and documenting the automated integration tests, without dedicating large amounts of time for maintenance. I’m limited to open source or otherwise free tools due to the client’s government budget, and have not yet found a tool to support test case management that has the right level of flexibility vs. ease of use. Since it looks like these tests will be the most detailed documentation we will be providing the client, they need to be in a format that is organized and approachable, so large buckets of code will not work. Suggestions are welcome!

Teaching collaboration – overview

There are quite a few aspects to this that I think are important. As I considered the comments posted here and other verbal comments that I have received, it has become clearer that I think the factors fall into at least two main groups: classroom constraints, and task aspects. In the area of classroom constraints are several issues that the teacher / facilitator needs to work around. In the area of task aspects, I’m jumping off from Ellen Gottesdiener’s book, Requirements by Collaboration as well as other sources. (I haven’t had Ellen’s class, so if I’m off base, please let me know!)

Under classroom and teaching constraints, we have several challenges:
- varied skill levels of the students / participants
- varied goals of the students / participants / (and, at times, sponsors, alas)
- varied learning style preferences (I’m appreciating this aspect more and more with Andy’s research)
- varied past experiences with trust, with the other students, people at large, and in this kind of situation
- and the need to justify the importance of collaboration skills and experiences to the students to gain their cooperation

In the area of the task aspects, Ellen mentions the need for these things for collaboration to take place:
- a shared purpose exists (see the problem with goals above)
- mutual trust
- agreed upon approaches to work

She also mentions the need to acknowledge the existence of different, equally valid viewpoints. While she is focusing on developing requirements, it would seem that these things would be equally true for any collaborative efforts.

Harrison McKnight , at Michigan State University, has done considerable research on the issue of trust, in general and as it relates to IT. I’ve been going back over some of his work as I think about collaboration between testers and the rest of the development team. I’m going to work through the topics I listed above in the next series of posts and will get in to his work on trust as I do.

In thinking about this, I also dug up an article from 1999 by Timothy Lethbridge on “Priorities for the education and training of software engineers”. I was curious to see if there was anything resembling this topic in his list of 75 topics that they surveyed junior and senior practictioners about. The closest I could see were negotiation, psychology, and management. They asked, among other things, how much people had learned in their formal education about each topic. Negotiation was # 75 out of 75. All of the “soft skills” were in the bottom third. What did they forget the most from their education? Differential equations, differential and integral calculus, and linear algebra and matrices.Hmmmm.

Preparing CompSci students for group work?

“Are schools not properly preparing computer science students for group work in professional situations?” asks Tim Van Tongeren in response to a quote about students actively avoiding group work in completing their (group) assignments. My first thought is “How do you prepare someone for group work as a collaborative experience?” It also made me remember a comment from Brian Marick’s blog back in June: “Agile projects have some tricky differences from conventional projects – …(including) a greater dependence on trust between members of different interest groups “ At the time of Brian’s post, I considered responding by saying that I would hesitate to say that there is a significant difference in the dependence on trust between members in agile and non-agile projects. I think that all projects require strong trust to be collaborative, not just agile projects. (I admit to very limited exposure to agile projects, but I have seen the impact of trust and dis-trust on non-agile projects. In my experience, it is absolutely critical to effective collaboration of any sort.)

That brings me to Tim’s question, and a second question in return: How important is trust, or lack of trust, in the students’ choices about whether to work together for group projects? If trust is required for collaboration, as I have observed and as some research is beginning to bear out (I can provide the academic references if anyone is interested), is it reasonable to expect that a teacher can somehow generate trust between group members to enable this collaborative experience?

I do think that it would help students to have had a genuinely collaborative experience. Without it, they may be less likely to put in the work to be part of a collaborative team. I’m not sure that any computer science, or even MIS / business school, class is going to address that well without specifically addressing trust issues. (Business schools are notorious for their group projects, but I haven’t seen much greater success there either.)

I’ve been on a project that did not begin, but became incredibly collaborative. Without exception, the team members were changed by the experience. It never would have happened by just having someone tell us to be collaborative and/or to trust each other, though. The trust was earned, over time and through positive experiences, at both the group and the individual pair levels. It was also fostered and consistently demonstrated by a talented project manager and the project leads. Given the constraints that schools are functioning under these days, can we expect them to prepare students for that aspect of professional work? If you think of what has made your groups successful, could that be replicated through class assignments?

Or is there a different question altogether – if most projects are not as collaborative as we would like, is the classroom perhaps more correctly mirroring reality than our desires?

Life is better as a tester because…

I recently interviewed a candidate for a testing position with our company. This person had started as a developer before moving to testing, and I asked him what drew him to testing. I’ve heard it said before that developers who end up in testing are the ones who couldn’t cut it as a developer, so I’m always curious when I see this kind of career pattern. One part of the answer surprised me as something that had never occurred to me:

“As a developer, you generally don’t get to interact as much with the user or have the chance to work with the application as a whole. Unless you are an architect, you are expected to focus only on this one narrow area. As a tester, you have a lot more contact with the entire team and when you are doing integration testing you get to work with the entire range of the application. You get to see how it works and why.”

Now, this person’s experience is not particularly broad, so one could argue with the degree to which this is true of developers in general, or the degree to which testers get the chance to interact with the entire team, etc. etc.. On the other hand, I have seen developers on big teams appear to be handcuffed by their lack of understanding of the broader picture of the application. It’s as if the leads get to think on that level, but the lower level developers aren’t given the same opportunities to attend the design meetings or do other things that would stengthen their understanding of the application as a whole.

When I compare that to what I have seen within testing teams, there doesn’t seem to be the same level of tunnel vision. Is it because testers tend to be involved in integration testing of some sort on most projects? Or do they end up needing the broader knowledge just to write their test cases or accomplish their executions effectively? Or are we just so short staffed that we have to wear many hats, so we don’t have the luxury of being siloed? I’m not sure. But I do know that I appreciate that aspect of my work. This is one thing that is really *good* about being a tester rather than a developer!

The Documentation Controversy

I know that many in the software field are averse to producing documentation. Some are willing to allow that there is more than one kind of documentation: useful documentation, and then blah blah blah documentation. Others seem to condemn the idea almost completely. I can understand that many people have been burned by an expectation to produce x amount of design specification documentation, or y number of test cases by some arbitrary date. This would be a good cause for aversion, particularly if no one actually checks the documents to ensure that they contain useful information. However, I have to wonder how much of this is blaming the tool for the faults of the tool users and/or the project environment.

While poor documentation can admittedly be harmful by providing an excuse for avoiding real work (“I gave you the requirements. What more do you want?”), good documentation can be worth its weight in gold. As a visual person, I am biased towards models. I am even more biased towards the fruitful conversations that if have often seen follow the introduction of almost any kind of picture of a system, whether static, such as a data model, or dynamic, such as a use case diagram or other process model. I love seeing the understanding that dawns as a diagram is revised and honed to represent some portion or aspect of the system. Hearing someone say “I hadn’t thought of that. What if we do this, will that work?” tells me that the design will be just that much better for the sake of the discussion and the picture.

I have also begun to notice another impact of documentation when there are trust issues in the team or organization:
- it gives others something concrete to look at and evaluate,
- allows them to see what I am doing without having to expose themselves first, and
- gives them a way to improve my understanding.

When they see that I make changes as a result of their comments, they also have the chance to understand that I am listening and respecting their input. If they aren’t forthcoming with comments, I can use it as an excuse to ask them questions. I can say: “I appreciate you taking the time to help me with this. I really need to get this clear so that I don’t do anything that will cause problems for you when we get to testing.” It not only helps to create a bridge, it also strengthens it by creating a foundation for trust.

For me, the bottom line is that the the idea of documentation is not inherently flawed. Without putting something in writing, some conflicts will continue to occur over and over again since no one can “prove” what was agreed to. When I have a chance to gain so much from using documentation effectively, I will just have to cope with the skeptics who have been burned before and hope that I can show them that documentation can have its place.

Paired debugging

I was looking at Chris Sepulveda’s recent post on “Integrating Testers into XP: Initial Thoughts”, and was intrigued by his mention of “paired debugging”. I had the good fortune to be able to do that a couple of times on a recent project and feel that it worked incredibly well. I’m interested in how it has worked for others and whether it was intentional or not. (If I have misunderstood your use of the term “paired debugging”, Chris, please correct me!)

In my case, we had a team that was highly collaborative and incredibly competent on all fronts. We didn’t start that way, of course, but by the time the paired debugging sessions took place, we had developed a healthy mutual respect and a good deal of trust. It was not an XP project, but it was not entirely a traditional approach either. None of our sessions were called out as intentionally paired debugging events. They emerged from the desire of the programmers to solve some tricky and complex coding challenges and the willingness and ability of testers to sit through debugging sessions. The developers wanted to nail their bugs for good after they had gotten a series of related defects on several releases in a row. The testers wanted to stop finding variations on the same bugs in release after release. I was the tester and went through the code in question in an extensive debugging session with the programmer. It was very satisfying and productive in many ways.

On the other hand, I was glad that I didn’t have to explain to any project manager what category I would have charged that time to since it was far in excess of what was considered normal for “defect analysis”. I also don’t know how much programming knowledge the tester needs to have to really be helpful. It did seem to help considerably in my situation, but I don’t have anyone else’s experience to compare to. What experiences have others had with these two issues, time accounting and development expertise of the tester? How important is the existence of trust and a collaborative environment to the effectiveness of paired debugging? Has it been applicable to non-XP projects for others as well?

And so it begins for me…

To provide some context for my comments, I want to explain some of my biases and background. If this sort of thing bothers you, by all means skip the rest of this post. For those who appreciate knowing a bit more about me, I hope this helps.

I confess that I probably would not have heard of blogs, or have gotten so interested in them, if it weren’t for my husband’s current passion for them (see Andy Tinkham’s blog). As it is, I could only hear him bring up someone’s comments from their blog so many times before I had to see what they said for myself. At that point, it became but a short leap to wanting some outlet to express my thoughts on their thoughts. (Anyone who knows me or has worked with me for long can tell you that it is rare for me to lack ideas on others’ thoughts!)

I started out primarily as a math and science geek at a time when only some women were doing so. As an undergraduate in a liberal arts school, I discovered that I had a chance to explore things that didn’t make as much sense to me, like those “artsy” sorts of people. I chose theatrical design for a major, ended up learning far more and very different things than I had imagined I would, and struggled the entire time to excel in a field that I was just not really built for. Later ventures into computer science, accounting, management information systems, organizational theory and industrial psychology were much more my style. The end result of that initial diversion has been a permanent focus on the value of effective communication, developing and percieving multiple perspectives, and the need for collaboration in the activities of my chosen field of software development. (I also made the statement that I would never again work with computers after a harrowing experience with the Dartmouth timeshare system and a statistics class, but that’s another story.)

As I considered this initial post, I struggled with defining myself in a cohesive and focused manner. Do I see myself as a software tester, business analyst, project manager, QA manager, business process architect, or all of the above? Does it matter that I have also dabbled in design, coding, and across many kinds of systems? How does my role impact my statements and thoughts about my field? Andy insightfully suggested that I just begin rather than fret about the answers to those questions since the act of blogging would, over time, answer them anyway. As a consequence, I will mention that my current work involves software testing, so most of my thoughts these days are biased in that direction. Other than that, it will just have to emerge as we go.