Testing Non-Functional Requirements

or, The role of Acceptance Criteria

A previous note opened a discussion about requirements and particularly non-functional requirements, or NFRs. Since then I have had a number of discussions with learners on courses about these, and specifically how we can prove that the NFRs have been achieved. This is the domain of Acceptance Criteria.


Acceptance Criteria

Acceptance Criteria apply to functional and non-functional requirements alike. They may reasonably be thought of as test conditions, and we all know what they are, but sometimes people can have difficulty when it comes to applying this to NFRs. So, let's make it simple and do it by example.

Take a functional requirement such as capture Customer data and let's imagine that we have defined the set of data that is to be captured and that we know that it is to be captured interactively. This is relatively easy to test. Obviously enough we would look for accurate and complete capture of whatever data is required.

Keeping it really simple, we might know that we need to capture the Customer name and address. So the Acceptance Criteria are that the values of name and address should each be captured and that they should be recorded accurately.

Relationship to testing

This is no more clever than saying that we will need to create a test case which tests that our solution can capture and record accurate values for name and address. So far so good. Now take a non-functional requirement and let's stay with capturing Customer data.

It is usual to need not just to capture data but to capture it quickly. Let's say that the aim is that the transaction that records Customer data should be completed within 1 second, that is from hitting the Enter key to receiving confirmation of acceptance (or not). This speed characteristic is another requirement, a non-functional one and we are used to setting up tests for speed of response. Yet people seem to find a problem at this point. Their difficulty seems to be that, because they have stated that the functional requirement should be completed accurately, completely and within 1 second, that they believe they have already stated the tests that would be needed to be applied. In a way they have, but if we take a different NFR we will see that the seeming simplicity of the NFR can bely a rather more awkward issue.

Continuing the example

For our example imagine that we agree with our customers, the ones paying for the development, that the transaction to capture Customer data should be 'easy to learn'. Now we have a different problem. What constitutes 'easy'? From discussion we may refine this to be a training issue, and perhaps we then define that the way we will prove our solution is to set up training where the target will be that users become proficient in data entry within a set time. Does this give us a new problem? Define 'proficient'. Well in this example we are probably in good shape as we can use the accuracy measure, and so on.

Thus our FR of capture and record Customer name and address is now associated with three Acceptance Criteria which could be expressed as:

  • complete and accurate record of data
  • system response of less than 1 second
  • a training session of no longer than 1 hour to teach users to enter the data with 95% accuracy (say)

Does this give us what we need? Yes, it defines the measures by which we will be able to prove that we have developed what was required. It gives a chance to answer the question 'We may be doing the right things but how will we know we have done them the right way or to the right extent?' which, from my own experience, can be difficult at times.

Summary

To recap, FRs and NFRs need to be tested and Acceptance Criteria is the title we give to the tests we will eventually apply. Sometimes the Acceptance Criteria are implicit in the definition of the requirement but sometimes we have to do a bit more work to define an appropriate test.

Alright, I know that I have taken some easy NFRs. I can almost hear you asking “what about some of the more awkward ones? How do we handle look and feel, or conformance to standards, or even worse, ease of use?”

Well, watch this space.