>Agile Testing for Standards Organizations

>I jumped over to the Agile Development way of working several years ago. One of the problems with the way current standards are developed in my opinion is the fact that there is very little validation of the requirements. If there is validation of the requirements it is an entirely manual process that is pushed to the very end after “we have all the requirements done and implemented.”

This approach to validation of your end users requirements is the same problem that happens with traditional waterfall development of sofware applications, you miss things. In fact, you may accidently have deleted something several months ago, that is now just being caught. Or if your standard organization releases it’s specifications once everything is done, the mistake might not be caught for several years.

Unit Testing and User Acceptance Testing can provide the following:

  • Verification that existing requirements still work.
  • Verification of backwards compatibility and forward compatibility.
  • Verification that new requirements are implemented as asked.
  • Verification that a team working on the same schemas doesn’t accidently break other consumers of the schemas. (i.e. common development, and continuous integration)
  • Automated Testing – elimination of the human error that can/will happen with manual reviews.

There are several ways that an organization can implement unit testing. Depending on the tooling you are using and your comfort level here are some options:

  1. XML Schema Unit Testing – integrates with existing JUnit style tests and leverags XML Unit for parts of the framework. Current framework is setup for a UNCEFACT Core Components, UBL, STAR, or OAGIS style xml schemas. (STAR currently uses this framework daily.)
  2. SchemaTron – If your developers and community are comfortable with XML related technologies, then SchemaTron can be used to verify the requirements for the various W3C XML Schemas that are produced.
  3. Quality Design Toolkit – verification of XML Schema Naming and Design Rules. Used in combination with SchemaTron and XML Schema Unit Testing it can provided another layer of verification.
  4. XML Unit – a general framework for testing a variety of xml file formats. Also able to do difference comparison on xml.

The time it takes to set any of these up and integrate it into your development process is easily made up in the amount of time that is saved in the verification and quality of the standards that are produced. Implementing these is the easy part, the hard part comes from the cultural changes that have to occur in the development processes. Everything needs to be done on shorter cycles, and testing needs to be stressed as a critical piece of the process. Implementing the unit testing is the easiest and most beneficial part of the process.

If anybody has thoughts on how to get users that submit requirements to actually report back User Acceptance Test results, I would love to hear them.

This entry was posted in agile, standards, testing, xml. Bookmark the permalink.

2 Responses to >Agile Testing for Standards Organizations

  1. SteveL says:

    >I explored "test-driven standards" for a grid-related standard 5+ years back; Ant has the <schemavalidate> task as a souvenir, and I did some hacks behind Junit to drive it off XML files containing before and after documents. With CI tooling (then, Cruise, now -hudson), you can be sure that your schemas are consistent with your examples. With tests written at the same time as the specs, you also catch when you have specified something that is untestable -and therefore that you've gone wrong.One of the biggest problems here is structural: the standards orgs don't believe in testing, many of the architects in them hadn't lived in the TDD land, and used to think that time spent on tests was a waste of time -or worse, an engineering issue. This isn't just a GGF issue either; I cornered our local W3C TAG member over why XML Schema had been released at 1.0 before a single test was done -"testing isn't an architectural matter" was the response.

  2. David Carver says:

    >@SteveL: You are correct in that many do not do testing. It took some convincing at STAR to switch over to a complete Agile Methodology that had testing in grained from the start.Not testing the user requirements is a huge problem for getting the specifications out to the user community. The W3C is starting to change from when XML Schema was originially created. The XQuery, XPath, XQuery, and DOM specifications now all have test suites that can be used to verify implementations. There are a few standard organizations starting to look at TDD for their data standards as well.However change in development processes are very difficult in many organizations to implement. There is always resistance to change even when that change has been proven to work and is better for the community.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s