- It is an amazingly high quality open source test tool. The use of groovy scripts, xPath, and dynamic property expansion created a very robust development environment. Although not being Java programmers, it was initially a bit confusing to sort out where Groovy stopped and Java began. A good book, Groovy in Action, solved that problem.
- We found test case development a bit more tedious than expected with a test cases consisting of a dozen steps and within each step perhaps a dozen assertions. It was just a lot of detail, with lots of mouse clicking to drill into things. We wrote some helper routines to consolidate a lot of the assertion checks which helped to some extent.
- We wound up buying the Pro version of SoapUI as we needed some technical support and this turned out to be a great decision. The support was excellent, we were more productive using the Pro version, and the code remained compatible with the open source version of SoapUI as long as we stayed away from certain Pro features.
- The SoapUI documentation was quite well written, but initially it was a challenge to sort out which things applied specifically to REST based testing.
Solution
- Create a SoapUI project, a rest service within the project, defined each of the 100 resource paths within the service, and created an empty test suite.
- Utest suite properties as a set of global variables. Some properties were used to store dynamic data that was stored and retrieved by various test steps, while other properties contained static content that represented various possible values for customer records. Referencing these properties in a variety of ways throughout the test suite was simple. A few examples.
- Simple Property Expansion: ${#TestSuite#password}
- Dynamic Property Expansion where multiple comma separated values are stored in one property: {=context.testCase.testSuite.properties["accountNumber"].value.split(",")[1]}
- We made extensive use of random numbers for generating values that had to be unique. Yes, there was a remote possibility that we might get a test case failure as a result of a random number collision, but from a practical perspective, this was a non-issue.
- We structured our test suite to run sequentially, with the following initialization routines:
- Generate unique “random” values and store in properties
- Use REST calls to populate the data base with sufficient framework data to register a customer. Example, define the various valid address types such as home, business, other, etc.
- Use REST calls to register a set of customers using characteristics pulled form test suite properties. These customers will not be modified during test execution and will be used exclusively for “get” and “search” customer related testing.
- Define a series of test cases that register customers for subsequent record updating. These can be called from other test cases as needed. Note that we encountered problems exporting and importing test cases that used the Goto Test step (they reference a GUID not the name of the test case), so we chose to call the test cases from a groovy script…
- def testCase = testRunner.testCase.testSuite.testCases["Another TestCase"] def properties = new com.eviware.soapui.support.types.StringToObjectMap() def async = false testCase.run( properties, async )
- Once the basic plumbing was in place we needed to grind out hundreds of test cases that when completed represented thousands of REST API calls and many thousands of test assertions.
- Perhaps a note on assertions is warranted. There are a multitude of ways to validate that you get the expected values in a response using SoapUI. We utilized the following methods
- Contains assertion – On the surface this is a simple text comparison, but we also used regular expressions to accept multiple values such as (?s).*ALREADY.*|.*SUCCESS.* as well as property expansions.
- Script Assertion – This was our most common assertion tool, which allowed to use the getXmlHolder object provided by SoapUI to easily query and assert conditions within a groovy script, such as..\def groovyUtils = new com.eviware.soapui.support.GroovyUtils( context ) def holder = groovyUtils.getXmlHolder(messageExchange.responseContent) def xPath = "//statusLabel[../SystemLabel='" + \ context.testCase.testSuite.properties["SystemLabel"].value.split(",")[1] \ + "' and ../statusLabel='SUCCESS']" assert holder[xPath] == "SUCCESS"
- xPath Match – This was nice tool in that I could use an xPath statement to grab a parent node, then compare expected results using both wild cards and property expansions child elements. The only gotcha that I found was that if I wasn’t very careful with my xPath statement and multiple parent blocks were returned, xPath Match would compare the first instance, which wasn’t always what I wanted. But with simpler scenarios, this was definitively a very quick and dirty way to check a bunch of response values.
So, the utility routines were pasted into assertions throughout the suite (yuck). Just as the test suite was being finalized we discovered that there was a bug in the helper routines (not zero stuffing MD5 hash values) and were faced with editing 147 instances of the helper routines in the test suite (nightmare!). Fortunately, we were able to pull the entire SoapUI project into XMLSpy and globally edit the routines. This notion of editing the project from outside SoapUI might open up some interesting productivity opportunities.
We are now moving on to using SoapUI to load test this implementation and I will share our experiences with this effort in a week or so.
