Recent Publications - The Testing Planet

The Testing Planet is a promising young publication devoted to software test, produced by The Software Testing Club. The November 2011 issue is available here.

I happen to have an article in this issue, discussing how there is such a push in the industry for unscripted “Exploratory Testing” that people seem ready to abandon test scripts altogether. I believe this is an overreaction, since scripts serve a number of useful functions that exploratory test cannot replace.

Check it out for yourself, and please let me know what you think.

Exploring Scripted Testing by Sean Morley

The market looks good for software testers... and one reason why

Two articles related to software test caught my eye this week. First, from Forbes - Who's Getting Hired Right Now. Here is an excerpt of the relevant bits:

Online job aggregator Indeed.com has sifted through its database to find the occupations that are hiring the most right now.   [skipping]

Here are the jobs with the most postings. 

Registered Nurses 

Job postings: 132,283 

Truck Drivers, Heavy and Tractor-Trailer 

Job postings: 100,917 

Software Quality Assurance Engineers and Testers 

Job postings: 83,206 

Customer Service Representatives 

Job postings: 67,958 

Sales Managers 

Job postings: 65,925

Interesting that with all of the outsourcing/offshoring we do in the software quality field there is still a strong demand. These figures sparked a very insightful discussion on The Software Testing Club forum on, among other things, how the relevant skills for the field have evolved in the last few years towards programming/automation.

The second piece that struck me was an amusing anecdote about untested software from Jeff Bezos in an article in the Wall Street Journal about the early days of Amazon.com. (Thanks to one of my favorite blogs, QA Hates You, for bringing it to my attention.)

At launch, the site wasn't even truly finished. Mr. Bezos's philosophy was to get to market quickly, in order to get a jump on the competition, and to fix problems and improve the site as people started using it. Among the early mistakes, according to Mr. Bezos: "We found that customers could order a negative quantity of books! And we would credit their credit card with the price and, I assume, wait around for them to ship the books."

Yep, don't forget to test your boundary values. Like Clint Eastwood said, "A man's got to know his limitations."

Who are your customers?

A few years and a few mergers ago, my company hired a new president who brought us all together for a big rah-rah session which included a motivational speaker. And while we all tend to be a little cynical about these things, I do feel they serve a purpose, and I genuinely try to make the most of it. Sorry, I forget the speaker's name... but the gist of his message was giving the customer more than they expected. But what if you don't work in customer service? Or don't interact with customers at all? Well, he explained that we are all consumers and producers in some way. Whatever you do in your job, there is likely some person or group that directly depends on the output or results of those actions (even if they do not directly pay you). They may very well be your colleagues.


I have done my share of work with customers. Back at the small engineering software company that I helped found, I handled much of the customer interaction. Coming from that background myself, I could understand our customers and really relate to them. I enjoyed getting to the crux of their problem and helping them find a solution. And I generally learned something along the way. But now, working as a Quality Engineer (aka software tester) in a mid-sized software company, I had much less opportunity for direct customer interaction.


Then came the epiphany (enabled by Mr. Motivational Speaker). As a software tester, my customers were the developers! QE provides a service to Dev.


If you work for a Test Contractor then you probably just issued a resounding "duh!" But maybe you view your customer as the company that retained you, or perhaps the manager you send reports to. But no, for those of us who test software, our customers are the ones to whom we report the issues, the men and women who wrote the code, who will fix the bugs. We provide them with the information gleaned from our quality assessment, the bugs we believe we have found, the behaviors we question.


We like to think that Dev and QE are on the same team, working toward the same goal. But the reality is we are usually in separate groups, answering to different managers. All too often the specter of competition casts a pall, and bugs start to seem like game points. That is why I have found this new perspective particularly helpful. By keeping in mind that my job is to provide the developers with the information they need to produce a quality product in a timely manner, that relationship stays much more positive and productive. I am more likely to preface my testing of a new feature by sitting down with the engineer who developed it.

"Is there anything I should know that wasn't in the spec? Any area of concern where I should take a particularly careful look? Hey, I have this data set - do you think it would be appropriate? Can I run these usage scenarios by you?"

If I see something unusual but not clearly wrong, I am more likely to send an email before filing a bug. If a developer has difficulty reproducing a bug I filed, I will ask what debug trace I can produce to help. Or just offer my environment. This is the type of interaction that builds trust and mutual respect. And leads to a surprisingly productive relationship.


Perhaps the best explanation is another management slogan I learned working here. The path to success is to make your customers successful. But that is another story.

Obligatory Duct Tape origins sidebar

I recently posted a blog entry on my negative experiences with duct tape, using it as an analogy for work, where the quick patch seems expeditious but in the long run leads to more rework or lower quality. But from whence doth come this ubiquitous duct tape, and how didst it earn its appellation?

Well, a little research shows that the product we now call duct tape originated during that great mother of invention, World War II. Permacell, a division of Johnson & Johnson, started with a cloth based medical tape, added a new super-sticky rubber based adhesive, and then laminated it with a coating of polyethylene. The result was a strong, flexible, durable, (almost) waterproof tape that could be torn into strips or other convenient sizes. Soldiers used it to seal ammunition boxes and make field repairs to equipment, including guns, jeeps - even aircraft. Returning GI's were enamored with the stuff and bought it from military surplus for civilian uses, including sealing ducts. Manufacturers started making it available commercially in the now familiar metallic color rather than military olive green.

It was not called duct tape during the war, however ... so how did that name enter the common vernacular? There are a couple of possible explanations, and they probably all played a factor. The most obvious derivation would be from popular use of the product to seal metallic ducts. (Standard duct tape is actually no longer acceptable for this purpose according to most state building codes; there is special purpose stuff that holds up better.)

Another common name for the stuff is duck tape. Cotton duck is a type of canvas, and duck tape is an old fashioned name for strips of material. Hence this new fabric backed tape would have fit the generic term. Now combine that with the story that soldiers referred to the stuff as duck tape due to its ability to shed water. One enterprising company (Manco) jumped on that bandwagon by creating Duck Brand Duct Tape, with a duck on the logo. They continue to manufacture a variety of forms of the iconic product today, sponsoring contests and promoting creative uses for it.

So, which came first, duct or duck? Is one name a homophonic bastardization of the other, or did they evolve in parallel? Where's MacGyver when you need him.

Real Engineers Don't Use Duct Tape

I have a love/hate relationship with duct tape. It's like that bottle of Peach Schnapps that resulted in one too many hangovers, or the hot date who borrows your car and returns it with an empty gas tank and a new dent. Enticing at the time, but you generally regret the encounter.

Wait a minute - an engineer disparaging duct tape? What would MacGyver say? You know MacGyver, that American TV show from the 80's? An agent with some nebulous secret government agency, he chose to solve problems with his wits and extensive scientific knowledge rather than with guns. Each episode featured him concocting a way out of a tricky situation using the materials at hand - which generally included his Swiss Army knife, a paper clip, and duct tape. He was a rock star among engineers; we all have a little MacGyver in us. (The Principles and Practices portion of the Professional Engineers exam really ought to have a section that begins "solve the following problem using the contents of your pockets and materials found in this room.")

So what's the problem with duct tape? I'll tell you what. Duct tape is a patch. A kludge. A temporary fix until a real repair can be accomplished. It's great for the field. You really ought to carry some in your car, or when you go camping. It might be just the thing to patch a hole in a tent or keep an auto part from falling off until you get home or to a repair shop.

What you generally can't do with duct tape, however, is actually repair something. Despite its metallic backing, cloth reinforcement, extra sticky glue, and water resistance, duct tape will eventually fray, tear, curl at the edges, slip, and leave behind sticky residue. Heck, current building code does not even allow it to be used on ducts! I can think of many times I have made a "repair" with duct tape, only to regret it later, when the "repaired" object is again failing, but now covered with unsightly peeling sticky dirty silver tape. I would have been better served to reach for the tools and supplies to make an actual repair, or replace the item, as I had to do eventually.

Consider MacGyver. When trapped in an unfriendly country with a wounded scientist, he might scrounge together a lawnmower, ladder, and parachute - with a whole lot of duct tape - to build an ultralight and fly them both across the border. But the next day do you think he says to his wife "Hey honey, let's take the kids for a ride in the ultralight, maybe stop for some ice cream." No, that thing probably fell apart when he landed!

So what's my point?

Engineers design and build; we test and fix. Sometimes we patch or hack. But it's important we make that decision - fix or patch - consciously, intelligently, and ethically. When that hack becomes part of the process, when that debugging kludge gets labeled a fix - are we trading future viability for present expediency? Are we compromising quality? Are we building products with duct tape?

So next time, think twice before you reach for that duct tape.

Just what kind of sort are you?

More fun with sorting...

In the last post I discussed how an incorrect type of sort (alphabetic vs. numeric) led to bad data. A more common concern related to sorting is performance, either from coding inefficiency of an inappropriate choice of algorithm for the application. This is a surprisingly complex subject, which I will just scrape the surface of here.

The key characteristics of a sort algorithm are:

• typical number of operations for various data sets (fully random to already sorted)
• typical amount of data access (a swap is more expensive than a compare)
• amount of memory usage (some algorithms work within the original array space, while others expand into additional, potentially large, work areas)

And so the things to consider when evaluating a sort algorithm for your application are:

• the size of your data
• whether your data is always random, or partially ordered
• your available memory

Let's consider some popular algorithms. First the simple classics.

Bubble sort

Popular in introductory programming courses, probably because it is easy to understand and very easy to code. The basic method involves continually looping through the list comparing adjacent members and swapping places if the second is ever greater than the first, until a complete loop yields no swaps.

swap = 1;

while (swap) {

    swap = 0;

    for ( i = 0; i < max; i++ )

        if ( A[i] < A[i+1] )  {

            swap( A[i], A[i+1]);

            swap = 1; }

}

Memory efficient. Speed is not so bad for a small, mostly ordered list. Bad for everything else. Performance can be estimated with O(n^2), i.e., relative to the square of the number of elements in the list. For the novice only.

Selection Sort

Here, an outer loop through the list contains an inner loop which finds the maximum value amongst the remaining members, swapping it with the value in the outer loop location.

for ( i = 0; i < max-1; i++ )

    for ( j = i+1; j < max; j++ )

        if ( A[j] > A[i] )

            swap( A[j], A[i])

Logical, consistent, memory efficient - but still O(n^2). For large data, you can do better.

Insertion Sort

Creates a new list from the original by taking members from the first and inserting them in their sorted order in the second. Simple, uses some extra memory, but not efficient for large data.

Now let's move on to the serious contenders.

Quicksort

Clever programmers love the Quick Sort algorithm, which uses a divide and conquer type approach. It's probably your fastest choice for a large random dataset, with the average case O(n*log(n)). So what's the catch? Worst case for this algorithm is a dataset that is already sorted, where performance creeps back to O(n^2).

But, hey, how often does that happen? And how bad can it be? Well, pretty often, and pretty bad. It's a common occurrence to have a small amount of new data added to a large ordered set, requiring a reshuffle to fix the order. This also brings to mind a situation I ran into some years back, working on a pretty compute-intensive circuit analysis program that implemented an NP-complete algorithm. (That means it could not be guaranteed to complete in a reasonable time.) A new analysis routine was introduced that we estimated could add maybe 10% additional processing.

Run-time doubled.

We eventually determined the reason was back-to-back sorts in the code.The new analysis required the data to be in order, so the developer began the new function with a sort - not realizing that the preceding function had modified the data and so ended with a sort to restore the order. The first sort was very fast. The second sort was more than 100X slower. Ironically, this same developer had implemented the quick sort function some months earlier recognizing that performance would be crucial given our large datasets. The fix was a single comment symbol to remove the redundant sort. Now sensitized to the issue, we went on to remove another unnecessary sort, further improving performance.


Continuing on our survey...


Merge sort
Does a recursive divide-and-conquer sort and merge. While Quicksort may be a little faster on large random lists, Merge sort has no Achilles heel. Performance is O(n log n) for average and worst case, approaching O(n) for nearly sorted lists. Does use some extra memory (2n). All-in-all, a pretty darn good choice for most applications, and is the algorithm used in the standard library sort routine for a number of modern programming languages.

Heapsort
Combines a selection sort with a type of binary tree called a heap. Competitive with Quicksort and Merge sort for speed (performance is O(n log n) for average and worst case), it is a good choice when memory is tight.

Shell sort
An evolved combination of bubble and insertion sort, it's generally quite efficient, but does have some gotchas. Easy to code, it's a good choice for many applications.


Some endnotes:

• This vulnerability to presorted data is sometimes used in Denial of Service attacks, given the popularity of Quicksort.
• Many versions (especially early ones) of the C library function qsort() exhibit this behavior as well. The function qsort() is not required to be implemented as a Quicksort, though it often is.
• Some variations to the textbook algorithm can alleviate Quicksort's vulnerability to the presorted data worstcase.

References:

There are a lot of great resources out there. A variety of sort and other algorithms are covered in detail in these books:

• Robert Sedgewick, "Algorithms in C"
• Mark Allen Weiss, "Data Structures and Algorithm Analysis"
• Tenenbaum, Langsam, Augenstein, "Data Structures Using C" 

A couple of really good overviews, with links to explore further:

• http://en.wikipedia.org/wiki/Sorting_algorithm
• http://en.wikiversity.org/wiki/Sorting_data

The next two links provide some fascinating comparative visualizations of the algorithms, while the last provides some of the best critical analysis:

• http://www.sorting-algorithms.com/
• http://people.cs.ubc.ca/~harrison/Java/sorting-demo.html
• http://www.softpanorama.org/Algorithms/sorting.shtml

Sordid sorting

Here's a quick testing tip based on something I ran into this week. If you are testing an application that uses ordinal numbers, make sure your dataset:

• crosses a decade boundary (e.g., 7, 8, 9, 10, 11, 12)
• includes a 9, or numbers that have 9 as a digit
• but the last / highest number should not include 9

Why? A common programming error is to use an ASCII sort when a numerical sort was intended. If you have the numbers 1 through 12, you want a sort to produce
{ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}

A numerical sort will give that. But an ASCII or alphabetical sort will produce
{ 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9}

If your dataset was 1 through 9, or 4203 through 4207, you would not notice a problem. In my case the data was sufficient to trigger the problem, which was especially apparent because a developer had put a sanity check in the code.

if ( current_version > highest_version )
    print ("ERROR: current_version > highest_version")

We had to track down what created the condition, which turned out to be the bad sort. But if not for the error message, who knows what odd behavior this erroneous data would have caused - and whether I would have caught it. Everybody makes mistakes; good developers work to catch them.

There is a lot that could be said about datasets for testing ... but there are a couple of interesting points on sorting, test, and performance issues that I would like to cover next time.

Man's capacity for error

In my first job at a software company, I worked with a senior engineer named Leon. A PhD in Physics, Leon was known to toss out nuggets of wisdom with the laid-back intellectualism of a college professor. Here is one of my favorites, that became more meaningful to me over the years. I am paraphrasing here, but I think it conveys the gist.

I used to be a staunch supporter of Nuclear Energy. But it wasn't until I started working in software that I came to realize man's capacity for error. Intelligent people, well-educated people, people who cared about their work - made lots of mistakes! And that includes me! So even though I understand the protocols under which nuclear material can be handled safely, I have no faith in a human's ability to reliably follow those protocols.

My message here has nothing to do with nuclear energy. The point is that when humans are involved, errors occur. We call them bugs, and our job is to find them, or reduce their occurrence. Without anger or recrimination. We work to improve the quality of the things we test. That is the service we provide. That is our mission.