Why Are GUI Testers So Bad?

I’m not sure that everyone is familiar with GUI Testers, so let me describe them to you: You have it record your interactions with a Windows application, have it check some fields for you, and you save it. Next time it validates that thing still are the same as they were before. Sounds simple, right?

Of course, that’s in a dream-world. Reality is very different.

In my last position, I was a developer with the job title “QA Engineer” and I started using TestPartner from Compuware. I found it extraordinarily difficult to use: I would have it save my actions and have it immediately play it back. But things would go wrong and it wouldn’t work. Why? I was never sure — it couldn’t find a menu, it couldn’t recognize a right-click, some little thing changed and things just blew up. I never figured out how to tame it’s wildness, but that was okay — I figured that TestPartner was just tempermental.

In my current position I’m a Java Developer and we have to write some scripts using WinRunner from Mercury/HP. Since I’ve used GUI Testers before, I was elected to head it up. I thought WinRunner had to be a lot better than TestPartner but I was wrong. Oh, in some ways it’s better, but in some ways it’s worse. The warts are a lot different, but they are still warts. And, yes, I can record an action and that action won’t play back. Different system, same problem.

I’m not going to compare them, but I will list out what I think the biggest warts these two applications have in common, besides the example I cited above:

• They don’t store their scripts in text files. One stores everything in a SQL database (!!!) and the other stores things in a directory (which has a text file called “script” but also has lots of data files, etc.) This forces you to use their editor. And forget grep and other ways to quickly find a useful code snippet.
• The documentation is horrible. It’s hard to find something useful, it’s hard to find a simple reference, and the examples are less than useless.
• Good ideas that are half-way implemented. One of the apps above signifies that you can use regular expressions to find the window names. That is a good idea, but it takes a lot of reading to find their long-winded prose explanation of regular expressions and then you find out that it’s only a small subset anyway. Example: in their regular expression syntax * will repeat the preceding expression 1-many times. Oh, and there is no + or ? characters or anything like them.
• It’s hard to have a common function library. One has you use shared modules, which are shared for all scripts within that category (again, another good idea half-way implemented). The other has you “compile” your functions, and then in your scripts you have to click the button to add it in. If you change your “compiled” functions, you have to manually remove it and add it again to each script. Or perhaps just restart. Either one is undesireable.

Maybe my problem is that I’m a developer living in testing world. The mindsets of a good tester and a good developer are similar, but by no means the same. Maybe these tools are good for someone with a testing mindset. If these were developer platforms, no one would use them. That said, I think the reality is that testing applications lag greatly behind developer applications.