You have probably heard about model-based testing (MBT), but like many software-engineering professionals who have not used MBT, you might be curious about others’ experience with this test-design method.
From mid-June 2014 to early August 2014, we conducted a survey to learn how MBT users view its efficiency and effectiveness. The 2014 MBT User Survey, a follow-up to a similar 2012 survey (http://robertvbinder.com/realusers-of-model-based-testing/), was open to all those who have evaluated or used any MBT approach. Its 32 questions included some from a survey distributed at the 2013 User Conference on Advanced Automated Testing. Some questions focused on the efficiency and effectiveness of MBT, providing the figures that managers are most interested in. Other questions were more technical and sought to validate a common MBT classification scheme. As there are many significantly different MBT approaches, beginners are often confused. A common classification scheme could help users understand both the general diversity and specific approaches.
The 2014 survey provides a realistic picture of the current state of MBT practice. This article presents some highlights of the survey findings. The complete results are available at http://model-based-testing.info/2014/12/09/2014-mbt-user-survey-results/.
Survey Respondents
A large number of people received the 2014 MBT User Survey call for participation, both in Europe and North America. Additionally, it was posted with various social-networking groups and software-testing forums. Several tool providers helped distribute the call. Last but not least, European Telecommunications Standards Institute (ETSI) supported the initiative by informing all participants at the User Conference on Advanced Automated Testing.
Exactly 100 MBT practitioners responded by the closing date. Not all participants answered every question; the number of answers is indicated if considerably below 100. Percentages for these partial response sets are based on the actual number of responses for a particular question.
The large majority of the respondents (86%) were from businesses. The remaining 14% represented research, government, and nonprofit organizations. The organizations ranged in size from three to 300,000 employees (Figure 1).
As shown in Figure 2, almost half of the respondents had moved from evaluation and pilot to rollout or generalized use. On average, respondents had three years of experience with MBT. In fact, the answers ranged from zero (meaning “just started”) to 34 years.
To get an impression of how important MBT is with respect to other test-design techniques, the survey asked for the percentage of total testing effort spent on MBT, hand-coded test automation, and manual test design. Each of the three test-design methods represented approximately one-third of the total testing effort. Thus, MBT is not a marginal phenomenon. For those who use it, its importance is comparable to other kinds of test automation.
Nearly 40% of the respondents came from the embedded domain. Enterprise IT accounted for another 30%, and Web applications for approximately 20%. Other application domains for the system under test were software infrastructure, communications, gaming, and even education. The exact distribution is given in Figure 3.
The role of external MBT consultants turned out to be less influential than expected. Although we initially speculated that MBT is driven mainly from the outside, the survey showed a different picture. A majority (60%) of those who answered the survey were in-house MBT professionals. Only 18% of respondents were external MBT consultants, and 22% were researchers in the MBT application area (Figure 4).
Does MBT Work as Expected?
In our projects, we observed the expectations regarding MBT are usually very, if not extremely, high. The MBT user survey asked whether the expectations in five different categories are being met: testing is becoming cheaper; testing is better; the models are helping manage the complexity of the system with regard to testing; the test design is starting earlier; and, finally, models are improving communication among stakeholders.
So, we asked: “Does MBT fulfill expectations?” Figure 5 shows the number of responses and the degree of satisfaction for each of the five categories. The answers reflect a slight disenchantment. MBT does not completely fulfill the extremely high expectations in terms of cost reduction, quality improvement, and complexity, but, still, more than half of the respondents were partially or completely satisfied (indicated by the green bars in Figure 5). For the two remaining categories, MBT even exceeds expectations. Using models for testing purposes definitely improves communication among stakeholders and helps initiate test design earlier.
Overall, the respondents viewed MBT as a useful technology: 64% found it moderately or extremely effective, whereas only 13% rated the method as ineffective (Figure 6). More than 70% of the respondents stated it is very likely or extremely likely they will continue with the method. Only one respondent out of 73 rejected this idea. Participants self-selected, however, so we cannot exclude a slight bias of positive attitude toward MBT.
To obtain more quantitative figures on effectiveness and efficiency, the survey asked three rather challenging questions:
- To what degree does MBT reduce or increase the number of escaped bugs—that is, the number of bugs nobody would have found before delivery?
- To what degree does MBT reduce or increase testing costs?
- To what degree does MBT reduce or increase testing duration?
Obviously, those questions were difficult to answer, and it was impossible to deduce statistically relevant information from the 21 answers obtained in the survey. Only one respondent clearly answered in the negative regarding the number of escaped bugs. All others provided positive figures, and two answers were illogical.
On the cost side, the survey asked respondents how many hours it took to become a proficient MBT user. The answers varied from zero to 2,000 hours of skill development, with a median of two work weeks (80 hours).
What Kind of Testing is Model-Based?
Model-based testing is used at all stages of software development, most often for integration and system testing (Figure 7). Half of the survey respondents reported model-based integration testing, and about three-quarters reported conducting model-based system testing. Nearly all respondents used models for functional testing. Performance, usability, and security testing played a subordinate role with less than 20% each. Only one participant found it difficult to fit MBT into an agile approach, while 44% reported using MBT in an agile development process, with 25% at an advanced stage (rollout or generalized use).
There was a clear preference regarding model focus: 59% of the MBT models (any model used for testing purposes is referred to here as an MBT model) used by survey respondents focused on behavioral aspects only, while 35% had both structural and behavioral aspects. Purely statistical models played a minor role (6%). The trend was even more pronounced for the notation type. Graphical notations prevailed with 81%; only 14% were purely textual MBT models—that is, they used no graphical elements. All kinds of combinations were used, however. One respondent put it very clearly: “We use more than one model per system under test.”
Test modeling independent of design modeling. Note that more than 40% of the respondents did not use modeling in other development phases. Only eight participants stated they reuse models from analysis or design without any modification. Figure 8 shows the varying degrees of redundancy. Twelve participants stated they wrote completely different models for testing purposes. The others more or less adapted the existing models to testing needs. This definitely showed that MBT may be used even when other development aspects are not model-based. This result was contrary to the oft-voiced opinion that model-based testing can be used only when modeling is also used for requirements and design.
Model-based testing compatible with manual testing. The 2014 MBT User Survey also showed that automated test execution is not the only way that model-based tests are applied. When asked about generated artifacts, the large majority mentioned test scripts for automated test execution, but more than half of the respondents also generated test cases for manual execution from MBT models (see Figure 9). One-third of the respondents executed their test cases manually. Further, artifact generation did not even have to be tool-based: 12% obtained the test cases manually from the model; 36% at least partly used a tool; and 53% have established a fully automated test-case generation process.
Conclusion
Some 100 participants shared their experience in the 2014 MBT User Survey and provided valuable input for this analysis. Although the survey was not broadly representative, it provided a profile of active MBT usage over a wide range of environments and organizations. For these users, MBT has had positive effects on efficiency and effectiveness, even if it only partially fulfills some extremely high expectations. The large majority said they intend to continue using models for testing purposes.
Regarding the common classification scheme, the responses confirmed the diversity of MBT approaches. No two answers were alike. This could put an end to the discussion of whether an MBT model may be classified as a system model, test model, or environment model. It cannot. Any model used for testing purposes is an MBT model. Usually, it focuses on all three aspects in varying degrees. Classifying this aspect appears to be an idle task. Some of the technical questions did not render useful information. Apparently, the notion of “degree of abstraction” of an MBT model is too abstract in itself. It seems to be difficult to classify an MBT model as either “very abstract” or “very detailed.”
The work is not over. We are still searching for correlations and trends. If you have specific questions or ideas regarding MBT in general and the survey in particular, please contact us.
Related articles
on queue.acm.org
Managing Contention for Shared Resources on Multicore Processors
Alexandra Fedorova, Sergey Blagodurov, and Sergey Zhuravlev
http://queue.acm.org/detail.cfm?id=1709862
Microsoft’s Protocol Documentation Program: Interoperability Testing at Scale
A Discussion with Nico Kicillof, Wolfgang Grieskamp and Bob Binder
http://queue.acm.org/detail.cfm?id=1996412
FPGA Programming for the Masses
David F. Bacon, Rodric Rabbah, Sunil Shukla
http://queue.acm.org/detail.cfm?id=2443836
Figures
Figure 1. Survey participants come from organizations of all sizes.
Figure 2. 48% of the respondents routinely use MBT, 52% are still in the evaluation or trial phase.
Figure 3. Various application domains represented.
Figure 4. Three in five respondents are in-house MBT professionals.
Figure 5. Comparison between expectations and observed satisfaction level.
Figure 6. Nearly all participants rate MBT as being effective (to different degrees).
Figure 7. MBT plays an important role at all test levels.
Figure 8. Reusing models from other development phases has its limits.
Figure 9. MBT is more than test-case generation for automated execution.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment