Calculating and Improving ROI in Software and System Programs

Constrained by limited budgets, most enterprises find it essential to apply unprecedented business discipline to the business function of software and system delivery (SSD) across entire software and system life cycles. For this reason, the CIO, CTO, or VP of software or systems development may be under increased scrutiny from the corporate chief finance office (CFO). When conversing with the CFO, money talks, so only one of two sorts of conversations can take place: software and systems as cost center or software and systems as value-creation center. The second is more involved than the first since cost is easily measured and the value of the software and systems under development is difficult to measure. Also, the second conversation entails treating software and systems programs as investments and calculating the return on investment (ROI) along with the investment risk.

Back to Top

Most common development-program and portfolio-management practices in software and systems organizations do not support the second conversation; benefits and risk are usually based on qualitative scores, determined either through team consensus or weighted sum of scores from a questionnaire,⁵ while costs are assigned monetary value. Since quantitative and especially monetary measures are more persuasive than qualitative measures, it is no wonder that software and systems are often managed as a cost center, not as a business function contributing to overall enterprise value.

Creating enterprise value often requires innovation, but innovative programs are inherently risky, both financially and technically. Almost by definition, innovative programs begin with incomplete information, resulting in uncertainty in both expected program costs and benefits. Reasoning about, justifying, and making trade-offs among programs with different risks and value requires determining the ROI in the programs. Here, I explore how to compute the ROI, given the inherent uncertainty of innovative programs.

To manage ROI, investors typically ask two key questions:

Retrospective, for tracking progress. Has the organization earned ROI for money invested to date?; and

Prospective, for making investment choices. Is the organization likely to earn ROI for money invested today and in the future?

To answer, the funding executive must answer two even more fundamental questions: How much is my investment in any given program worth today? and How much is it likely to be worth in the future? So the analysis starts by addressing how much money an incomplete development project is worth.

I begin by proposing the “investment value,” or IV, which is correct enough to make good investment decisions yet simple enough to be useful. I describe how to use it to address the two key ROI questions, follow with a fictionalized case study based on experience applying it in the New York City Health and Human Services Connect (HHS Connect) program (http://www.niem.gov/pdf/NYC_HHS-connect.pdf), and conclude with ways to use the calculations to manage ROI of SSD.

Computing ROI of software and systems programs involves multiple disciplines:

Software economics. Reasoning about the productivity of a development organization, as discussed by, for example, Jones⁹;

Methods for estimating the costs of a development effort. McConnell¹⁸ provides a good overview; and

Portfolio and portfolio management (PPM). Cooper et. al⁵ describe it well.

This article falls within the PPM discipline, as characterized by three major approaches to reasoning about the value and ROI of SSD:

Deterministic net present value (NPV) calculations.²⁴ These calculations do not account for the uncertainty of SSD costs and benefits but do account for the cost of money; they are used to, for example, compute the value of a fixed annuity;

Software as intellectual capital asset. It applies the capital-asset pricing model (CAPM),¹⁹ including industry revenue, development, and maintenance cost models.²⁶ In practice, it extends the deterministic NPV by accounting for the uncertainty of future values by adjustments to the discount rate in the NPV equation; and

Various equity option methods, including:

• Applying option models based on the chain of decision points tied to the development life cycle^4,12,15;
• Valuing the optionality, or monetary measure of having the flexibility to choose among a set of business decisions, made available by the modularity of the code,^2,23 or reuse of the code in other projects^10,11;
• Real options methods extending the Black-Sholes equation for pricing equity options to capital investments²¹; and
• Real options methods developed by Scot Matthews of Boeing and several non-Boeing colleagues accounting for option volatility through pessimistic, nominal, and optimistic scenarios.¹⁷

IV is most similar to Matthews, though it takes a somewhat different perspective by focusing on the uncertain NPV of programs and not reasoning about the value of optionality. Also, my to-date and to-go ROI computations not found in the earlier approaches involve taking quotients of random variables using the Monte Carlo method.

IV and ROI calculations are straightforward. The validation of the IV approach rests on its consumability, or the ease stakeholders are able to do the calculations and act on the results. The calculations are supported by the Investment Analysis tool included in the IBM Rational Focal Point 6.5 product (http://bit.ly/nvXF9d). There are several ongoing IBM internal adoptions, including one involving a customer using a technical preview version, called Financier, to create 13 NPV models for HHS-Connect, including seven for City of New York agencies in the health-and-human-services domain. Joe Fleischman, a former HHS Connect Program Manager, described it like this: “Experience with Financier and the valuation methodology associated with it has resulted in the continuation or initial funding of high-value initiatives and the discontinuation of low-value initiatives.”⁷

Back to Top

IV of SSD Programs

Computing IV, the value of an incomplete development program, follows from applying two axioms:

Time. SSD costs and benefits occur over time, so their present values are found through common NPV equations; and

Future values. The future values of costs and benefits are random variables, a quantity not known precisely as a single value but described as a statistical distribution.

---

IV is correct enough to make good investment decisions yet simple enough to be useful.

---

The second axiom requires elaboration. Various methods are available for estimating the costs and benefits of development efforts, all understandably imprecise, as knowledge of the future is always incomplete. Under the covers, these methods work with statistical distributions, so estimation tools should, but often do not, return a form of distribution to quantify the uncertainty of the estimate.

Applying the axioms allows a development organization to compute the distribution of the NPV of an incomplete software or system program and precisely define versions of ROI.

While the axioms are self-evident mathematically, I begin with the adage: Things are worth what someone might pay for them. Even though there is no market for an incomplete development effort, imagine a development organization could sell its incomplete development program and how it might set a price. As an example, consider U.S.-based airplane manufacturer Boeing, which has been developing the 787 Dreamliner, its next-generation aircraft, since 2004, though as of August 2011, the planes are in test flight, and Boeing has not delivered a single aircraft to a customer, yet has orders for more than 800 aircraft (http://en.wikipedia.org/wiki/List_of_Boeing_787_orders). Suppose Boeing might want to sell the program to competitors Airbus or Embraer. It certainly would not give it away, so how would it and the buyer set a price?

The answer depends on what the buyer would be buying. The buyer would get the option to invest in completing the program and receive its benefits at delivery. Reasoning like an investor, the buyer would compute ROI, but to do so, would first have to perform several key computations:

Costs. Expected costs to complete and expected costs after delivery (such as warrantees and liability insurance);

Benefits. Expected revenue after delivery and expected revenue and cost savings from reusing the design and components on future aircraft; and

Risk. The combined uncertainty in estimated costs and benefits.

In the Boeing example, the benefits and costs are cash flows; to compute the IV of an incomplete program (such as the 787), the buyer or seller would have to apply the axioms: Starting with the first, concerning time, since costs and benefits occur over time, the responsible business manager must deal with the time value of money computed as NPV; the standard equation is:

With:

• NPV = net present value
• FV_i = future value at time period i
• r = discount rate per period

The sum is taken over payment periods of, say, monthly or quarterly and can be used to calculate the present value of committed future payments with an assumed inflation rate.

The general formula for NPV of an SSD effort is NPV(Benefits) – NPV(Development and Delivery Costs) – NPV(after delivery costs)^18,26

With:

• B_i = Benefits future values
• D_j = Development expenses future value
• M_k = Maintenance, after delivery expenses future values
• t_t = Today, the current period
• t_D = Delivery period
• t_E= End of life period
• The r_B, r_M, r_D are discount rates accounting for the time value of money

Note these sums assume that today is before the delivery date, which is, in turn, before the end-of-life date. The calculation is still valid if today is after delivery. If this were the case, there would be no development costs, and all sums would start with tt.

In practice, there could be multiple benefit and/or cost streams.

Continuing with the second axiom, concerning future values, since the future values of the costs and benefits in formula 2 are estimates, they are random variables. Hence, I define The investment value IV of an incomplete program as its NPV, with all future values being random variables.

The fact that the future values are not fixed for SSD, as they are in annuities or, say, a lottery payoff, is the key distinction between standard NPV and IV. Since the computation combines random variables, the result, the IV, is also a random variable. In practice, the IV is found through the Monte Carlo algorithm.

Following common business analytics practice,²⁰ a practical way to specify each future value is as a triangular distribution specified by high, expected, and low values. Each present value PV_n, as outlined in Figure 1, the discounted FV_n, is also a triangular distribution in which high, expected, and low values are found by dividing the FV_n specifications by (1 + r)ⁿ.

The distribution of the IV is found through Monte Carlo simulation.

The IV is calculated only from today forward; as noted earlier, it is also a random variable. In particular, the value of the incomplete programs is estimated by the mean of the IV. The investment risk is measured by the coefficient of variation (standard deviation divided by absolute value of the mean) of the IV. The coefficient of variation measures the uncertainty of receiving the mean of the IV. Alternatively, it measures the likelihood of obtaining the program value

Back to Top

Calculating IV

To calculate the IV of a specific development program, one must consider the program’s costs and benefits. The cost and benefit streams vary from program to program; for example, for a commercial software development program, they are different from those of an IT development effort. Even among different IT programs, one finds different cost and benefit streams.

Example costs are:

For a commercial product. Support for core development, manufacturing and delivery, sales, and after-delivery; and

For an IT application. Core application costs, integration of the application into the business organization (such as data migration and retraining), and after-delivery support.

Benefit streams differ by domain:

For a business. Software and systems can provide value through direct product revenue, internal efficiencies, development-cost avoidance in future products, and client retention; for example, a gambling casino or online-retail-site business analyst might be able to statistically predict the monetary yield per customer and likelihood of an IT system improving that yield or attracting more customers. The operations staff at an enterprise’s package-delivery facility might provide the estimated savings due to improved automation; and

For a government or defense agency. Systems can create benefit streams capturing how new software or systems enable it to better carry out its mission, including effectiveness and internal efficiencies; assigning monetary value to mission effectiveness is more challenging than measuring commercial benefits.

An organization can occasionally realize benefits it can quantify but to which it is unable or unwilling to assign monetary value; for example, a public agency might invest in a system intended to improve public health or safety. Assigning these benefits monetary value might entail calculating and publicly stating the value of a human life. Because stating this value could be politically unacceptable, a software or systems manager might need to include quantified, not monetary, benefits; for example, the manager might measure ROI in units of “lives saved/dollar.”

Even so, public agencies often work with economists to assign monetary value; for example, the U.S. Department of Transportation uses the “value of statistical life,” essentially the conversion factor between lives saved and U.S. dollars, for making agency investment decisions about improving safety.²⁵

Here, I describe how to create the IV model:

Step 1. Specify program dates. The model builder should specify four key program dates:

Program onset. When the development spend begins;

Program delivery. Generally when development costs end and benefits begin, with delivery maintenance and operations costs beginning near this date;

Program end. When the asset is expected to be retired and all costs and benefits end; and

Today. The date for calculating answers to the two ROI calculations mentioned earlier.

Step 2. Specify cost and benefit streams. Different programs have different sorts of costs and benefits. Here, the model builder selects the cost and benefit streams that apply to a particular program. As discussed earlier, the costs and benefits streams are different for commercial firms and for public agencies. The discount rate for each cost and benefit usually varies from stream to stream, depending on the source of money and the opportunity costs.

Step 3. Populate the cost-and-benefit streams. Note that different stakeholders have the required information and insight to provide a program’s costs and benefits; standard organization roles include:

Development manager. Estimates future value of the cost of the core development;

IT manager. Estimates the future cost of deploying and integrating the software or system;

Support organization. Estimates future value of the cost of maintenance or customer support;

Product marketing. Estimates future value of product revenue; and

Business analyst. Estimates future internal value of an IT application (such as operational savings from automation provided by the application).

With each, the data entries are a time series of estimates, perhaps quarter by quarter. As discussed earlier, the most practical way to represent it is as a time series of triangular distributions characterized by the lower bound, expected value, and upper bound.

Stakeholders may be reticent about providing data, feeling they cannot possibly know what future costs and benefits will be, a common business dilemma, as no one truly knows the future. Nonetheless, organizations reason about future values when taking business decisions; all business decisions explicitly or implicitly involve guesses about the future. Businesses succeed because stakeholders generally know something or have good intuition that could form the basis for future estimates. A good way to deal with reticence is to ask a reasonable question, like: Based on your best understanding, what are the high, low, and expected values for each data point? This approach empowers experts to provide their best understanding of future values based on the information at hand without being required to do the impossible.⁸

The projections of costs and benefits are more easily achieved as calculations, including uncertain variables; for example, revenue projection involves three sources of uncertainty—growth of the market, future market share, and product price—all random variables, and their product, also a random variable, can be found through Monte Carlo analysis. The resulting distribution can then be entered in the model by following the common practice of using the 10% point as the low value, the distribution mode as the most likely value, and the 90% point as the high value.

Step 4. Complete the IV calculation. To finish calculating a program’s IV, the model builder uses a tool to take the sum of the NPVs of all benefits and subtracts the sum of the NPVs of all costs. The NPV of each cost or benefit for each input stream equals the sum of all discounted future values. In calculating a program’s IV, each future value is a distribution, so the next step is to sum all discounted future-value distributions. Since the IV is a sum of distributions, it is computed through Monte Carlo simulation; the result is also a random variable (see Figure 2).

Note the program value and coefficient of variation calculation is valid even if “today” is after “delivery.” In this case, the software or systems manager can determine whether to continue support and improvement of a delivered application or product. The calculation involves application of IV to the discipline of application portfolio management.⁸

Back to Top

Two Investment ROI Calculations

It is now possible for the software or systems manager to answer the fundamental questions raised earlier: Has the organization received return for the money invested to date? and Is it likely to earn return for money invested today and in the future?

Tracking progress. The first question is backward looking. At the beginning of the program, program value, the mean of the IV, is likely quite low, with high investment risk. As discussed earlier, if the money is well spent, the costs to complete it are less, and the team will have a better understanding of future costs and benefits and can adjust its activities to improve program value. As part of the program-management process, the team should periodically update the cost and benefit estimates, replacing past estimates with actual reported expenditures, and use the improved understanding of the time series of future costs and benefits to update estimates of future values with, hopefully, narrower ranges. These updates result in a revised IV calculation, with associated program value and investment risk. Following this reasoning, here’s how I define the to-date ROI

Note that to-date ROI is a real number, a summary measure of the value received for money spent to date; in particular, it measures the ROI of the money spent on the program-value-improving activities described later, in the section on the implications of IV calculations for program management.

Using to-date ROI as a program measure (perhaps in addition to the more standard earned-value calculations) reinforces two behaviors many industry thought leaders consider development-management best practices: invest early in gaining information needed to improve estimated costs and benefits, and invest in well-designed reusable assets with likelihood of future value.

Investment choices. Making investment decisions involves maximizing future return of current and projected expenses, hence requiring a forward-looking ROI calculation. Making investment decisions requires an estimate of the ROI at some future date; for example, ROI of a development program at a future date involves extending the to-date formula by calculating improvement in program value from today to the specified date divided by estimated costs from today to that date. To this end, I define future ROI as:

With:

• d_f is some future date
• IV(d_f) is NPV of the program computed setting today as d_f
• Cost(d_f) is the random variable capturing the costs from today to d_f

Note, too, that future_ROI is a random variable computed through the same Monte Carlo techniques used to find the IV, reasonable since future_ROI is a forecast based on current assumptions.

As it is occasionally useful to refer to IV and derived financial calculations at planned delivery dates, I define strategic ROI as the mean of the future_ROI computed at the delivery date:

Back to Top

IV Model Example

Now consider the following fictionalized example based on a real case study: The patients of a major city’s children’s hospital are also clients of the city’s social-work agency. Both hospital and agency maintain separate case files for their clients, coordinating by periodically printing out case-file updates and sending the printouts to each other for manual input. This inefficient labor-intensive process results in case-file databases being rife with errors, with the goal of coordinating the services of their common clients rarely met. The situation becomes urgent when a local newspaper reports that several children have died from abuse, suggesting better coordination between hospital and social-work agency might have saved their lives. With federal funding, the hospital and the agency agree to jointly develop a common case management (CCM) system.

The hospital and the agency each has its own CIO, who agree to hire a system integrator to build the core system and jointly hire an IT service organization to maintain and operate the CCM. All stakeholders—CIOs, system integrator, and IT service provider—decide on the features and schedule for system development and deployment. Working with the IV model-building process, they identify the following benefits:

• Each CIO separately provides data-entry costs saved by his/her organization;
• A business-analyst consultant hired by the integrator and the organizations’ business analysts provides professional labor saved by having better data needed to handle more clients;
• The hospital’s medical-economics consultant provides emergency-room costs saved due to fewer clients visits; and
• The medical-economics consultant provides children’s lives saved (intentionally in non-monetary terms in light of public sensitivity).

And costs:

• The integrator’s project manager provides CCM platform-development costs;
• Each CIO provides integration costs for his/her organization (such as for data migration, hosting, and training); and
• The IT service organization provides maintenance and operations costs.

Each stream is provided as a separate time series of future value expressed as triangular distributions, as described earlier.

Figure 3, a screenshot from IBM Rational Focal Point 6.5, is an input stream from the core development costs provided by the system integrator. The horizontal axis marks time, and the three dots on the time grid mark the three values of the triangular distribution of the future value: the middle curve is expected value; the upper curve is high values; and the lower curve is low values. The table at the bottom of the figure includes the same quarterly values as shown in the graph.

Figure 4 is an input screen the CIO of the agency used for capturing CCM’s labor-savings benefits.

Using the input from all streams, Figure 5 shows the NPV of the CCM program at the onset of the CCM program example.

Initially, the program has low program value, only about $70,000; the uncertainty of this value is high, with a coefficient of variation around 4 (a unitless ratio with no fixed scale that can be any real number). Low value and high uncertainty is no surprise, as only limited hospital and agency money has been invested in the program. Moreover, the low initial value represents an opportunity to invest wisely and improve the IV to generate good ROI.

The decision to invest is not based solely on the current value of the program but also on the likely value at delivery. Wise stakeholders count on their team’s ability to manage their programs so the value increases and risk is reduced over time. Consider the initial value of the CCM in Figure 5; after a few months of investment, the development-cost estimates are replaced with actual reported expenditures, so estimated future costs and benefits are updated with less uncertainty. Figure 6 is the IV of the same CCM program after nine months; note that value increased and risk diminished, reflecting good ROI.

In a well-managed program, program value should increase and investment risk should decrease. Such improvement can be visualized through a version of a value/risk graph, with each point having coordinates (program value and investment risk) at some time t. Figure 7 is the chart for the CMM program. Note that each successive value in time moves from lower right to upper left, a healthy sign; the chart is also useful as part of a project-management dashboard.

Back to Top

Implications for Program Management

Program value is based on the shape and position of the distribution of the assumptions of the future values rising with changes in the following assumption distributions:

• Decreased expected value of assumed remaining costs;
• Decreased low values of assumed costs;
• Decreased high values of assumed value;
• Increased expected value of assumed benefits;
• Increased low values of assumed benefits; and
• Increased values of assumed benefits.

Conversely, program value decreases if the parameters of the cost assumptions increase and/or the parameters of the benefits assumptions decrease. Note, too, program value can change without changing the assumptions of the expected value. The program value is affected by changing the high or low value of the costs and benefits triangular distributions. Also, the standard risk is less when the range between the high and low values decreases.

This calculation provides guidance as to how to manage programs, aiming to place the investments in those activities that improve program value. A key concept is program novelty; by definition a novel program is new to an organization so might have incomplete information about what is required to receive value over time. Lack of information is reflected in the ranges of cost and benefit assumptions. The more novel the program, the greater the uncertainty and the wider the range of future value random variable assumptions.

Program value generally improves when program artifacts have matured, so there is less work to complete the effort, and assumptions on future costs and benefits are more certain, with less range. Also, program value improves when any of the three parameters of the triangular distributions increase for the benefits and any of the parameters of the costs decrease. To improve program value and reduce investment risk, the program manager should invest in activities that “mature,” adding to and refining content, thus increasing the parameters of the benefit distributions and decreasing the parameters of the cost distributions. For costs, such activities might include:

• Focusing on the riskiest requirements early to reduce uncertainty in the schedule;
• Investing in good design to lower after-delivery costs; and
• Ongoing building and testing to ensure the program meets requirements.

For benefits, they might include:

• Ongoing validation with stake-holders;
• Continual content management, ensuring the most important features are available at delivery; and
• Investing in reusable components, enabling the organization to more efficiently create new programs.

As the program team implements them, it learns more, reacts to the new information, and is able to refine its assumptions. If the program is well managed, future values are adjusted with tighter estimates. This way, program value is increased and investment risk is reduced. Note, this aspect of managing ongoing improvement of IV measures represents best practices in Agile development¹⁶ and iterative practices in the Unified Process.¹³ This NPV calculation is a mathematical justification for these best practices.

Note, too, the tension between certain cost and benefit activities; for example, some of the most valuable features may also be the ones that add the most schedule risk. The NPV calculation provides a quantitative framework for balancing these concerns. In practice, balance is achieved by building models for different content-management scenarios and comparing results.

Back to Top

IV for Managing Program Portfolios

Many managers of software portfolios use some version of the strategic-value-versus-risk calculation in Figure 8 to reason about investment decisions.

There is no generally accepted mathematical definition of strategic value or risk in software or system PPM.^3,5,6 However, using IV software or system, organizations are able to adopt a monetary version using the diagram in Figure 7, including that strategic value is program value at the delivery date and strategic risk is investment risk at delivery.

The advantage of using the delivery date is its help comparing programs at different stages of development for ongoing investment.

Note strategic value is different from the program’s current value, assuming the product will be delivered on time according to plan. Figure 9 outlines the strategic value and risk of five different programs; in it, Program 1 on page 124 is high value and low risk so seems a good choice, Program 5, on page 126, seems a bad choice, and the other choices are less clear.

Charts like the one in Figure 9 do not provide sufficient information to make investment choices.

Comparing future_ROI of each program in a portfolio allows the program-portfolio manager to decide how best to invest; for the programs in Figure 7, this comparison yields the data in the table here. With it, a development executive can make well-reasoned choices concerning allocation of limited resources. Program 3 on page 124 is a sure winner. If further assets are available, Program 2, also on page 124, is a good next choice. Programs 4 and 5 are relatively bad investments. Additionally, the development executive might want to invest limited resource to further study high-value/high-risk programs with expected low ROI, possibly suspecting the information gained from a limited investment might result in a redefined program with better strategic ROI.

Without ROI measures, there is real danger an organization will spend money far too long on a program that will never deliver value. With them, stakeholders are better able to make wise investment decisions on development efforts.

Back to Top

Conclusion

The ROI approach I’ve outlined here is based on two assumptions:

• Reasoning about the value of a development program requires dealing with its future cash flow and in turn requires computing the NPV of uncertain costs and benefits; and
• The calculation can be performed using triangular distributions as input and Monte Carlo analysis for computation, an approach that is standard fare in modern business analytics.

IV and ROI calculations follow logically.

The IV approach has inherent advantages:

Monetary value. Currency provides a common scale for comparing and weighing trade-offs;

Collaboration. The calculation integrates the separate concerns of the cost and benefit stakeholders;

Financial. IV is the basis for computing financial ratios (such as ROI and return on assets);

Risk measurement. Allowing ranges in inputs frees stakeholders from having to commit to low-confidence assumptions. A range—the difference between high and low values—indicates the uncertainty of the person entering the data. The standard deviation in the output measures the risk inherent in the uncertainty of an input;

Accountability. By tracking the history of the assumptions and their basis and comparing them against actual reported costs and benefits, the organization improves its ability to make and meet commitments. Hubbard⁸ discussed calibrating the organization to improve its ability to define triangular distribution assumptions;

Continuity. Conventional wisdom, based on a common interpretation of accounting standards holds that an ongoing program has no value until it ships, creating discontinuity of program value at that time. This discontinuity inhibits ongoing value management. If everything is worthless, how does the responsible business manager set priorities? The ongoing calculation of the program value removes the discontinuity (see Figure 10). As discussed here, the continuity in the program-value measure supports more effective program management.

---

If everything is worthless, how does the responsible business manager set priorities?

---

I’ve described the elements of IV and derived ROI calculations, though other more-advanced techniques can be added, including:

Asset management. Extending the IV calculation to include continued investment in delivered assets, including an additional retrospective ROI measure (such as to-date operational benefits);

Resource optimization. Providing input to a stochastic optimization tool, enabling optimized investment choices in software and system programs, given a fixed budget and/or fixed number of staff members of varying capability;

Portfolio optimization. Reasoning about the best way to calculate the equivalent of the efficient frontier for program portfolio, a common calculation in financial-investment portfolio management²²;

Feature management. Determining the marginal monetary value of product or system features for decision support in content management; the answer may lie in the conjoint analysis techniques in market research¹⁴;

Optionality. Extending the model to include investment decision points when management might want to kill the program to cut its losses; and

Option creation. Modeling the value of reusable assets for inclusion as a benefit stream.

Back to Top

Acknowledgments

This work evolved from conversations with and the articles of Scot Matthews of Boeing Corp., Seattle. Thanks, too, to IBM colleagues Clay Williams, Arthur Ryman, Jim Densmore, Peri Tarr, and Paul Matchen for reviews of early drafts. Also thanks to Daniel Sabbah and Martin Nally of IBM, Carliss Baldwin of Harvard Business School, and Cliff Berg, a consultant to the U.S. Forest Service for discussing development of these concepts.

Figures 2–7 and 9 are screenshots of the Investment Analyzer feature in IBM Rational FocalPoint, implementing the concepts described here. IBM Research staff in Hawthorne, NY, primarily Jacqueline Martino, Paul Matchen, and Peri Tarr, developed the extension.

Thanks also to Kamal Bherwani, the former CIO of New York City Health and Human Services Connect (NYC HHS-Connect), now chief digital officer and managing director of Prisa Digital, Madrid, Spain, and Joe Fleischman, formerly program manager of the HHS-Connect initiative, now PMO director of Prisa Digital. Much of the functionality and user interface realizing the IV and ROI approach described here resulted from lessons learned in the HHS-Connect pilot.

And thanks also to the reviewers whose comments, questions, and suggestions were instrumental in shaping the article.

Back to Top

Back to Top

Back to Top

Back to Top

Back to top