Dr. Bill Curtis is Senior Vice President and Chief Scientist at CAST, a leader in providing technology for software measurement and analysis. He heads CAST Research Labs and publishes biennial reports on the global state of software structural quality. He is also the Director of the Consortium for IT Software Quality sponsored by Software Engineering Institute (SEI) at Carnegie Mellon University and the Object Management Group. While he was the Director of the Software Process Program at SEI he led the development of the Capability Maturity Model (CMM) and the People Capability Maturity Model. The CMM and its successor, CMMI, have become the world's leading standards for evaluating the capability of a software development organization. After leaving the SEI he was Co-founder and Chief Scientist of TeraQuest in Austin, Texas, the global leader in providing CMM-based services, which was acquired by Borland in 2005.
“The entire value chain for delivering and operating high quality software systems must be integrated, measured, and continually improved”.
Prior to joining the SEI, Dr. Curtis directed research on advanced user interface technologies and the software design process at MCC, developed a global software productivity and quality measurement system at ITT’s Programming Technology Center, evaluated software development methods and metrics in GE Space Division, and taught statistics at the University of Washington. He has published four books, over 150 articles, and has been on the editorial boards of numerous journals. In 2007 Dr. Curtis was elected a Fellow of the Institute of Electrical and Electronics Engineers for his contributions to software process improvement and measurement.
Q: What have been the major drivers of software quality? What developments you are seeing in the software quality space?
A: The biggest current drivers of software quality are the risks associated with operational incidents and the costs associated with maintenance and enhancement. We are in the era of nine-digit defects—not bits or bytes, but rather dollars or euros. Software is rapidly becoming one of the largest risks in most businesses. The over USD 200 million security breach at Target Stores cost the CIO and CEO their jobs. When a faulty upgrade activated a bad algorithm lodged in dead code triggering USD 440 million of bad trades in 30 minutes, Knight Trading was bankrupt. When the damages are in nine digits, software quality becomes a boardroom issue.
In the 1990s and early 2000s the focus was on improving software development and maintenance processes. For organizations that have implemented mature processes, the focus is now on the architecture and structural integrity of the software product. The software industry has gotten pretty good at functional testing. However, analysis of the non-functional aspects of the software, the engineering and structural aspects, has lagged behind. Most of the software disasters that make headlines result from structural flaws rather than functional defects. Now the focus is very much on these architectural and structural aspects because of their impact on risk and cost. Poorly conceived architectures are the mothers of legacy code. We find that the costliest defects involve faulty interactions among components spread across the system.
The pace of industrial change and competition has exacerbated the ways software can harm a business. Here are three examples. First, poorly architected, excessively complex, sclerotic software severely impedes business agility because of the lengthy duration required for enhancements. Second, badly designed software that swallows resources and fails to defend its perimeters can be costly and risky in the Cloud. Finally, in a system of systems where component systems are produced by different companies such as in the Internet of Things, quality must be managed through the entire supply chain to avoid propagating defective behaviors across the system.
Q:How do you think the issue of software quality will be viewed in the Digital era? What will be its relevance as the world goes Digital?
A: Over two decades ago a former CIO of Citigroup described his company as a software development house masquerading as a bank. Decades later this is becoming uncomfortably true for companies in most industries. Where previously software applications mostly managed the data of business transactions, digitalization of the business now puts the entire business process, and therefore the functioning of the business at risk. We are already in the era of 9-digit defects, and the share of corporate risk situated in software is growing.
As digitalization of businesses accelerates, corporations may wonder if software development and enhancement should be treated as core competencies. If digitalization becomes a core competency then customers will increasingly re-absorb software that executes the business process, shifting more work from outsourcers to captive centers. As labor arbitrage wanes, captives could be increasingly co-located with business centers to expedite the interaction between business owners and business digitizers. In this scenario, retaining captives implies capturing the business center in addition to its IT support center.
Further, there is also growing concern over the role of software in the Internet of Things. The frightening demonstration of a hacker breaching and taking control of a car is only the beginning of risks associated with products interacting with or controlled by other systems through the Internet. A vulnerability in any of the interconnected systems puts the entire chain at risk. Home automation, medical devices, and inflight avionics are only a few examples of interconnected systems in the Internet of Things that elevate safety, security, and reliability to the top of the software quality hierarchy. Because of these risks, we will see a growing demand for certification of software systems, especially by executives who want to demonstrate due diligence in governing their application portfolio.
Q: What in your view will be the next focus of standards after CMM and CMMI?
A:In the process space, the CMMI Institute is developing what it calls ‘Next Generation CMMI’ for release in the near future. The SPICE world is transitioning from ISO 15504 to the new ISO 33000 series, which is primarily a continuous or process-based maturity framework rather than a staged or organizationally-based framework. Needless to say I recommend the staged framework because it gave the original CMM its unique power to create improvement. That is, rather than focus on improving individual processes in isolation, staged approaches focus on transforming the organization by improving an integrated set of processes that enable new levels of organizational capability. It is a unique approach to organizational development. The days of large bloated process conformance models is over. Future models need to embody lean principles to focus on practices that enable improved outcomes.
In the software quality space ISO is transitioning from the old ISO 9126 to the new ISO 25000 series of standards. In particular, ISO 25010 defines the various quality characteristics, or ‘-ilities’, and ISO 25023 defines the actual measures of these quality characteristics. However, ISO 25023 does not define these measures at a level where they can be quantified from the source code. Consequently, the Consortium for IT Software Quality (CISQ) was formed to develop specifications for automatable measures of software size and quality characteristics. CISQ standards for Automated Function Points, Automated Enhancement Points, Reliability, Security, Performance Efficiency, and Maintainability have now been approved as international standards by the Object Management Group (OMG). Since they are defined at the source code level, these OMG/CISQ standards supplement ISO 25023, and some will be submitted for consideration to ISO.
There are many other ISO and industry specific standards related to security, safety, and other quality issues, and more are being developed as the amount of software grows exponentially in modern products and digital organizations. Some of these standards come from regulatory agencies in government while others developed by industry trade associations. The development of software standards will continue as the operational risks of software grow. For instance, what bounds do we put on the decisions a class of autonomous systems is allowed to make without human intervention.
Q: How can Indian companies continue to achieve software process improvement and make it measurable? What is the path forward?
A: First, stop focusing solely on compliance to process standards unless it is absolutely required to win a contract. For example, the bloat of practices and the bureaucracy of documenting their performance for CMMI appraisals has in many cases harmed process improvement. The primary focus should not be on compliance, but rather on creating a lean mindset for improvement. This focus was lost in assessments that required every practice to be documented even if it added little to no value.
Organizations should be constantly evaluating all processes to see if they belong in the end-to-end software value chain and identify improvements that will produce higher quality outcomes. Appraisals must measure project quality outcomes to determine if the practices are achieving their intended objectives. With a strong focus on measurement, companies can provide an evidence-based case demonstrating their improvements in quality along with reductions in cost and time to delivery. Integrated process and quality measures across the development-to-delivery value chain provide the strongest evidence.
Most companies are now adopting some form of ‘agile methods’. Do not overbuy the agile religion. Rather evaluate practices from the various agile methods for each project, then select and adjust them as necessary to deliver the target quality. Data in our most recent CRASH report at CAST revealed that hybrid methods incorporating upfront architectural analysis followed by feedback in short delivery cycles resulted in the highest structural quality scores. Especially for large or mission-critical systems, focusing improvement efforts on early architecting with feedback from fast code development cycles would appear to be the best process improvement target.
When implementing DevOps, the process value chain targeted for improvement is even larger. Early reports from companies that are implementing disciplined agile/DevOps processes are showing 50% improvements in productivity and dramatic reductions in injected defects as developers learn more about their previous mistakes. Properly instrumented, the software value chain can become an extraordinary learning environment for developers who will then embrace rather than resist adopting new practices.
Q: Quality compliance has often proved to be a challenge for smaller players within India’s software and services industry. What would you say to these companies? How important is it for India’s current ever-growing pool of software product developers to focus on quality?
A: Regardless of their size, companies that produce unreliable and insecure software will not survive unless they have no competitors. Even in start-up mode, a poor architecture and complex code can be serious impediments to growth because of the limits they put on product scalability and flexibility. Too many companies get passed by competitors when they slow the development of new capabilities to re-architect the system or fix a flood of defects. While innovators and early adopters may tolerate problems, the early majority in customer environments will not. A reputation for frequent incidents or inability to scale will increase the probability of plunging into the chasm rather than crossing it.
Small companies are often providing components into a larger integrated system. Concern is growing over validating the capability of each member of the supply chain, and of ensuring their code does not contain flaws that compromise the rest of the system. This trend will increase with the digitalization of the business and the Internet of Things.
Q: Is software analytics the next big thing?
A: It depends on what you mean by software analytics? Let me define software analytics as the collection and analysis of data related to the size, quality, and operational performance of a software system. To engage software analytics successfully you will need a statistician who knows software to reliably separate signals from the noise. You will also need to train managers and executives in the interpretation and use of software data, as well as practices to avoid. For instance, using data from analytics in performance appraisals will cause developers to cleverly undermine the analytics program. However, when used intelligently and focused on continual improvement, analytics can be the primary guide for improvement as well as aiding governance.
Visualize all the measures taken along a product line in a manufacturing environment, especially one practicing lean or six sigma. Why don’t software folks do the same in development and maintenance? CMMI Level 4/5 organizations are supposed to be doing this, but too many just develop control charts on a few processes such as inspections to comply with required CMMI practices. Consequently, they do not produce the level of quality software customers expect from a high maturity organization. Their objective should be end-to-end lifecycle measurement and prediction of product quality through all the iterations leading to a delivered product. High maturity organizations should deliver predictable quality.
Software development organizations need to take an analytical approach and instrument development and maintenance environments with analytics that aid developers in learning from their performance. Summary measures can provide managers and executives with the insights they need to ensure their products do not exceed the level of risk customers can tolerate, or exceed the costs that that meet business objectives. By the way, measurement is not an activity separate from engineering, as it is often portrayed in software. Measurement is engineering and should not be separate from development activities. Measures should be automated into each development activity where they are relevant.
Both individual development environment (IDEs) as well as application lifecycle management (ALM) tools need to have analytic and measurement capabilities. For instance, we are finding that if developers receive feedback on structural flaws from analysis of an integrated code base at least once during a sprint, they inject fewer defects in future releases. The feedback on system level flaws teaches developers about unexpected and incompatible system-level interactions. This learning motivates them and reduces resistance to measurement.
Managers and executives need deeper insight into their software products and systems than in the past. They need to anticipate the level of risk to which they are exposing customers or the business. In addition, they need reliable indicators of cost of ownership over future releases. Executives in every other part of a business have these analytics and predictive measures. Unfortunately, in some organizations, software continues to operate as a craft rather than a business. Software development and maintenance can only be fully industrialized if we are willing insert analytic capabilities across the lifecycle.
Q: What would you say has been the contribution of organizations such as CAST in promoting the concept of quality across global organizations?
A: CAST’s primary contribution has been in emphasizing the non-functional, structural, engineering quality of a software system. Within this space, CAST’s unique contribution has been the analysis of system-level violations of good architectural and coding practices. Modern business applications are typically composed of multiple layers written in multiple languages sitting on multiple platforms. Although APIs alleviate some of the problem, we have reached the stage where no developer or team can know all of the languages or cross-system interactions. Automated analysis must now carry much of the load for analyzing and ensuring system-wide quality.
Agile methods have eliminated formal inspections of designs and code, a practice which has been shown empirically to be one of the most effective for detecting system-level defects. CAST’s build-time technology can replace much of the visibility into system-level flaws that were formerly found in inspections. This automation is a critical support for developers on short release cycles who have precious little time to investigate flaws. As we increasingly industrialize the software value chain, system-level technologies like CAST are critical for augmenting quality practices with tools whose capabilities are beyond those of developers.
Q: While quality was once a key differentiator for Indian software development and services companies, today it is really just a hygiene issue. Is this a good thing or bad thing?
A: Software quality as a hygiene factor is a good thing. It means that achieving the required level of quality is a bare minimum customers expect from their suppliers. However, I am not convinced quality has achieved true hygienic status. To do so, high quality software would be the norm today, and it isn’t. I see data on the structural quality of thousands of applications and it is very disappointing. The industry has far to go before considering itself hygienic. With all the effort put into better processes and quality assurance practices, why are the newspapers still filled with stories about outages, data breaches, degraded performance, and failed upgrades?
A reputation for producing dependable, trustworthy software can still be a differentiator, but results have to be consistent across the organization. The quality of a software system is dependent on both the quality of its processes and the quality of its developers, and we still have issues with both. First, too many firms are now more into complying with process standards such as CMMI than they are with actually implementing an end-to-end software process that consistently yields high quality software on schedule and budget. The latter will almost always achieve the former, but experience has shown that the reverse is not necessarily true.
Second, the demand for software developers is so great that many companies, regardless of their locations on the globe, are having to accept second and third tier talent. We are seeing the rise of computer training companies that claim to produce developers in just a few months of instruction. Even worse, many developers admit they are self-taught. Therefore, one of the biggest challenges and potential differentiators for Indian companies will be in sustaining consistent professional capability across the workforce as demand grows and technologies evolve. Managing the talent shortfall requires continual training and mentoring, matching assignments to capability, having a process and measures that catch mistakes early, and developing deep domain knowledge.
Many customer organizations are re-absorbing their outsourced applications to better manage their quality and operational risk. They have learned that the cost arbitrage achieved through outsourcing can disappear when the costs of poor quality are factored in. As the compensation of Indian software developers rises toward eventual parity with global salary levels, cost arbitrage will disappear. In that future, Indian outsourcers will have to compete on greater speed and quality of delivered software enabled by deep knowledge of the business domain.
Q: What are the kind of skill sets that organizations need today to give quality an important place in their agendas and remain quality-driven?
A: A commitment to engineering discipline is critical. It is not ‘software engineering’ unless the engineering part is taken seriously. That means the entire value chain for delivering and operating high quality software systems must be integrated, measured, and continually improved. It also means that if developers want to claim they are ‘artists’, then they need to start behaving like artists. The great Italian sculptor Bernini would model his design long before committing it to marble, often iterating through numerous prototypes in terra cotta to perfect the details. Great artists have a rigorous process and are disciplined in their work. Dali and Picasso spent years imitating the styles of other great artists before developing their own unique style. They mastered the art of painting before becoming creative geniuses. Yet the proper mind-set for software development more often comes from the discipline of engineering, so that should be a foundational component of the software skill set. Although software is an intellectual rather than a physical product, it still requires a strict discipline of logical analysis and attention to syntactic detail.
Notice I have not listed specific skills such as Python or micro-services or cognitive computing because specific skill requirements will continually evolve. Changes in languages, technologies, platforms, Open Source offerings, and the like are coming at a mind-numbing pace, a pace that guarantees early obsolescence to anyone who does not keep up. You cannot do quality work in a medium you do not understand. We still find that the greatest source of variation in the quality of components is in the skills of the individuals who developed them. Consequently, continual learning should be another critical skill both for the organization that provides the opportunities and for the developers who should rush to take advantage of them. Regardless of whether we acknowledge it, there is a period of apprenticeship for learning all the technologies and the domain specifics required to build high quality systems. With the pace of technological change, in some regards this apprenticeship never ends.
For organizations with growing software staffs, knowledge capture and deployment is a key competitive skill. Many quality problems result from a weak understanding of the customer’s business domain. Computer science students rarely graduate with a deep knowledge of domains like banking, avionics, or telephony. Much of this is learned on the job. Capturing domain knowledge from engagements and codifying it in a form easily navigated and understood by developers will provide a competitive advantage in dominating an industry or technology segment. Organizations with deep domain knowledge and the ability to deploy it rapidly across development teams will have created a substantial barrier to entry for competitors.
We have always known that the ability to work in teams is important. However, the growth in social connectivity within a workforce is expanding the boundaries of what teamwork implies. The ability of social media to rapidly spread a request for a solution to a problem or a specific component that can be reused creates a sense of one team across an organization of many projects. The development staff become more united as a team rather than existing as small tribes interacting only within project silos.
Finally, management, and especially executive management must be skilled in the practices that produce high quality software and must not sacrifice them to the latest unreasonable request. They must be good at team, project, and organizational development as they move up the ranks. Their advancement must be rewarded primarily for their ability to grow business through building projects and organizations that customers have confidence in for delivering high quality systems on budget.
Q: How can Indian software companies achieve and sustain leadership in software?
A: India’s reputation as a high quality supplier of software is under pressure as customers have trouble with delivered or maintained systems. Although it was 25 years ago when I led development of the original CMM and haven’t done an appraisal in over a decade, I still have CIOs bark at me about their problems with software supplied by CMMI Level 5 companies. While CMMI does not guarantee defect-free code, there are nevertheless some deeper issues involved such as the compliance-versus-improvement problem and the availability of talent. So how does India sustain leadership?
The countries that do the best job of producing talent and instilling in them the basics of engineering discipline will be leaders in meeting the world’s need for software intensive systems. This has to start early in the schools where excellence in mathematics, analytical reasoning, and communication skills provide the foundation. Investments in improving early education are investments in the future of India. Many Indian companies already work with the universities to provide feedback on the capabilities needed and the knowledge and skill shortfalls they find in the graduates they hire. Expand these collaborations and emphasize that the curriculum must instil engineering discipline and a commitment to technical excellence. Students should also be exposed to the array of current and future software and system architectures and technologies. While much of this knowledge is gained on the job, the better prepared the graduates, the shorter and less defect-ridden the learning curve.
Finally, national leadership will depend on the discipline demanded by executives. Indian software executives have an excellent reputation in this regard and names such as Kohli, Murthy, and Premji are revered far beyond India’s borders. I once heard Kohli ask one of his executives what he was doing about public education in India, and then remind him “That is your supply chain!” Continuing to develop executives who accept as their first priority creating great organizations that sustain superior performance is the key to attaining and sustaining leadership. Organizations will outlive their leaders, so national leadership will ultimately depend on the quality of India’s organizations.