PulseUX Blog

Theory, Analysis and Reviews on UX User Experience Research and Design

HomeMissionAbout Charles L. Mauro CHFP

Critical Analysis: Why McKinsey’s “The Business Value of Design” Report Ultimately Misstates the Value of UX Design and Design Overall


By Charles L. Mauro CHFP and Paul W. Thurman DBA, MS

Recently, there has been an ongoing fascination with McKinsey’s acquisition of design agencies and how the renowned management-consulting firm would turn a profit from such investments. One would expect such an integration of high design and management consulting by a world-class firm like McKinsey to produce industry changing insights and expertise options. However, if the recently published and highly publicized McKinsey Design industry report “The Business Value of Design” (hereafter referred to as BVD) and associated McKinsey Design Index (hereafter referred to as MDI) is any indication, the grand vision may need reconsideration. It appears, based on critical analysis, that the breathless excitement presented by McKinsey Design in the BVD release video, claiming to establish for the first time “individual design actions that leaders have taken…tied back to business performance” is not only inaccurate, but unsupportable. Sadly, given the business gravitas of McKinsey and its obvious financial resources, it is clear that this a major missed opportunity for McKinsey Design, and design overall.

Important Note: Prior to publication of this review, the authors attempted to contact McKinsey Design BVD and MDI authors, but received no response. It was our intention to seek clarification and elaboration on the findings and supporting data before publication of this review. We welcome a response to this analysis by the McKinsey Design research team. The business value of design is of value to all who engage in the theory and practice of design and design science.

Analysis of the McKinsey BVD Report and MDI Index

Many Others Have Come Before: Taken from a historical perspective, the business value of design has been proffered extensively by business leaders for decades, starting with Raymond Loewy in the 1930’s, Thomas J. Watson Jr. of IBM in the 1960’s, and of course, Steve Jobs more recently.

In the last decade, there has been significant prior research focused on exploring the business value of design, including the excellent work of the Design Management Institute (DMI), which has previously presented data on increases in asset value for a core set of design-driven corporations. The McKinsey BVD Report is by no means the first data on the relationship between design and business performance. There are also the yearly reports of tech design industry trends by John Maeda and others.

Litigation Driving The Value Of Design: It is not generally well-understood by most practicing design professionals, but the recent design patent litigation environment in the US has actually generated a serious level of research focused on providing objective data on the economic value of design in successful products. The inspiration for this research was the recent Apple v. Samsung case, and the various appeals which finally reached the US Supreme Court. That case generated a robust series of amicus briefs written by collaborative teams of attorneys and leading design research professionals, focusing on helping the courts understand the value of design. The most widely read and quoted amicus brief submitted to the US Supreme Court (SCOTUS) was written by design research professionals and a leading legal team. That specific brief was signed by 113 of the world’s leading design professionals, spanning several design disciplines, including product design, architecture, fashion, and automotive. Those signing the brief included, among many others, Dieter Rams, Sir Norman Foster and Calvin Klein. The SCOTUS brief has become required reading at leading design programs and law schools in the US and abroad.

Additionally, there has been extensive funded research in Britain and other countries documenting the actual business value of design. There are numerous recent books on the topic that explore the impact of design on a wide range of business-critical applications and related consumer decision-making. Furthermore, there is an increasing field of peer-reviewed literature on the business impact and neuroscience of design. The following is what a baseline review of the BVD and MDI reveals about the data and related findings.

Correlations Are Not Causations: Why would a world-class management consulting firm like McKinsey proffer broad-ranging business impact causations, based on simple, unspecified correlational data and without supporting statistical significance values? The issue here is more than nitpicking statistical methodology. Objectively, McKinsey cannot proffer opinions that design actions measurably impact business performance using only correlation analysis. Many research studies report correlational values, and such correlations are accepted for what they are…interesting possible associations but NEVER causations. In a fundamental way, the McKinsey research team failed to properly state the obvious global limitations of the study.

Who Was Interviewed And How Representative Were They? While the BVD report indicates that some 300 companies were involved in the study, there is virtually no indication that the sample of companies were representative of the larger universe of businesses. Yet, the authors suggest that the BVD data somehow applies to all businesses, or at least the three major sectors which are the apparent focus of the study. Many industry-trend studies undertaken by top-tier management consulting firms are based on access to companies and executives that are current or past clients. However, in terms of professional research, it is well-understood that such studies are not representative of the larger universe of companies. Baseline research protocol requires that these limitations be stated explicitly in such reports.

This is a curious oversight by McKinsey in the BVD report, because at the heart of what McKinsey does in management consulting is produce extensive segmentation analysis. McKinsey is famous for these types of studies. Proffering the broad-ranging implications of the BVD research without even the most basic statements related to sampling bias and segmentation analysis would likely never pass muster on the management consulting side of McKinsey. This is actually much more than sniping at McKinsey study design. These problems go to the heart of the study overall and its claimed actionable insights.

No Meaningful Citations Or Supporting References: This problem extends across the entire report and related findings. Was this simply sloppy reporting, or a larger issue related to an actual lack of underlying and supporting data? Again, this is more than simply criticizing McKinsey for defective methodology – it goes to providing support for many of the findings in the study and the related global impact statements.

This issue is especially relevant in the section that discusses tools and methodologies for solving complex design problems. Specifically, McKinsey mentions “Design Thinking”, a methodology known to be of questionable utility without any referenced data. Was this omission evidence that McKinsey simply had no idea how ineffective Design Thinking really is? The same problem exists for the conjoint analysis, which is well-understood by design research professionals to undervalue design variables when applied to product feature priority definition. Again, the BVD failed to make available any supporting research on this second methodology. The important point is that there is a large and expanding body of peer-reviewed, science-based publications that address the underlying value of design in consumer decision-making (See Batra, Seifert, & Brei, 2016 for comprehensive review) and product development. A literature search and inclusion of appropriate references in the BVD report would have dramatically improved the quality of the BVD report and seriously tempered the overstated implications of the data. The design research examples they proffer are well-understood to be of questionable utility when applied to complex product design and development problems.

Design Actions And Their Lack Of Specificity: There is no description in the BVD report concerning how the so-called “Design Actions” were actually identified, categorized, rated, counted and/or analyzed for statistical significance. This is especially important given that the entire findings of the report rest on a corporation’s use of design actions and how they map to the MDI framework.

Obvious Design Action Insights: An interesting point about the design action items identified in the BVD report is that none of the items offer anything new or insightful in terms of corporate action-taking to improve business performance. Specifically, the four primary design actions that supposedly led companies to greater business performance were:

a. Analytical Leadership: Measure and drive design performance with the same rigor as revenues and costs.
b. User Experience: Break down internal walls between physical, digital and service design.
c. Cross-Functional Talent: Make user-centric design everyone’s responsibility, not a siloed function.
d. Continuous Iteration: De-risk development by continually listening, testing and iterating with end-users.

These so-called design actions are really nothing more that User-Centered Design (UCD) practice that has been in place for literally decades. Are these really the only four design actions that were found to be CORRELATED with business performance?

What Could McKinsey Actually Have Studied? A truly interesting and useful study would have been a deep dive into any one of the design actions versus a simple restatement of what is plainly obvious to anyone involved in the application of UCD theory and practice. The actual list of design actions in the BVD report seems more tightly aligned with how McKinsey Design might structure client engagements rather than having any real connection to the drivers of business success.

Where Is Design IP? There are significant design actions totally missing from the report that should have been considered. The most important would have been an analysis of the intellectual property (IP) portfolio of the corporations involved in the study. There is a growing body of research on the impact of design on consumers’ valuation and choice of products which relates directly to how design may be assigned to a corporations’ assets. This is an absolutely critical missing component of the BVD.

A second and even more interesting and useful study would have been an analysis of how design actions are prioritized by executives in different industry verticals and then mapping such data to the business metrics and market conditions for each specific business sector. That is of course a complex and more interesting problem, but one that McKinsey’s management consultants undertake every day (minus the design action inputs).

No Data To Support The Validity Of MDI Index: If one reads the BVD report carefully, it is clear that the research is focused on one important McKinsey Design business objective: To have potential clients fill out the MDI online survey, which McKinsey positions as an “Index” rating system of substantial power and reach. The MDI places McKinsey Design in the critical position of offering companies meaningful insights into how the proffered design actions impact their businesses based on their apparent level of design sophistication. It may seem a bit harsh, but the MDI is little more than a lead generation system for the McKinsey Design practice. Here is more on why that is the case.

Design Of A Reliable Index Is Actually Complex: It often comes as a surprise to those without a formal background in consumer research that developing and validating a reliable index is actually a complex and difficult behavioral science problem. Such indexes are common in various fields of research, including psychological assessments, decision-making research and other applications in which creating a supportable and common rating system is necessary and useful. There is a large body of literature that defines how indices of measurement are developed, tested, validated and updated to ensure that they are providing valid and reliable results (See Creswell, 2007, p. 201, for comprehensive review of validation and reliability in qualitative research). One cannot simply toss a large number of design action questions into an online survey and expect that the index will be either reliable or useful. In fact, without validation, the index may be totally wrong. There is virtually nothing in the BVD report that suggests McKinsey Design applied any modern psychometric techniques in the design of the McKinsey Design Index (MDI).

What About Basic Market Conditions And Investor Interest? A critical factor that was apparently not considered in the BVD report is the contribution of fundamental business market conditions versus the so-called impact of design actions. Missing entirely is the relationship between the overall growth in the market for the industry sectors cited, versus the so-called business impact of design actions. It is far too simple to state that the sole contribution to the increased success of corporations was the presence of design actions for the sectors in the top quartile. Wasn’t it true that the three industry sectors cited as top-performing were also those that experienced the most significant improvement in business fundamentals over the same period?

This Raises An Obvious Question: What other more general business factors were not integrated into the BVD data analysis to account for industry growth, while controlling for effects of the so-called design actions? Such factors may include the extent of stock buy-backs for the firms included in the sample, and how such buy-backs (and other corporate fiduciary actions) impacted asset improvement and growth compared to design actions. Did McKinsey really believe that all corporate earnings/growth were driven essentially by design choices alone? Did they consider the countless other factors that may have impacted business performance during the period these design actions were analyzed? Again, this is much more than nitpicking the reports methodology. These issues relate to the complexity of corporate asset appreciation and the many underlying factors…not just design actions. In this critical way, the BVD report probably misstates the actual business value of design and related design actions by oversimplifying and falsely attributing business performance solely to the value of design, rather than to potentially other well-known business factors. It is interesting to note that the McKinsey management consulting practice makes an entire industry out of helping companies deal with this exact form of complexity. This seems like a rather surprising disconnect for McKinsey overall, regardless of which group produced the BVD report.

No Supporting Information On The “Advanced Regression Analysis”: The report fails to reference or footnote this analysis and how it was utilized to inform the impact of design actions on business performance. Giving McKinsey Design the benefit of the doubt, let’s assume that such a methodology is some form of regression analysis; it is possible that the data may not have been complete at the ends of the distribution, and therefore, McKinsey may not have actually had the data available to support the hoped-for findings in a manner aligned with the marketing objectives of the study. Instead, perhaps the stated impact of design actions on business performance were projections of business impact at the top quartile based on regression analysis. This sounds like statistics-speak, but it is actually very important. Again, this is not nitpicking the McKinsey methodology, but pointing out a major structural weakness in the study. It is hard to imagine how this got past the scrutiny of the McKinsey senior partners in the management consulting portion of the business, who are known to be highly demanding on all issues related data integrity.

The BVD Report Is Understandable, But Not Supportable: The fact that McKinsey funded and published the BVD report as a thinly-veiled lead generation scheme is understandable from a business point of view. Assuming McKinsey does actually have 300 designers on staff, that is a lot of design talent looking for billable hours…and given that there is a well-understood mass transfer of design expertise to corporate in-house design groups, one can imagine that there is probably significant pressure for the acquired McKinsey Design assets to create a viable business generation framework. This is made even more complex by the fact that design services in general are rapidly becoming commoditized with significant downward pressure on fees and shorter development schedules. Going massively upmarket with a large design staff may well find McKinsey Design paddling upstream against a very strong current. By simple market inspection of design services, one can see immediately that bigger and more expensive may not be either better or even workable.

It is worth noting that management consulting firms have acquired design agencies for decades, going back to the sale of the Loewy firm and other large design firms in the 60’s and 70’s. In the majority of cases, when the first major economic downturn after the acquisition took place, the design agencies were sold back to their original partners for pennies on the dollar. Will this time be different? The answer to that question probably resides in real objective and validated integration of services not simple acquisition of expertise, which remain in narrow expertise silos.

Complicating the problem is the fact that the design services marketplace has seen a recent dramatic increase in the quantity and quality of design services offered to global clients from Asia, and specifically China. A few days in Shenzhen will convince any design executive that there is a massive new skill set forming, that will be both lower cost and faster to market than anything traditional design firms have seen before. These factors mean that design as a billable service is going to be under increasing pricing and scheduling pressures. Large design groups like McKinsey Design must find large business opportunities or lose out to a new generation of smaller, more nimble design research firms offering highly specialized expertise. This does not bode well for high-priced design services, unless they are legitimately linked to a value proposition that delivers real measurable value to increasingly demanding and design-aware clients in areas that other smaller firms cannot cover based on market penetration.

UX What? The recent rise of UX design has caused serious confusion on the part of many large clients due to a lack of explicit functional definition, and generally varied quality and wide-ranging expertise provided by individuals offering UX design services. In reality, when UX design is applied to truly complex business problems, such as those that the McKinsey Management Consulting Group is called upon to address, UX solutions may not be compelling or even executable due to a misalignment between the subjective nature of UX design and the rigor of most management consulting engagements. This leaves large clients who fund projects with firms like McKinsey Design wondering what the value of UX design is, and whether or not they can trust design agencies and management consulting firms to understand their unique business problems and assign expertise suitable for the task at hand. Over the past 5-7 years, UX design has been sold to many clients with sometimes-marginal outcomes. This is also true for Design Thinking research.

Show Me The Money: Given the well-known partner profitability targets at McKinsey, there is likely significant pressure to generate real sustainable revenue from the McKinsey Design acquisitions. Taken from that perspective, one can imagine that the BVD research and related MDI Index looked both enticing and arguably very clever. It probably seemed like a winning proposition to wrap a lead generation system in a “McKinsey” industry report. The mistake on the part of McKinsey Design was not being clear and careful about the real meaning of the data and its generalizability to specific business sectors and industry at large. This left McKinsey Design open to serious criticism for sloppy and unsupportable research, as noted above. It is important to note that McKinsey Design should be applauded for attempting to objectively link design with business performance. However, the current effort falls short on the most fundamental level.

In the end, it is well-understood by practicing design executives that selling design is hard, selling great design is even harder and execution of robust science-based design research is the most difficult of all. In this regard, there is a growing field of neuroscience-based design research that will eventually deliver on the promise of objectively defining the business value of design. Until that happens, even world-class firms like McKinsey should understand that this is a non-trivial problem that will not yield to anything less than rigorous and professionally structured research. Such research will likely require broad industry involvement, combined with leading academic research institutions and design research professionals.

Should McKinsey retract its BVD report? Read our next post, then you decide.

Charles Lee Mauro CHFP
MAURO Usability Science
Founded 1975 NYC USA

Paul W. Thurman, DBA
Lead Research Scientist
MAURO Usability Science

About MAURO Usability Science Founded in 1975, we are among the most experienced international consulting firms focused on helping world-class clients and leading startups solve business-critical problems related to the usability and interactive quality of their products and services. We have a long history of applying advanced science-based design research methodologies to the resolution of complex human/technology problems. Our clients include many leading design-driven corporations. We have received major awards from The Human Factors and Ergonomics Society, NASA, The Association for Computing Machinery, Industrial Designers Society of America and others. Mr. Mauro has served as an expert witness in over 75 major design IP cases. In short, we help make complex products simple and simple products empowering. We are proud to have solutions that are running at the heart of the world economy.

Additional Links of Interest From MUS Writings:

Post a Comment

Your email address will not be published. Required fields are marked *