Two weeks ago, George Mason University’s GovernmentContracting Center held a webinar “Measuring R&D Productivity” that sought to explore not only the concept, but how well the defense industry measured up. I found this topic particularly timely just as there are more and more questions around whether DoD is spending enough on R&D or not.
A key question I wrestle with is, “Do we need to be spending more on R&D and how would we know?” The conventional measures of R&D are all inputs — # patents, # PhDs, etc. What about outputs? What is good R&D productivity?
Ann Marie Knott who created a measure of research productivity was joined by Byron Callan of Capital Alpha Partners, a financial analyst, and Kevin Fahey former Assistant Secretary of Defense forAcquisition. We invited Anne Marie Knott to panel because the metric she created — Research Quotient — allows firms to measure their R&D productivity and compare themselves with benchmarks from within their industry sector. It’s the closest objective measure we have, at least for commercial companies. What the panel was looking to do was to explore DoD spending onR&D and whether or not more needs to be allocated, especially given the amount of spending that the tech sector has been doing.
For background, here is a quick summary of Anne Marie’s work:
Anne Marie’s RQ takes inspiration from the basic economic production function. The original economic production function is based on capital and labor. Anne Marie points to Nobel Prize winning economist Paul Romer’s work, which adds Ideas/Knowledge as a key input. According to Romers, ”ideas make material progress possible…” She took this stipulation about knowledge as an input and, being an inventor herself, looked at R&D Investment as an input and sales as an output. Specifically, she looked at sales over a 7-year period to assess a firm’s research productivity. Links at the end of this post provide further reference to RQ and her work.
Anne Marie notes that one can’t just compute RQ. One must estimate it statistically using sales over R&D spend over time. The image below shows the estimate of firm RQ using Compustat data up through 2011.
1. RQ provides an industry-specific benchmark for any company competing in the industry starting in 1965 and continuing into the future.
2. R&D spend and GDP growth both track positively upward up until the 80s, then never picks back up again. We are getting less productive despite more spending on R&D. “ What we want to do is get industry to restore their RQ so we can revive economic growth. (Economic growth is not the same as the stock market going up).
3. 63 percent of companies are currently overspending and 33 percent of companies are underspending. This means only 4 percent of companies, using the methods available to them today, have made decisions that puts them in the zone of “right spending.”
4. The RQ average result for each individual industry falls within a narrow range across all industries. So, RQ is valid only from within an industry.
5. Best practice companies in each industry have an RQ score that is four times higher than the average for that industry.
The panel unpacked these RQ principles and discussed what we could conclude about the defense sector. Anne Marie does not yet have a direct way of measuring public sector productivity. One cannot use sales or revenues as an output, unfortunately, to measure effectivity of governmentR&D spend as an input.
A few of the key insights are:
Defense sector public companies have been spending 2%-3%sales on R&D whereas other industries spend a lot more (on average 16%), according to Byron Callan. I pulled a chart from Statistica depicting these details for 10industries.
This comparison augers the need for the defense sector to spend more. Even the broader industrials sector, of which A&D sometimes is considered a component of by financial analysts, spends ~10%.
Anne Marie asserts that according to RQ for the defense sector, defense is are more productive. They are overspending less. This was confusing/thought provoking.
A few thoughts:
The way defense companies calculate and depict R&Din their financial statements is inconsistent and does not track with IR&D reimbursement. Basically, they have help from the government. This could be squelching the signal that they could be overspending.
Overspending means a firm is not efficient. It doesn’t mean it should not spend more. Or, rather, it should not spend more until the firm figures out why it is less efficient, or is not pointing its R&D towards the right markets (considering sales is used as a measure of output, and if invention and R&D is not pointed at the right market — even new markets — there may not be any sales gains).
This last point is especially interesting for the defense sector. If sales do not increase, then RQ doesn’t improve. The defense budget basically is set by the government. It is not projected to increase over the next few years. Does that mean defense sector RQ will not improve? (Not really. One could argue that spending will fluctuate, increase and decrease within the overall budget.)
The government asserts that firm IR&D spend does not align with defense priorities. In fact, according to one study, only 38% ofIR&D spend tracked with the 10 promulgated areas for defense investment, according to the GAO. So, from a public sector output perspective, 72% of IR&D created no output.
I do want to suggest that we should not falsely conclude the industry should not be spending more on R&D. At some point, sheer quantity/volume matters. The image below pulled from Recode.net lays out the figures of the top R&D spenders. Amazon alone spends $16B, far exceeding the five major defense primes combined, at least 8 times the budget of any one of our service research agencies (ONR, AFRL).
Anne Marie is right; RQ suggests the sector is doing well. But, what if we could better understand how inputs interact to form outputs — and perhaps outputs is not just to be inserted in a PoR — should the sector not invest more?
Amazon, at the top of the chart, is also one of the RQ50companies, which means it’s one of the most productive when it comes to their R&D investments. Amazon has figured out how to target its R&D spend, how to manage its R&D projects, how to assess markets/customer needs and properly re-allocate R&D funds when it needs to do so. It may be worth unpacking Amazon’s knowledge-creation system to learn from!
Here are a few other observations to keep in mind:
RQ is backwards looking, not forward looking. It doesn’t help an organization postulate what future market needs or sectors are valuable. However, one can use the measure to track productivity and a decline would signal a need to investigate further.
Speed — RQ doesn’t account for speed. At least it doesn’t directly. If that industry is moving fast, and the firm’s research productivity was not keeping up — in speed and efficiency- it’s RQ would reflect that. However, only after the fact. The firm needs to continually observe and assess its performance to not lag behind, get disrupted.
R&D expense does not capture the other tools that companies can use to access invention and innovation. These include personnel recruitment, partnering and M&A. Should one consider venture capital investments as IR&D? Byron Callan suggests that the venture model some defense companies are just starting to leverage could be a faster way to achieve results. Indeed, this could be a model worth further exploration.
Outsourced R&D is does not benefit those companies who outsource it. A firm is able to get R&D done, but that R&D will not have residual positive impacts for that firm. Since one of the inherent benefits of R&D is knowledge generation, it follows that if an organization were to outsource R&D, that organization would not have the personnel whodid the work, who gained the experience, who benefited from ideation and creative activities, in order for that organization to reap the benefits later.
The one exception I would note is the Lifescience/Healthcare sector. Biotech companies are essentially outsourcedR&D shops for pharmaceuticals. This decoupling started to happen in the80s. Pharmas recognized their research productivity is not as good as a startup's; the whole industry is now structured around this fact.
The panel attempted to answer the question of how to improveR&D productivity in the pubic sector. I fear that we only surfaced more questions. But, these are questions worth noodling over.
· We need to better understand measures ofR&D in the public sector
· We need to better understand the factors that result in good return on R&D investment — from project selection to project management.
· Spending less won’t mean productivity goes up. What it means is that we need to better understand the interaction between the mix of inputs, processes and activities and figure out what levers to pull to improve output. My hypothesis is that the process is getting in the way of proper framing of the problem R&D is to solve, and the best firms have an inherent ability to divine the future, even create markets (Amazon).
The Trillion-Dollar R&D Fix — HarvardBusiness Review
RQ consulting firm: amk Analytics