Will big data solve the innovation gap?

For many companies, I think there is a relatively significant gap between what they actually generate as compelling new products and what they wish they generated.  We could call this the "innovation gap".  The gap is real, and it means that many companies aren't as profitable or as competitive as they'd like to be.  Many of have (in some cases for years or decades) advocated tools and methods as a way to improve the innovation funnel, to create more innovation more readily that had value as new products and services.  To date, there's been some improvement but the innovation gap still remains.


Lately, with the advent of "big data", machine learning and other factors associated with data and more intelligent processes, the argument has been made that these capabilities will solve the innovation gap.  This claim seems to suggest that big data and analytics and machine learning can do a better job in the front end generating new ideas that lead more rapidly to new products and services.  And at some level I agree, but I think placing too much emphasis on big data or machine learning for all of your innovation work is a mistake.


What big data and machine learning could do


Basically, the 'front end' of innovation is an exploration and discovery activity, meant to discover needs and opportunities and assess customer needs in order to generate new ideas that may or may not solve the problem.  Good innovation is both a discovery and a combinatorial effort, which in some regards means that machines and algorithms must be able to parse through more potential combinations than humans can.


The challenge with this thinking is that in many cases, humans create the rules by which the algorithms work, and if humans are often blinkered to new ideas or emerging technologies or unusual combinations, then the algorithms may be as well.  Further, having an algorithm spit out hundreds of potential combinations without the ability to assess their viability or value seems rather meaningless and complicated.  Beauty and efficacy is often in the eye of the beholder.  I never knew I needed or wanted a multi-tool until I got one for Christmas.  And even today, while it's breadth of tools is undeniable, it often sits in my drawer at home, as I suspect many of them do.


There is definitely a place for big data, analytics and machine learning, but I think more as a component of a viable front end process than a replacement.  For at least some time into the future, humans and their ability to connect and assess ideas and identify trends and opportunities will do a better job than machines alone.  Intuition and past experience count a lot, but leveraging the data and insights that big data and algorithms can create will increase value in the front end.


What humans do better


Machine learning is still nascent, and still trying to capture the spark of real intuition and foresight.  This means that today most machine learning is exceptionally good at anticipating and predicting outcomes when the conditions are similar.  This means that for some time into the future, machine learning and algorithms should be able to anticipate incremental innovation demands.  However, I'm not so sure about disruptive needs and opportunities.


Humans are omnivorous connectors.  We are happy to ignore what should be incompatible needs or standards to connect things that don't seem to be connectable.  We connect things in random, often unexpected ways (you got chocolate in my peanut butter) that sometimes lead to spectacular failures and other times lead to amazing successes.  No algorithm could have predicted Steve Jobs and Apple combining MP3 players, digital music and a distribution mechanism called iTunes.  Therefore, companies that focus on machine learning and algorithms in the front end may perfect their incremental innovation and completely ignore disruptive innovation.


Watch this space


With all that I've written above, the capability of the algorithms and machine learning is advancing quickly.  It would not surprise me to find algorithms that get much better at predicting future disruptions and breakthrough innovations that emerge in the next 5-10 years.  Even then there will be a significant human component to understand and place the disruptive opportunity into context.


Fifty years ago we were promised individual jet packs and residence on the moon by the year 2000.  2001 promised a journey by an intelligent AI and astronauts to Jupiter.  Here in the real world some technologies have advanced quickly, but I think more development is necessary before we hand over the innovation reins to AI.  But that doesn't mean machine learning and big data doesn't have a place in the front end now, and those that start incorporating these as an input (not a replacement) to the front end will learn and benefit in ways that will cause others jealousy or regret.

• Tweet
• Share
• Share
• Share
• Share