Digital Dark Matter: The Unseen Forces That Influence Innovation

Editor’s note: Christian Cantrell is a software developer and science fiction writer. He has written two novels and several short stories, and his third novel will be out early next year.

Around AD 150, the Egyptian astronomer Claudius Ptolemy compiled a model of the universe that accurately predicted the movements of the sun, moon, planets, and stars. It was a remarkable achievement, diminished only by the fact that it happened to be entirely wrong. Rather than a relatively simple heliocentric model where planets revolved around a stationary sun, Ptolemy appealed to convoluted constructs to design a celestial Rube Goldberg Machine that kept the earth at the center of the cosmos where everyone felt it belonged.

And yet, it worked. In fact, it worked so well that  it remained the predominant cosmological model for roughly 1,500 years. Even now, projectors in planetariums are essentially inverted mechanical implementations of the Ptolemaic System. Ptolemys geocentric model was indeed a brilliant achievement, though in retrospect, its brilliance obviously had nothing to do with its veracity.

Its real genius lay in its ability to satisfy two seemingly incompatible requirements: the accurate prediction of the position and movement of celestial bodies, and the need for humans to feel that they were at the center of everything they could see. In other words, while the Ptolemaic System was objectively incorrect, it was very much right for its time.

All of this unexpectedly came to mind while discussing self-driving cars. The virtues of autonomous cars, I was told, were that they increased productivity because they allowed passengers to engage in tasks other than driving; they were safer than manually operated vehicles; and with properly optimized algorithms, they were certain to reduce traffic congestion.

Admittedly, they weren’t practical today, but in ten, twenty, thirty, forty years, tops — once the technology was ready, the necessary laws established, and our cultural biases overcome — driverless cars would not only be commonplace, they might even be mandated by law. The future, my conversation partner concluded, would finally be here.

While I didn’t disagree with any of these hypotheses, just for fun, I pointed out that all of these advantages could easily be achieved right now. In fact, they could have been achieved several decades ago with nothing more than a well designed and executed public transportation system.

Not only does good public transportation provide the same benefits as autonomous cars, but it helps reduce greenhouse gas emissions, releases us from the financial and logistical burdens of owning our own vehicles, creates new jobs, promotes better health, requires no new laws or technologies, and is accessible to nearly every demographic.

In terms of results, commuter trains, buses and good old fashioned walking are probably far superior to technology that is likely decades away in almost every respect but one: driverless cars are probably doable whereas a comprehensive, country-wide public transit system (such as those enjoyed by many other countries around the world) is almost certainly economically and politically infeasible.

Driverless cars are the Ptolemaic Systems of transportation. They’re an over-engineered solution to a relatively simple problem., but because they are compatible with things like existing infrastructure, cultural expectations, and well-established economic and political power structures, they probably have a much better chance of success in areas where public transportation hasn’t already taken hold.

Public transit, in stark contrast to autonomous vehicles, isn’t very sexy. It threatens our individuality, requires short-term investment for long-term gain, and many consider it uninspired and even unsightly. If that wasn’t enough, large-scale investment in public infrastructure presents tax burdens in most municipalities versus a massive economic opportunity for private industry with driverless cars.

As symbols of the future, bullet trains, maglevs, and monorails are almost quaintly anachronistic; best relegated to secluded and dusty dioramas at Disneys Tomorrowland; reminders — along with flying cars, video phones, and housekeeping robots — of what the future used to be.

The term “dark matter” refers to a hypothesized form of matter that cannot be detected, and is therefore inferred by observing its gravitational influence. Its equally mysterious counterpart, dark energy, is a hypothetical form of energy that appears to be accelerating the expansion of the universe.

Combined, dark matter and dark energy constitute about 95.1% of all the mass-energy of the known universe which means that we can only directly observe and account for a mere 4.9% of whats out there. Despite all we think we know about the nature of the universe, the overwhelming majority of the cosmos lies outside our current powers of observation, yet it profoundly affects everything.

So it goes with the forces that influence innovation. Advances in artificial intelligence; improvements in sensor technologies; and the perseverance of Moores Law are only the most obvious components needed to create and sustain a technology like autonomous vehicles. Cultural expectations, politics, and economic incentives are all equally influential, but far more difficult to detect and quantify.

Now lets consider a technology thats more accessible and familiar to many of us: the fitness tracker. Without getting into a public health debate, its safe to say that Americans have reached a point where we are looking far more critically at our health and how our physiology affects the overall quality of our lives. It’s also safe to say that various forms of technology are major contributing factors to our less healthy lifestyle.

Not only are we spending far more time in sedentary positions, looking at screens of various sizes, but food science also has increased the convenience and appeal of our meals without increasing their nutritional content.  While there are plenty of exceptions, I dont think its unfair to say that the more technology we introduce into our lives (cars, overnight and even same-day delivery, instant meals, etc.), the less healthy many of us become.

Yet rather than attempting to improve our health by reducing our dependency on technology, some of us are turning to technology to solve the very problem it helped create. Part of what makes fitness trackers economically viable is that devices like smartphones, tablets, and video games — the very innovations that often encourage a more sedentary lifestyle — have driven down the costs of components like multi-axis accelerometers, Bluetooth radios, and LCD screens.

This is only one example of how technology has a tendency to self-propagate; to use humans as a vector to create copies and mutations and derivatives of itself; to recursively generate demand for itself in order to not only ensure its own survival, but its constant and even exponential proliferation. The relationship between humans and technology is becoming increasingly enmeshed, and in some cases, even ambiguous.

Im not proposing that technology is behaving consciously, nor do I believe that our increasing dependency on technology is a bad thing. Im a huge fan of fitness trackers, and I experiment with just about every new technology to hit the market (though admittedly, I find the majority of it to be little more than a diversion).

The extent to which humanity adapts its needs to its environment and adapts its environment to its needs, is one of the most amazing things about our species. It’s something to be celebrated and fostered. But having a clear idea of the invisible forces that influence our technology is critical to maximizing its benefits, and ultimately establishing our control over it.

While fitness trackers may be emblematic of technology heavily influenced by well-obscured forces, they may not be the most enduring examples. Most technology pundits and consumer electronics insiders agree that devices like health bands are probably only temporary solutions — stopgap measures until dedicated exercise and wellness technology is absorbed into other devices.

This time, rather than convergence around a single form factor (the smartphone), many futurists are predicting the exact opposite. The distribution of functionality will occur across swarms of connected devices: referred to as wearable devices (when attached to our bodies) or the Internet of Things (when dispersed throughout our environment).

There are tremendous advantages to making everything from our appliances to our clothing to our fashion accessories connected and more intelligent, but the extent to which connected devices become assimilated into our daily lives also depends on overcoming a pretty substantial list of challenges. Among them are cross-device interoperability, robust data synchronization, practical battery life, persistent and dependable connectivity, the creation and adoption of all kinds of new user interface paradigms, aesthetics, and, of course, affordability.

The one thing all of these problems have in common is that there are huge amounts of resources being devoted to solving them. But connected devices also have one major challenge ahead of them where there are at least as many forces working against the problem: security.

Consider all the hacking, spying, and social engineering that we hear about. Now imagine how much of it goes on that we either; havent found out about yet, will never find out about, or that occurs on a scale too small to merit a headline.

There are several reasons why hacking is becoming prevalent, but one of the most important is that the overall attack surface is getting larger. Devices and personal data have already proliferated to the point where our security and privacy are under constant threat from black-hat hackers, foreign governments, and even our own government, and if the Internet of Things becomes a reality, the problem might get worse by several orders of magnitude.

Technology is inherently intent-agnostic. Just as cloud computing enables us to quickly and cheaply spin up massive amounts of computing resources for things like gene sequencing or the analysis of radio telescope data, it can also be used to brute-force passwords or decrypt sensitive personal information. While “big data” solutions enable us to mine historical records for the purposes of uncovering previously obscured connections, it can just as easily be used for warrantless tracking, profiling, and spying. And while increasingly affordable storage means never having to delete data, it also means data never getting deleted.

Its obvious that technology will continue to play an increasingly important role in almost every aspect of our lives. But it might also be true that we are entering a time when the wisdom of surrounding ourselves with yet more computers, cameras, and sensors is legitimately called into question — when the risks of going from a handful of connected devices to dozens, then hundreds, and maybe even thousands could easily outweigh the benefits. Its undeniable that there are forces encouraging the proliferation of connected devices, but before we declare their unfettered and exponential growth to be preordained, its important to recognize that there are also plenty of valid forces working to achieve a much more conservative and practical equilibrium.

As a science fiction writer, gadget addict, and software developer, I have no intention of cautioning anyone against the potential dangers of scientific advancement and technological innovation. Humans have been dependent on technology of one kind or another for many thousands of years, and its probably inevitable that our dependency will not only continue, but dramatically increase. In my mind, becoming more aware of the obscure forces that guide, define, and influence the ways in which we shape our world is not about avoiding the perils of the future, but about helping to ensure that we are able to achieve our full potential.

Image via Shutterstock user andrey_I