Integration of AI, IoT and Big Data: The Intelligent Assistant

Five years ago, I wrote a post on this blog disparaging the state of the Internet of Things/home automation market as a “Tower of Proprietary Babble.” Vendors of many different home and industrial product offerings were literally speaking different languages, making their products inoperable with other complementary products from other vendors. The market was being constrained by its immaturity and a failure to grasp the importance of open standards. A 2017 Verizon report concluded that “an absence of industry-wide standards…represented greater than 50% of executives concerns about IoT. Today I can report that finally, the solutions and technologies are beginning to come together, albeit still slowly.

The Evolution of These Technologies Is Clearer

The IoT Tower of Proprietary Babble Is Slowly Crumbling

The Rise of the Intelligent Assistant

Five years ago, I wrote a post on this blog disparaging the state of the Internet of Things/home automation market as a “Tower of Proprietary Babble.” Vendors of many different home and industrial product offerings were literally speaking different languages, making their products inoperable with other complementary products from other vendors. The market was being constrained by its immaturity and a failure to grasp the importance of open standards. A 2017 Verizon report concluded that “an absence of industry-wide standards…represented greater than 50% of executives concerns about IoT.” Today I can report that finally, the solutions and technologies are beginning to come together, albeit still slowly.

One of the most important factors influencing these positive developments has been the recognition of the importance of this technology area by major corporate players and a large number of entrepreneurial companies funded by venture investment, as shown in the infographic above. Amazon, for example, announced in October 2018 that it has shipped over 100 Million Echo devices, which effectively combine an intelligent assistant, smart hub, and a large-scale database of information. This does not take into account the dozens of other companies which have launched their own entries. I like to point to Philips Hue as such an example of corporate strategic focus perhaps changing the future corporate prospects of Philips, based in Eindhoven in the Netherlands. I have visited Philips HQ, a company trying to evolve from the incandescent lighting market. Two years ago my wife bought me a Philips Hue WiFi controlled smart lighting starter kit. My initial reaction was disbelief that it would succeed. I am eating crow on that point, as I now control my lighting using Amazon’s Alexa and the Philips Hue smart hub. The rise of the “intelligent assistant” seems to have been a catalyst for growth and convergence.

The situation with proprietary silos of offerings that do not work well or at all with other offerings is still frustrating, but slowly evolving. Amazon Firestick’s browser is its own awkward “Silk” or alternatively Firefox, but excluding Google’s Chrome for alleged competitive advantage. When I set up my Firestick, I had to ditch Chromecast because I only have so many HDMI ports. Alexa works with Spotify but only in one room as dictated by Spotify. Alexa can play music from Amazon Music or Sirius/XM on all Echo devices without the Spotify limitation. Which brings me to another point of aggravation: alleged Smart TV’s. Not only are they not truly “smart,” they are proprietary silos of their own, so “intelligent assistant” smart hubs do not work with “smart” TV’s. Samsung, for example, has its own competing intelligent assistant, Bixby, so of course, only Bixby can control a Samsung TV. I watched one of those YouTube DIY videos on how you could make your TV work with Alexa using third-party software and remotes. Trust me, you do not want to go there. But cracks are beginning to appear that may lead to a flood of openness. Samsung just announced at CES that beginning in 2019 its Smart TV’s will work with Amazon Echo and Google Home, and that a later software update will likely enable older Samsung TV’s to work with Echo and Home. However, Bixby will still control the remote. Other TV’s from manufacturers like Sony and LG have worked with intelligent assistants for some time.

The rise of an Internet of Everything Everywhere, the recognition of the need for greater data communication bandwidth, and battery-free wireless IoT sensors are heating up R&D labs everywhere. Keep in mind that I am focusing on the consumer side, and have not even mentioned the rising demands from industrial applications. Intel has estimated that autonomous vehicles will transmit up to 4 Terabytes of data daily. AR and VR applications will require similar throughput. Existing wireless data communication technologies, including 5G LTE, cannot address this need. In addition, an exploding need for IoT sensors not connected to an electrical power source will require more work in the area of “energy harvesting.” Energy harvesting began with passive RFID, and by using kinetic, pizeo, and thermoelectric energy and converting it into a battery-free electrical power source for sensors. EnOcean, an entrepreneurial spinoff of Siemens in Munich has pioneered this technology but it is not sufficient for future market requirements.

Fortunately, work has already begun on both higher throughput wireless data communication using mmWave spectrum, and energy harvesting using radio backscatter, reminiscent of Nikola Tesla’s dream of wireless electrical power distribution. The successful demonstration of these technologies holds the potential to open the door to new IEEE data communication standards that could potentially play a role in ending the Tower of Babble and accelerating the integration of AI, IoT, and Big Data. Bottom line is that the market and the technology landscape are improving.

My IEEE Talk from 2013 foreshadows the development of current emerging trends in advanced technology, as they appeared at the time. I proposed that in fact, they represent one huge integrated convergence trend that has morphed into something even bigger, and is already having a major impact on the way we live, work, and think. The 2012 Obama campaign’s sophisticated “Dashboard” application is referenced, integrating Big Data, The Cloud, and Smart Mobile was perhaps the most significant example at that time of the combined power of these trends blending into one big thing.

READ MORE: Blog Post on IoT from July 20, 2013

The term “Internet of Things” (IoT) is being loosely tossed around in the media. But what does it mean? It means simply that data communication, like Internet communication, but not necessarily Internet Protocol packets, is emerging for all manner of “things” in the home, in your car, everywhere: light switches, lighting devices, thermostats, door locks, window shades, kitchen appliances, washers & dryers, home audio and video equipment, even pet food dispensers. You get the idea. It has also been called home automation. All of this communication occurs autonomously, without human intervention. The communication can be between and among these devices, so-called machine to machine or M2M communication. The data communication can also terminate in a compute server where the information can be acted on automatically, or made available to the user to intervene remotely from their smart mobile phone or any other remote Internet-connected device.

Another key concept is the promise of automated energy efficiency, with the introduction of “smart meters” with data communication capability, and also achieved in large commercial structures via the Leadership in Energy & Environmental Design program or LEED. Some may recall that when Bill Gates built his multi-million dollar mansion on Lake Washington in Seattle, he had “remote control” of his home built into it. Now, years later, Gates’ original home automation is obsolete. The dream of home automation has been around for years, with numerous Silicon Valley conferences, and failed startups over the years, and needless to say, home automation went nowhere. But it is this concept of effortless home automation that has been the Holy Grail.

But this is also where the glowing promise of The Internet of Things (IoT) begins to morph into a giant “hairball.” The term “hairball” was former Sun Microsystems CEO, Scott McNealy‘s favorite term to describe a complicated mess. In hindsight, the early euphoric days of home automation were plagued by the lack of “convergence.” I use this term to describe the inability of available technology to meet the market opportunity. Without convergence, there can be no market opportunity beyond early adopter techno geeks. Today, the convergence problem has finally been eliminated. Moore’s Law and advances in data communication have swept away the convergence problem. But for many years the home automation market was stalled.

Also, as more Internet-connected devices emerged it became apparent that these devices and apps were a hacker’s paradise. The concept of IoT was being implemented in very naive and immature ways and lacking common industry standards on basic issues: the kinds of things that the IETF and IEEE are famous for. These vulnerabilities are only now very slowly being resolved, but still in a fragmented ad hoc manner. The central problem has not been addressed due to classic proprietary “not invented here” mindsets.

The problem that is currently the center of this hairball, and from all indications is not likely to be resolved anytime soon. It is the problem of multiple data communication protocols, many of them effectively proprietary, creating a huge incompatible Tower of Babbling Things. There is no meaningful industry and market wide consensus on how The Internet of Things should communicate with the rest of the Internet. Until this happens, there can be no fulfillment of the promise of The Internet of Things. I recently posted “Co-opetition: Open Standards Always Win,” which discusses the need for open standards in order for a market to scale up.

A recent ZDNet post explains that home automation currently requires that devices need to be able to connect with “multiple local- and wide-area connectivity options (ZigBee, Wi-Fi, Bluetooth, GSM/GPRS, RFID/NFC, GPS, Ethernet). Along with the ability to connect many different kinds of sensors, this allows devices to be configured for a range of vertical markets.” Huh? This is the problem in a nutshell. You do not need to be a data communication engineer to get the point. And this is not even close to a full discussion of the problem. There are also IoT vendors who believe that consumers should pay them for the ability to connect to their proprietary Cloud. So imagine paying a fee for every protocol or sensor we employ in our homes. That’s a non-starter.

The above laundry list of data communication protocols, does not include the Zigbee “smart meter” communications standards war. The Zigbee protocol has been around for years, and claims to be an open industry standard, but many do not agree. Zigbee still does not really work, and a new competing smart meter protocol has just entered the picture. The Bluetooth IEEE 802.15 standard now may be overtaken by a much more powerful 802.15 3a. Some are asking if 4G LTE, NFC or WiFi may eliminate Bluetooth altogether. A very cool new technology, energy harvesting, has begun to take off in the home automation market. The energy harvesting sensors (no batteries) can capture just enough kinetic, peizo or thermoelectric energy to transmit short data communication “telegrams” to an energy harvesting router or server. The EnOcean Alliance has been formed around a small German company spun off from Siemens, and has attracted many leading companies in building automation. But EnOcean itself has recently published an article in Electronic Design News, announcing that they have a created “middleware” (quote) “…to incorporate battery-less devices into networks based on several different communication standards such as Wi-Fi, GSM, Ethernet/IP, BACnet, LON, KNX or DALI.” (unquote). It is apparent that this space remains very confused, crowded and uncertain. A new Cambridge UK startup, Neul is proposing yet another new IoT approach using the radio spectrum known as “white space,” becoming available with the transition from analog to digital television. With this much contention on protocols, there will be nothing but market paralysis.

Is everyone following all of these acronyms and data comm protocols? There will be a short quiz at the end of this post. (smile)

The advent of IP version 6, strongly supported by Intel and Cisco Systems has created another area of confusion. The problem with IPv6 in the world of The IoT is “too much information” as we say. Cisco and Intel want to see IPv6 as the one global protocol for every Internet connected device. This is utterly incompatible with energy harvesting, as the tiny amount of harvested energy cannot transmit the very long IPv6 packets. Hence, EnOcean’s middleware, without which their market is essentially constrained.

Then there is the ongoing new standards and upgrade activity in the International Standards Organization (ISO), The Institute of Electrical and Electronics Engineers (IEEE), Special Interest Groups (SIG’s”), none of which seem to be moving toward any ultimate solution to the Tower of Babbling Things problem in The Internet of Things.

The Brave New World of Internet privacy issues relating to this tidal wave of Big Data are not even considered here, and deserve a separate post on the subject. A recent NBC Technology post has explored many of these issues, while some have suggested we simply need to get over it. We have no privacy.

Stakeholders in The Internet of Things seem not to have learned the repeated lesson of open standards and co-opetition, and are concentrating on proprietary advantage which ensures that this market will not effectively scale anytime in the foreseeable future. Intertwined with the Tower of Babbling Things are the problems of Internet privacy and consumer concerns about wireless communication health & safety issues. Taken together, this market is not ready for prime time.

https://www.slideshare.net/mayo615/ieee-talk-integrated-big-data-the-cloud-smart-mobile-big-deal-or-not/mayo615/ieee-talk-integrated-big-data-the-cloud-smart-mobile-big-deal-or-not

The Internet of Things: The Promise Versus the Tower of Hacked Babbling Things

The term “Internet of Things” (IoT) is being loosely tossed around in the media. But what does it mean? It means simply that data communication, like Internet communication, but not necessarily Internet Protocol packets, is emerging for all manner of “things” in the home, in your car, everywhere: light switches, lighting devices, thermostats, door locks, window shades, kitchen appliances, washers & dryers, home audio and video equipment, even pet food dispensers. You get the idea. It has also been called home automation. All of this communication occurs autonomously, without human intervention. The communication can be between and among these devices, so called machine to machine or M2M communication. The data communication can also terminate in a compute server where the information can be acted on automatically, or made available to the user to intervene remotely from their smart mobile phone or any other remote Internet connected device.

But this is also where the glowing promise of The Internet of Things (IoT) begins to morph into a giant “hairball.” The term “hairball” was former Sun Microsystems CEO, Scott McNealy‘s favorite term to describe a complicated mess. In hindsight, the early euphoric days of home automation were plagued by the lack of “convergence.” I use this term to describe the inability of available technology to meet the market opportunity. Without convergence there can be no market opportunity beyond early adopter techno geeks. Today, the convergence problem has finally been eliminated. Moore’s Law and advances in data communication have swept away the convergence problem. But for many years the home automation market was stalled.

Is everyone following all of these acronyms and data comm protocols? There will be a short quiz at the end of this post. (smile)

The Internet of Things: The Promise Versus the Tower of Babbling Things

The term “Internet of Things” is being loosely tossed around in the media. But what does it mean? It means simply that data communication like the Internet, but not necessarily Internet Protocol packets is emerging for all manner of “things” in the home: light switches, lighting devices, thermostats, door locks, window shades, kitchen appliances, washers & dryers, home audio and video equipment, even pet food dispensers. You get the idea. All of this communication occurs autonomously, without human intervention. The communication can be between and among these devices, so called machine to machine or M2M. The data communication can also terminate in a home compute server where the information can be made available to the homeowner to intervene remotely from their smart mobile phone or any other remote Internet connected device.

The term “Internet of Things” (IoT) is being loosely tossed around in the media. But what does it mean? It means simply that data communication, like Internet communication, but not necessarily Internet Protocol packets, is emerging for all manner of “things” in the home, in your car, everywhere: light switches, lighting devices, thermostats, door locks, window shades, kitchen appliances, washers & dryers, home audio and video equipment, even pet food dispensers. You get the idea. It has also been called home automation. All of this communication occurs autonomously, without human intervention. The communication can be between and among these devices, so called machine to machine or M2M communication. The data communication can also terminate in a compute server where the information can be acted on automatically, or made available to the user to intervene remotely from their smart mobile phone or any other remote Internet connected device.

But this is also where the glowing promise of The Internet of Things (IoT) begins to morph into a giant “hairball.” The term “hairball” was former Sun Microsystems CEO, Scott McNealy‘s favorite term to describe a complicated mess. In hindsight, the early euphoric days of home automation were plagued by the lack of “convergence.” I use this term to describe the inability of available technology to meet the market opportunity. Without convergence there can be no market opportunity beyond early adopter techno geeks. Today, the convergence problem has finally been eliminated. Moore’s Law and advances in data communication have swept away the convergence problem. But for many years the home automation market was stalled.

The other problem is currently the center of this hairball, and from all indications is not likely to be resolved anytime soon. It is the problem of multiple data communication protocols, many of them effectively proprietary, creating a huge incompatible Tower of Babbling Things. There is no meaningful industry and market wide consensus on how The Internet of Things should communicate with the rest of the Internet. Until this happens, there can be no fulfillment of the promise of The Internet of Things. I recently posted “Co-opetition: Open Standards Always Win,” which discusses the need for open standards in order for a market to scale up.

Is everyone following all of these acronyms and data comm protocols? There will be a short quiz at the end of this post. (smile)

5 Ways Big Data Is Going To Blow Your Mind

Call it whatever you want — big data, data science, data intelligence — but be prepared to have your mind blown. Imagination and technology are on a collision course that will change the world in profound ways. Some people say big data is wallowing in the trough of disillusionment, but that’s a limited worldview. If you only look at it like an IT issue it might be easy to see big data as little more than business intelligence on steroids. If you only see data science as a means to serving better ads, it might be easy to ask yourself what all the fuss is about. If you’re like me, though, all you see are the bright lights ahead. They might be some sort of data nirvana, or they might be a privacy-destroying 18-wheeler bearing down on us. They might be both. But we’re going to find out, and we’re we’re going to find out sooner rather than later. This is because there are small pockets of technologists who are letting their imaginations lead the way. In a suddenly cliché w

“Big Data, Big Deal or Not?” Debate Continues

The Gigaom Structure Data Conference has just concluded in New York City. It has added significantly to the discussion and the debate on the significance of this phenomenon. The author, Derek Harris, has summarized my own view on the issue in his first paragraph. The notion that Big Data is little more than business intelligence on steriods is just wrong, and those who fail to understand its importance and exploit it, may well be the losers.

5 ways Big Data is going to blow your mind and change your world

by Derrick Harris

photo: Shutterstock / GrandDuc

SUMMARY:

Some people say big data is wallowing in the trough of disillusionment, but that’s a limited worldview. If you only look at it like an IT issue it might be easy to see big data as little more than business intelligence on steroids. If you only see data science as a means to serving better ads, it might be easy to ask yourself what all the fuss is about.

If you’re like me, though, all you see are the bright lights ahead. They might be some sort of data nirvana, or they might be a privacy-destroying 18-wheeler bearing down on us. They might be both. But we’re going to find out, and we’re we’re going to find out sooner rather than later.

This is because there are small pockets of technologists who are letting their imaginations lead the way. In a suddenly cliché way of saying it, they’re aiming for 10x improvement rather than 10 percent improvement. They can do that because they now have a base set of analytic technologies and techniques that are well positioned to solve, with relatively little effort, whatever data problems are thrown their way.

Here are some themes from our just-concluded Structure: Data conference that I think highlight the promise of data, but also the challenges that lie ahead.

Man and machine unite

Machine learning is already infiltrating nearly every aspect of our digital lives, but its ultimate promise will only be realized when it becomes more human. That doesn’t necessarily mean making machines think like human brains (although, granted, that’s a vision currently driving billions of research dollars), but just letting people better interact with the systems and models trying to discover the hidden patterns in everything around us.

Whatever shape it takes, the results will be revolutionary. We’ll treat diseases once thought untreatable, tackle difficult socio-economic and cultural issues, and learn to experience the world around in entirely new ways. Maybe that consumer-experience scourge known as advertising might actually become helpful rather than annoying.

That would really be something.

Man and Machine Unite

Data science, or data intelligence?

I’m not sure there needs to be a distinction between data science and data intelligence, but the latter does connote a grander goal. It’s about trying to solve meaningful problems rather than just serving ads; about trying to understand why things happen just as well as when they’ll happen. This means learning to work with smaller, messier data than we might like — certainly smaller and messier than the data sets underneath most of the massive web-company data science undertakings.

But just think about being able to go beyond predictive models and into a world of preventative — or even professorial — models. If you know what I like, where I go and who my friends are, it might be fairly easy to predict what I want to buy. Figuring out how my decision to buy something might affect my overall well-being and then telling me why? That’s a little more difficult and a lot more beneficial.

Telling stories with data

Have you ever looked at a chart and wondered what the heck it was supposed to be telling you? Or downloaded a report of your Facebook activity only to ask yourself if all the disparate data points come together to paint a bigger picture? Or tried — and failed — to stop a terrorist before his movement to recruit an army of followers gained critical mass?

A big problem with a lot data analysis right now is that it still treats data points as entities unto themselves, largely disconnected from those around them. However, data needs context in order to be really useful; it’s context that turns disparate data points into a story. Don’t just tell me how many steps I took today or the time of day I’m most active on Facebook, but tell me how that relates to the rest of my life.

And don’t just tell me that someone said he wants to kill Americans. Rather, tell me a story about how much more frequently he’s saying it and how much more inciteful his words are becoming.

The internet of things knows all

The mobile phone in your pocket is tracking your every movement and can also monitor the sounds that are surrounding you. That fitness tracker you’re wearing is identifying you by how you walk. Your smart meter data shows when you’re home, when you’re away and when you’re in the shower. Sensors in everything from toothbrushes to cars are quantifying every aspect of our lives.

This volume of data can still be a lot to deal with in terms of its volume, velocity and variety, and we’re still not quite sure what to do with it even if the right tools were in place. But all sorts of entrepreneurs, powerful institutions and intelligence agents have ideas. The technological pieces are coming along nicely, too. Just sayin’ …

This semantic life

The semantic web lives on; only it’s spreading well beyond our search engines and even our web browsers. Soon enough, we’ll be able to surface relevant content and people simply by highlighting passage of text in whatever we’re reading — web page or not — on any type of device. When we speak to our devices, they’ll not only know what we’re saying, but also what we really want even without the help of specific commands or keywords.

That’s a powerful proposition in a world where we increasingly expect our interactions to be hands-free and our answers to come as fast as our questions. Of course, what’s powerful in the hands of consumers driving in their cars or sitting on their couches iseven more powerful in the hands of doctors trying to diagnose difficult diseases or aid workers trying lend a helping hand in places where they don’t know the customs or even speak the language.

Big Data, The Cloud And Smart Mobile Are Actually One Big Thing

ToDaClo is a current buzz word of sorts for “touch-data-cloud,” (or Big Data, The Cloud and Smart Mobile) which appears to have been coined by a writer for Forbes magazine during a talk in Paris in May 2012. The speaker declared the death of the previous buzz word, SoLoMo (social-local-mobile). ToDaClo does not seem to have caught on beyond France as most of the writing and blogging about it is in French. SoLoMo had a following for some time, and even has an online manifesto, vaguely implying location based services, which have been a major mobile feature for some time in Asia, but not here. I think the bottom line is that both of these acronyms are trying to communicate the concept that Big Data, The Cloud and Smart Mobile are inter-related. I actually think of them as One Big Thing, even The Next Big Thing, or perhaps “Ne-Bi-Ng” (Nebing) as some may prefer, though I doubt Nebing will ever catch on. Sanjay Poonen, President & CEO of SAP also views them as One Big Thing.

Reblogged from Gigaom

The secret to tackling mobile, cloud and big data? Treat them as one.

by Sanjay Poonen

Sanjay Poonen, President & CEO of SAP AG

SUMMARY:It’s no secret that mobile, big data and cloud computing are transforming IT. Sanjay Poonen, president of SAP’s mobile division, says companies need a single unified strategy to tackle them, not three separate ones.

There is widespread agreement—across the globe and in every industry—that mobile, big data, and cloud computing are the three cornerstone issues of tomorrow’s business environment. In fact, a strong organizational response to each of these issues is already critical to competitive survival.

As a result, CIOs, business strategists and IT leaders are working furiously to make sure their businesses have plans in place to stay ahead of these challenges. But there is one subtlety that is frequently overlooked: When it comes to mobile computing, big data and the cloud, what we have is not three problems but one.

Rising in unison

It’s not a coincidence that the profile of these three business challenges rose in parallel. Mobile, big data, and cloud are not siloed concerns easily addressed in isolation. They exist in an overlapping matrix, where the importance of each issue increases because it leverages (or helps solve) an issue raised by one of the others.

For example, in the days before mobile computing, business users typically did all their work using just a handful of applications. Today, the average smartphone has 41 apps installed on it. And each of those applications sparks a need to consider security, since it generates data each and every time it is used. And because these devices are often connected to service provider networks – rather than directly with corporate servers – a great deal of that business app data requires secure cloud storage.

Thus the proliferation of mobile devices exacerbates the big data problem, which in turn precipitates the demand for cloud.

In short, they are all part of a single, converged and symbiotic trend. And to address them optimally requires a holistic perspective on all three.

No bottom in sight

With global demand for mobile computing at the heart of this escalation, it makes sense that IT strategists would be keenly interested in the trend lines for mobile adoption. Today, 87 percent of the world’s population owns a mobile phone; 60 million Android devices were sold in the second quarter of 2012, and now 1 million new Android devices are provisioned daily, according to Google. As of last month, there were likely more smartphones on the planet than humans, according to Cisco.

So the question is whether there is a saturation point on the horizon that could help curb this cloud/mobile/data demand? Surprisingly, no. The average number of mobile devices per employee worldwide has already reached three to five, and adoption rates continue to grow as consumers add tablets and ever-more capable smartphones to their mobile arsenals.

But consumers’ ceaseless enthusiasm for new form factors and functionality is not the whole story behind the world’s bottomless demand for mobility. Today, businesses themselves – rather than consumers – are adding fuel to the fire.

Not just a BYOD issue

As industries finally crest the hump of transforming their workflows to leverage mobile device availability, they drive new demand – not only for mobile devices, but for new scalable infrastructures that deliver more actionable intelligence from their big data.

Finance Consumer banks, operators and retailers are widely deploying mobile commerce capabilities, which, in addition to automating traditional transactions, must include on-demand access to unstructured data, such as check images.

Manufacturing Mobile devices on the factory floor automate manual processes, thereby feeding more rapid information into the system. This makes it possible to detect and respond early to issues that take a toll on quality or productivity, such as supplier errors.

Retail Retailers are giving regional store managers mobile app access to daily and even real-time sales performance data on the floor, allowing them to optimize displays and customer service to sell more of the most popular items.

Health care Thanks to new mobile apps and devices, the details of every patient interaction is now entered into the system nearly instantaneously. This provides a basis for a more efficient and orchestrated care response, and in some cases leading to more rapid or accurate diagnoses.

The internet of things

As mobile technology embeds itself into more and more objects, vehicles, buildings, sensors and machines, the heterogeneity of actionable business information will only grow. “Annual global IP traffic will surpass the zettabyte threshold by the end of 2016,” reports Cisco. “In 2016, global IP traffic will reach 1.3 zettabytes per year or 109.5 exabytes per month.” (As we already know, there are currently at least 2.7 zettabytes in storage globally).

Smart equipment and vehicles will upload data to service provider networks as well as private networks, and organizations will need a plan to normalize data in many forms and from many sources. The scalable infrastructures we design today to store and structure such varied data are critical to the enablement of the business innovations we will need in the future.

The effect of this convergence is already apparent, especially in the area of business intelligence. Mobile business intelligence makes it possible for organizations to provide analytics on key performance metrics to a wider variety of employees – not just for executives. Once employees get a taste for how mobile apps fuel greater effectiveness in their job duties, they will push for more dashboards and more data. And these big data stores can’t be undertaken without cloud, to facilitate real-time performance, nor mobile devices and apps, to deliver data into the field where it’s put to good use.

Embracing the Entanglement

The interdependence of mobile, big data and cloud is undeniable, and will only multiply as data growth and mobile use continue. Yet our strategic thinking lags behind the evidence. As we have learned from IT revolutions of the past, a partial strategy is worse than no strategy at all, as you can end up with an inflexible, tactical implementation that requires a ‘rip and replace’ approach.

Organizations that manage to avoid a false start with a siloed strategy will create a network design better aligned with where IT will be in five years. In short, the most successful organizations recognize the secret alliance of mobile, big data and cloud early, and develop a holistic strategy considering all three in concert.

Big Data: Big Deal Or Not?

I have been having a spirited marathon debate with a couple of my friends. Is this alleged new “Big Data thingy” so transformational that it will change our every day lives, or is it just an evolutionary advance that may improve productivity but not much else? The same arguments may apply to the concept of “The Cloud,” and “Smart Mobile.” The three, taken together, are coalescing into the major information technology forces that will drive innovation and productivity into the foreseeable future.

PollDaddy: What Is Your Opinion? Big Data: Big Deal Or Not? Or Comment Below

We are hearing regularly in the media about so-called “Big Data.” What exactly is Big Data? A number of differing definitions have been offered from a wide range of media sources. ZDNet‘s definition is one of the best I have seen so far. In essence, big data is about liberating data that is large in volume, broad in variety and high in velocity from multiple sources in order to create efficiencies, develop new products and be more competitive. Forrester puts it succinctly in saying that big data encompasses “techniques and technologies that make capturing value from data at an extreme scale economical” Prior to the emergence of commercial Big Data, the concept only existed where cost was no object: in the black world of the National Security Administration, and required the largest purpose-built supercomputers in existence.

A zettabyte (symbol ZB, derived from the SI prefix zetta-) is a quantity of information or information storage capacity equal to 10²¹ bytes or 1,000 exabytes (or one sextillion (one long scale trilliard) bytes).^[1]^[2]^[3]^[4]^{[5]…..I Billion terabytes….Today, you can walk into your local computer store and buy a couple of terabyes for a $100. Only $500 Million for a zettabyte. In real terms that is dirt cheap, and getting cheaper daily. Now that we have that cleared up, we can move to the next level.}

With regard to the obvious issue of personal privacy, the European Union and other organizations have made efforts to protect privacy, with very mixed results. Other governments, notably China, are aggressively implementing opposite policies to strictly limit privacy. Highly sophisticated telecommunications equipment has been available for years that enables deep analysis of all of your voice and Internet traffic. We learned this when Dick Cheney secretly set up such equipment to track and record all voice and data traffic in the United States. The equipment trapped and analyzed all of it in real time. You didn’t notice a thing. The thing about your personal data is that they already have it. Most of it comes from public sources you authorized. I not advocating this, I am only the messenger. The founder and former CEO of Sun Microsystems, Scott McNealy famously said, “You have no privacy. Get over it.” We must not ignore the serious issue of privacy, but the problem is already here and deep data mining is thriving. Privacy needs a revolution of its own.

The core question then becomes whether Big Data, and for that matter, the Cloud, and Smart Mobile, represent revolutionary and transformational changes in technological capability and also consequentially, human culture, politics: how we conduct ourselves in the World. Or is it just so many more boring zeros and ones zooming by at the speed of light, stored in chips, and processed by quantum microprocessors? No big deal, just IT management as normal. Frankly, this is a significant philosophical question. For this discussion, we will focus only on Big Data. Discussion of the Cloud and Smart Mobile will follow later. My most recent post on Smart Mobile gives a hint of my thoughts: Mobile Market Share: A War of Titans Worth Following, http://mayo615.com/2013/01/21/mobile-os-market-share-strategy-war-of-the-titans-worth-following/

In fairness, I cut my teeth on Marshall McLuhan‘s ideas while in university in the 1960’s. In an amazing irony, I soon fell into Intel Corporation at the birth of the microprocessor revolution, and later, I was also present to personally participate in the emergence of the personal computer. My memory of McLuhan kept popping up everywhere. As my career progressed, I seemed to jump onto each new wave: networking at Sun Microsystems, then the Internet infrastructure build out explosion with Ascend Communications, and finally a host of new companies, based on Internet-based capabilities. Through all of it, I could only conclude that somehow McLuhan, like some kind of modern Nostradamus, had foreseen it all. Most importantly, my own life was transformed by it all, and I saw with my own eyes the massive transformation occurring all around me.

So I have no doubt that Big Data is transforming our lives, and will continue to transform our lives, in ways we cannot yet fully grasp, as I could not grasp McLuhan when I first heard him, or the significance of the Internet as I sat right in the middle of it.

I have previously described Big Data as analogous to the evolution of Chaos Theory. For centuries, full understanding of the complexity of nature’s designs were thought to be the realm of God, and beyond human comprehension and explanation. Then in the 1960’s in places like Santa Cruz, California and Germany, the elegant simplicity of a solution to chaos began to emerge. The massive scale of Big Data is a very similar nut to crack. We are now seeing an elite group of data scientists and mathematicians begin to solve Big Data in a way similar to how chaos was resolved. Google, Microsoft Bing, Baidu, Yahoo and Amazon are driving the development of these mathematical skill sets.

Last year I showed my UBC Faculty of Management students a YouTube video on Data Mining. In the video, the two Hungarian mathematicians leading a data mining company, described how they had solved hideously complex problems that were previously beyond any computational solution. The key to their success was their ability to extract very precise useful information from extraordinarily large stores of information. The metaphor here is more like finding a particular grain of sand on a very large beach. A parallel key factor has been the incessant march of Moore’s Law. Even 10 years ago, successful data mining on this scale could not have been accomplished. The computational cycles and high speed mass storage were not available or were too expensive. Today those microprocessor cycles are available. The costs will continue to plummet, making further advances inevitable. Failure to consider Moore’s Law and available computational cycles has also been the cause of many failed ideas over the years. But the threshold has arrived.

Today, developments like Google Spanner, the largest known database architecture in the World, have joined with the computational solutions.

Unveiled this fall after years of hints and rumors, it’s the first worldwide database worthy of the name — a database designed to seamlessly operate across hundreds of data centers and millions of machines and trillions of rows of information.

Spanner is a creation so large, some have trouble wrapping their heads around it. But the end result is easily explained: With Spanner, Google can offer a web service to a worldwide audience, but still ensure that something happening on the service in one part of the world doesn’t contradict what’s happening in another.

Google’s decision to reveal Spanner has many dimensions. First, it provides a peek into the black World of the U.S. National Security Agency and the U.S. Defense Intelligence Agency. Previously, the existence of such large and sophisticated global databases were only imagined. We now know they exist and are a crucial component of Big Data.

Read more in my post, Google Spanner, the single largest database in the world

http://mayo615.com/2012/11/26/inside-google-spanner-the-single-largest-database-in-the-world/

For me, the most compelling example of how this all works, has been the extremely sophisticated Big Data mining used by the Obama campaign to achieve re-election. As early as March 2012, the Wall Street Journal began reporting about “Dashboard,” the Obama campaign app that was mining Big Data to find undecided voters in key states. But not only undecided voters. Dashboard can key in, find and persuade “Off the Grid” voters. Off the Grid is the term used to describe those people, such as students and other young people, with constantly changing locations and only a mobile phone. These voters have historically been virtually impossible to reach. This short PBS Newshour video below speaks volumes about the extraordinary impact and value of Big Data, not seen before.

Watch How Much Do Digital Campaigns Know About You? on PBS. See more from PBS NewsHour.

The campaign’s hiring of Rayid Ghani, as “chief data scientist,” and an army of data analysts, set the stage for what was to come. On election night, Mitt Romney and Paul Ryan were absolutely convinced that they had won the election, but were shocked to find otherwise. Working through their disbelief, both candidates later remarked about the enormous voter turnout for Democrats in key locations and the “technology advantage” of the Obama campaign.

So from my years of observation of the march of technology and its impact on my own life, I am convinced that we are entering another transformational period as profound as the emergence of the Internet itself.

I have been repeatedly drawn back to Steve Job’s 2005 Stanford University commencement address, in which he closes with references to Stuart Brand and The Whole Earth Catalog. Stuart Brand is an extraordinary futurist. One of Ken Kesey’s original Merry Prankster’s chronicled in Tom Wolfe’s book non-fiction novel, The Electric Kool-Aid Acid Test, Brand had been inspired by the legendary first photograph of the entire Earth taken by Apollo 8 astronaut Frank Borman. Brand is also the founder of The Well, the very early Sausalito-based Internet Service Provider, who is now considered one of the most important thinkers on human culture, technology and its impacts. Word of Job’s commencement address spread virally around the Valley...”Did you hear what Job’s said at Stanford today?” Steve was basically saying that he too understood what McLuhan had said, and that Stuart Brand also understood the transformational importance of the Global Village, by publishing The Whole Earth Catalog.

Management In The Brave New World: The Cloud, Big Data And Smart Mobile

This is the first in a series I will be posting on management education and the crucial link with cyber skills and awareness of how the Web works.

Profound changes in the World of Information, “the cyber world,” are dramatically impacting management: the urgent need for management to understand the brave new cyber world, to develop new management skills to cope with it, and to adapt their entire organizations to this new environment. It is not hyperbole to say that it is a “strategic inflection point” for the entire practice of management. I recently showed my undergrad and graduate strategy students a video of a very recent Charlie Rose interview with John Chambers, CEO of Cisco Systems. In that interview Chambers emphasized the acceleration of the Adizes corporate life cycle, in many cases to less than ten years, and the need for constant reinvention to survive in this challenging and rapidly changing new world. This is now also true about the teaching of Information Technology to management students and to all undergraduate students for that matter.

In the late 1980’s The University of California at Santa Cruz was a bit of an anomaly in requiring that all undergraduate students take a course in UNIX and C++ programming. The Internet at that time was little more than a text-based blinking green cursor on a tiny terminal. Tim Berners-Lee had not yet invented “Mosaic, the world’s first Internet browser, or the concept of URL’s. Despite some griping from the students, most went on to realize the value of this in their curriculum. There was even, Santa Cruz Operation, (SCO) a quirky little company in the heart of Santa Cruz, that had bought Microsoft’s proprietary version of UNIX, known as Xenix, and had carved out a modest niche market for it, and provided a conduit for some UCSC students to find work. Later, as the browser world began to explode, the UC system made HTML web development skills mandatory. Now we have even advanced beyond applications like FrontPage and Dreamweaver, which dramatically advanced Web page development for non-programmers, to XHTML and CSS, providing another leap forward.

I realized just how big this all would become when I was an executive with Sun Microsystems and we hosted an industry analyst conference down at the Carmel Valley Country Club. The first clunky browser, Mosaic, had just become available from a young guy named Marc Andreesen, then at the National Supercomputer Center in Chicago. Sun wanted to show off its big enterprise server systems.. John Gage, Sun’s Chief Technology Officer at the time, had other ideas. Gage’s keynote talk to the analysts after dinner was only about Mosaic and the big change it made in how one could use the Internet. From that point on, the conference was not about Sun Microsystems enterprise servers. It was about Mosaic and the Internet..

But to this day, the further away we get from California universities, the less pervasive are those skills among undergraduates, unless they motivate themselves to learn them on their own. Years ago, thinking of my own son, I began declaring that a world of “have’s” and have not’s” would emerge very soon: those with cyber skills and those without them, and the career consequences of that dichotomy were likely to be severe. As I am embarking on teaching Information Technology Management next semester, I am struck that much of the teaching material available has not yet caught up to this new world environment. Things are moving so fast, that it is almost inconceivable that a traditional print textbook could be written, reviewed, published and distributed before it was already obsolete. The very teaching of the topic implies the need to use the newest and most versatile online Web resources and hands on teaching methods.

My first shared video explores The Cloud and the problems of managing in the world of the Cloud.

[http://pro.gigaom.com/webinars/webinar-zenoss-managing-the-cloud/#ooid=s4ZTZvMjrFXhCd3-uaCC8H8ULCUUtRoe]

Seven Reasons Why Big Data is Worth Shifting a Career For

Seven Reasons Why Big Data is Worth Shifting a Career For.

Inside Google Spanner: The Single Largest Database in the World

http://www.wired.com/wiredenterprise/2012/11/google-spanner-time/all/

The fascinating description of Google’s Spanner, arguably the single largest “known” database in the World leads me to wonder about the National Security Administration and the Defense Intelligence Agency, as much as Big Data and data mining. Clearly, Spanner is a giant leap forward to a truly global database architecture that does not overload global network communication, and is essentially immune from replication latency or outages. The novel application of “time” is probably the key element. It also porttends further advances in massive data mining. Andrew Fikes’ paper on Spanner essentially makes it available to the World, which if you carefully consider the point of the architecture, it makes sense… It also seems a bit spooky.

Big Data: The Next Frontier in Competition, Innovation and Productivity

http://www.mckinsey.com/Insights/MGI/Research/Technology_and_Innovation/Big_data_The_next_frontier_for_innovation

This report from McKinsey adds to the list of similar reports coming from the HBR blog and numerous other sources, emphasizing that Big Data is here, and is going to have a huge impact on all business. What has become known as “Big Data,” refers to the terabytes and zettabytes (that’s a 1 followed by 21 zeros) of data that have been collected and stored on each one of us…for better or worse. Data mining is the industry that is emerging to make commercial sense of Big Data. It is a well-known fact that more data has been collected on us in the last two years, than in all of the previous years of computing combined.

Last week, PBS Newshour featured an election eve story about the Obama campaign‘s use of data mining. In the story, Obama election canvassers revealed how they were using Big Data and data mining to target their efforts. Rather than slog through neighborhoods knocking on every door, the Obama team had devised an ingenious field system using smartphones, GPS and mountains of data about the people living at each address. Each home that fit a precise profile that indicated their tendency to perhaps vote for Obama, or would be open to persuasion, was marked with a blue flag on the smartphone. The propensity to support Obama was determined by statisticians and mathematicians at Obama HQ, house by house, using a highly complex database of household information. This was not just income and age. It included information about magazines, organizations, that had already been shared publicly by the household. This dramatically improved the effectiveness and productivity of the field workers pounding on doors. From the election results, we now know that this system is worth its weight in gold.

Last term, UBC Faculty of Management 3rd year students investigated the emerging new “data mining” industry. We learned from example videos that data mining was being used to solve highly complex management, operational and marketing problems that had hitherto seemed unsolvable. Also, traditionally, assessment of industry trends had been highly subjective gut level judgments by expert researchers. Things like fashion trends, and video gaming preferences were thought to require mostly observation and guesswork. However, Harvard Business Review and other journals are seeing that Big Data is being mined to solve even these seemingly intractable problems.

The emergence of Big Data and data mining is essentially very similar to Chaos Theory, and the emergence of mathematical algorithms that solved the problem of apparent chaos in nature, which was discovered to actually follow an elegant order.

The Evolution of These Technologies Is Clearer

The IoT Tower of Proprietary Babble Is Slowly Crumbling

The Rise of the Intelligent Assistant

Share this:

Share this:

Share this:

Man and machine unite

Data science, or data intelligence?

Telling stories with data

The internet of things knows all

This semantic life

Share this:

Rising in unison

No bottom in sight

Not just a BYOD issue

The internet of things

Embracing the Entanglement

Share this:

Share this:

Share this:

Share this:

Share this:

Share this: