A Decision Tier for Data Capture Networks

I'm a big fan of Adron Hall and understand the nature of this piece. But, I respectfully disagree, at least to the initial premise.

So I’ve been in more than a few conversations about data structures, various academic conversations and other notions about where and how data should be stored. I’ve been on projects and managed projects that involve teams of people determining how to manage data so that other people can just not manage data. They want to focus on business use and not the data mechanisms underneath. The root of everything around databases really boils down to a single thing – how can we store X and retrieve X – nobody actually trying to get business done or change the world is going to dig into the data storage mechanisms if they don’t have to. To summarize,

nobody actually gives a shit…

At least nobody does until the database breaks, or somebody has to be hired to manage or tune queries or something or some other problem comes up.

The point at which I take exception is when considering data collection and distributed data processing in conjunction with the Internet of Things (IoT) and, particularly, Industrial Internet.  Frankly, I have not yet encountered any group (commercial, open source, ...) to whom I can attribute a complete grasp of the IoT topologies required for efficient (intelligent) data acquisition and its retrieval or analysis.

That said, I fully agree that, once understood, the questions of where best to place data and the rise of the Intelligent Data System is our objective.

What would happen if the systems storing the data knew where to put things? What would be the case for providing an intelligent indexing policy or architecture at the schema design decision layer, the area where a person usually must intervene? Could it be done?

A decision tier that scans and makes decisions on the data to revamp the way it is stored against a key value, geo, time series or other method. Could it be done in real time? Would it have to go through some type of processing system? The options around implementing something like this are numerous, but this just leaves a lot of space for providing value add around the data to reduce the complexity of this decision making.

Thanks, Adron, for mooting the notion of a 'decision tier'.  I'm planning on using this mercilessly in my contemplations of Data Capture Networks (DCNs) for Industrial Internet.



How We Killed Privacy Long Before PRISM

Foreign Policy has published an excellent review of US privacy and surveillance law and its evolution (if one can call it that) during an age of unprecedented change in the use of personal and mass media, and technology change at speeds which make legislative response look positively glacial.

The idea that it has suddenly and suprisingly been revealed that we have little or no privacy with the NSA revelations is, quite frankly, bogus.  It's a theme generated by an over-heated journalism (looking for eyeballs) and the unfortunate over-use of privacy threat by those organizations which legitimately are pushing back on abuse, have been for years, and now see an effective way of getting it in front of the general populace.  I find both disturbing and hypocritical


How We Killed Privacy -- in 4 Easy Steps - By Daveed Gartenstein-Ross and Kelsey D. Atherton | Foreign Policy:

Privacy in 2013 does not exist as we knew it in 2000.
But don't be fooled: The almost complete erosion of what we would have considered our private spaces at the beginning of this millennium is not entirely -- nor even mainly -- a result of the National Security Agency's surveillance. While nobody should doubt that the government's electronic spying is intrusive, we largely let online privacy slip away without any assistance from security agencies. Each step along the way was, for the most part, understandable and reasonable rather than nefarious. But the fact is that privacy in the United States is not what it used to be, and until we realize that, our debateSave & Close about electronic privacy -- Manichean as it is, and focused almost exclusively on the relationship between the government and its citizens -- will fail to resurrect its value.
Four distinct factors have interacted to kill electronic privacy: a legal framework that has remained largely static since the 1970s, significant changes in our use of rapidly evolving technology, commercial providers' increasingly intrusive tracking of our every online habit, and a growth in non-state threats that has made governments the world over obsess about uncovering these dangers. Only by understanding the interaction between these factors can we begin the necessary discussion about what privacy means in the 21st century -- and how to forge a new social compact to address the issue.



Growth Factors: IaaS Numbers and Expectations.

Lots of new IaaS reporting going on this week.  Gartner has a new Cloud Magic Quadrant, and Synergy Research Group has a new report on cloud market size and growth.  

The problem I have with the IaaS revenue numbers comes down to this: AWS hides them. IBM has been called on the carpet for being 'over-inclusive' in its reporting. The combination of IaaS/PaaS numbers into one figure for Microsoft and Google disguises the fact that MSFT and GOOG both started in Platform Services first, and have only recently offered up 'pure' infrastructure services. 

The important factors of IaaS/PaaS accelerated growth are these:

[1] Infrastructure Service adoption -- new customers and the expanded use by existing customers -- in all market segments and among all the enterpise size classes is explosive.  The opinion that "enterprise won't put their [data | processing] in the Cloud" is bogus.  We're all -- ALL -- using cloud infrastructures, and will continue to add new data and processing to the cloud as comfort levels increase.

[2] The growth in IaaS and PaaS will continue on this path, and probably see acceleration, as enterprise and government customers become 'comfortable' with the upcoming offerings for cloud data management (e.g., security, privacy, distribution, meta-data generation, …) and for 'cloud automation' that includes APPLICATIONs, not just infrastructure configuration management. (We really need the cloud punditry to get straight on what it refers to when using the term 'orchestration.')

[3] Finally, just as Infrastructure Services understand the value of giving development and operational IT communities self-service and the ability to 'have it their way', they will recognize the need for fine-grained, customized alternatives for consumption and economic terms.  That is, more choice in the packaging and economic terms on which Infrastructure Services are procured.

On this last point: I expect considerable push back from enterprise customers on the measly number of choices they have to procure on-demand IaaS at either (a) rack rates using a credit card or (b) 'reserved' use in one-year or three year packages.  That is  far too little operational and economic choice.  The same kind of late-binding, fine-grained control the characterizes cloud development, deployment and functionality needs to be available to the businsess functions of IaaS/PaaS procurement, metering and billing.

More proof that Amazon still leads the IaaS pack, but watch out for those other dogs — Tech News and Analysis:

Amazon leads Google, IBM, and Microsoft in cloud. So what else is new? The fact that the other guys are growing like weeds.



Is Hybrid Cloud a Coping Strategy for 20th Century Vendors

Interesting analysis from the FT subsequent to the release of Gartner's latest Cloud Magic Quadrant:

Companies such as VMware and IBM, having initially turned their noses up at the idea that they needed to build public clouds of their own, are racing to add this as an option. They cling to the hope that “hybrid clouds” are a halfway house that play to their strengths, as customers look to move their IT workloads seamlessly between their own systems and “overflow” networks operated by others.

Much of the money to be made in this new world comes from the software needed to manage the more complex infrastructure. Yet the high ground of cloud management software only has room for a handful of players. Meanwhile, the profit margins in technology markets – in server, networking and storage hardware, and in the software on which these machines depend – face remorseless compression.