Data is everywhere, but we already know it. What we might not know or always think about is the plumbing that goes on under the data services that we all connect to every day.
In today’s age of smart toasters and smart doorbells, we are drinking in very real terms like a fire hose in terms of the amount of data ingested. What matters now is that we are not trying to drink from the data fire hose if it is fed by a âchokedâ low level information architecture; the risk of pipes cracking and creating a flood or gush that drowns us is (or at least should be) too great a risk to take.
What is stove pipe technology?
So what do we mean by this term silo technology? In both real world engineering and software engineering, a stovepipe system is a kind of single chimney that is only able to push its flow (smoke, water, or data) into a single channel. A stove pipe works by itself, individually. It does not share any connection with other pipes, channels, or exhausts, so it is more likely to be overloaded, cracked, or otherwise compromised.
If you let your applications drink in the fire hose of modern data ubiquity, you don’t want them exposed to all this power with the fragility of an archaic system underneath.
âStovepipes are a good way to support unique assignments, such as automating a single business process or reporting the activities of a sales team. But to be truly insightful, you need a complete picture of all your business processes and teams. In the world of the cloud, web, and modern apps, the mantra that I fear should be (and cover your ears if you have to) bulkheads, said Shawn Rogers, vice president of strategy. analytics at TIBCO.
The real problem here for contemporary technology platforms is the fact that many organizations will be running legacy relational databases with siled architectures. In contrast, the cloud computing model of virtualized and abstract interconnected systems is almost the antithesis.
The stove pipes have a single channel. In contrast, the cloud has a potentially infinite number of nodes and scalable channels distributed throughout its architectural makeup.
âWhile the cloud has an inherent technological advantage, the data flowing through it also needs to be harmonized. Take customer data in sales and marketing or accounts and receivables. If these data sources are siled, they stay in the dark. However, to delight your customers and every time you engage with them, you must first align your customer data using master data management to provide a clear channel through which businesses can breathe, âadded Rogers of TIBCO.
An impedance offset
The end result of siled legacy data structures attempting to serve higher level cloud-based expansion is an impedance mismatch, the two constructs cannot exist in the modern computing universe without leakage, blockage, or disastrous disruption.
This subject is close to the heart of Jim Walker. In his role as Vice President of Product Marketing at Cockroach Labs, Walker speaks out on the use of cloud-based data orchestration technologies, such as the popular Kubernetes.
Walker reminds us that old data systems were not designed to work in the distributed information universe in which we exist today. Worse yet, although organizations attempt to modify, strengthen, and change these legacy systems, the inevitable result is the creation of a bottleneck, or worse, a single point of failure for the application.
âRunning a NoSQL database on Kubernetes is a better alignment that may help overcome some of the challenge here, but organizations will still face transactional consistency issues. The siled architecture of our legacy relational databases contradicts the scalable distributed architecture of Kubernetes, as they were not built with the same architectural primitives, âWalker said.
A contained response to the challenge
Much of the tech industry has seen containerization as a way to build the next era of cloud-native technologies, but what everyone has realized is that when applications are containerized (i.e. (say built from discrete smaller individual components), our databases must be too. Are legacy databases built with the kind of modern modular interchangeability and connectivity that contemporary software application development engineers are capable of using? Well, in a nutshell, no.
Walker of Cockroach Labs advises that at the end of the day, a future-oriented, future-proof database should look like a traditional database, while simultaneously enjoying all the benefits of cloud infrastructure. .
âLike the parts in a car, you can replace some parts but the rest of the car is the same. This is the type of architecture developers and businesses need to think about in order to effectively develop, deploy and maintain software containers, âhe said.
From stove pipes to internet pipes
By thinking about the way forward here, we can use much of what we have learned in the silo era (the last two decades of the last millennium … and perhaps further back in some cases) as logic absolute fundamental of data movement is still there. . In tech circles we still call the internet a pipe and we need it to be just as pipe-like as a stove pipe, we just need it to be smarter and stronger.
In the future, maybe we can think about opening wide and drinking the data fire hose with bigger sips, the stove pipes will collapse, and new plumbing is on the way.