It is without doubt that agility is synonymous with fast-paced turnaround time. The speed at which IT personnel and processes should respond to user requirements, either new or enhancements, is becoming the main factor between winning, keeping or losing customers in this digital era.
IT professionals are finding they are confronted with this reality due to accelerated technological innovations, coupled with numerous sources and data types to consider, as well as more demanding users to support. Meeting these new and ever-evolving data demands requires companies to create a data infrastructure and architecture that is agile enough to keep pace with the market, as well as the needs of the company to respond to swift opportunities.
As data capabilities continue to mature, how a business structures its data architecture for agility becomes imperative. Data architecture is one of the key pillars of enterprise architecture, alongside application, business and integration architecture.
In fact, the data architecture pillar is the definition or blueprint of the data design that will be used by the business in achieving the implementation of a physical database, as well as describing the way data will be processed, stored and used.
Where to?
Relational databases and data warehouses have served businesses well in the past by collecting and normalising relational data where the data format and structure is known and doesn't change frequently. However, the relational model and process for defining schema in advance cannot keep pace with the rapidly evolving variety and format of data today. The integral Waterfall design and modelling methodology means users have to wait for long before they can realise some value from the life cycle. At times, the data analysts just want to mine the data for possible insights; however, they are not sure of what questions to ask to ensure the data is truly beneficial. Legacy databases are simply not agile enough to meet the growing needs of most companies today.
As a result of this, an attempt was made to invoke agility by implementing legacy data models in smaller sprints. This worked well as long as the end state was known or predictable and there was no need to respond to changing business environments. However, these are key benefits of agility and cannot be omitted! Within the Waterfall model, once the architecture is implemented, enhancement agility can be achieved, provided the new or changed user needs will be within the boundaries anticipated during the requirements phase.
"What is required for the digital era is a scalable integration software built for modern data environments, users, styles and workflow - from batch and bulk to IOT data streams and real-time capabilities - in other words, an agile data fabric," says Mike Tuchen, CEO at Talend.
Recent technology innovations permit large volumes of data (at a low cost) to be ingested into the business environment and reserve the structuring and modelling for later during consumption. This was the initial agility handle for data architecture. However, the conversation has changed to the speed of which one can extract value from the available data and how quickly the data can be translated from information into actionable intelligence. Getting data into the business environment easily does not mean a data analyst can just as easily and quickly get insight from the data out.
Mix it up
Arguably, I find the best data architecture for agility includes a mix of on-premises and cloud physical data architecture. It has been noted the mix can also consider a distributed implementation focusing on domain-specific knowledge pillars with solid security levels. It can then be complemented by a logical model that presents a logical consolidated front for analysis, without moving data into a single location.
Companies should strive to remain technology agnostic by choosing data processing technologies that can seamlessly move processes and logic into a different domain in just a few clicks and minor configurations. All the work executed for one technology should be easily transferable to the next, providing the organisation with economies of skills and scale. The legacy modelling techniques still have a handful of use cases that may warrant a space in a business's data architecture landscape for those well-known structured data.
The promise of faster time to market, better management of change and disruption, and leaner, more efficient operations is the trademark for true agility. If agility has any shortfall, it will be the lack of adequate focus on dependency analysis; however, the benefits outweigh the drawbacks. It's almost impossible to conclude on the topic of agility fabric without dropping some big names; however, I am resisting!
Share