History of Data Warehousing
Last Updated : 17 Dec, 2021
The data warehouse is a core repository that performs aggregation to collect and group data from various sources into a central integrated unit. The data from the warehouse can be retrieved and analyzed to generate reports or relations between the datasets of the database which enhances the growth of many industries. Data warehouse comes under Business Intelligence. The data warehouse is designed to provide real-time information. Storage of data has evolved from simple magnetic tapes to integrated data warehouses. This article will give an overview of the history of warehousing.
Early mechanisms to store data:
Early methods to hold data started with punched cards, paper tapes. Then the development of magnetic tapes took place. Though we can write and rewrite data in magnetic tapes, it is not a stable medium to hold data. Disk storage came into existence where you can store and access large amounts of data.
DBMS in disk storage:
Later DBMS (Database Management Systems) was integrated with disk storage to store the data directly on the disk itself. The main advantage of integrating DBMS is we can locate the data fast. The features include location and deletion of data, solving problems when two different data are mapped to the same location. The physical location can be extended when the data exceeds the storage limit.
Online Applications:
There came the advent of online applications after the usage of DBMS in disk storage. Online applications are the products of online processing which has its applications in the commercial industry. For e.g. Retail and sales processing, ticket reservation systems, Automated Teller processing, etc. Online applications play an important role in the current years due to their intertwined applications. But it has a drawback which the end-users of the application put forward. Since there is an enormous amount of data, the end-users find it difficult to retain the desired data. Even if they obtained it, they are not sure whether it is correct or accurate due to the constant escalation of data.
Fourth Generation Technologies (4GL) and Personal computers:
The motive of 4GL technology is to provide end-users the direct opportunity of accessing data, using the programming languages and system development without the interference of the IT department. The same happens with personal computers. So, the individuals can bring their own personalized systems into the business firm and can access the specific data accessible to them. This reduced the need for a centralized department of technology to provide the requested data to the users. Spreadsheets are a good example. But it has its drawbacks. The data retrieved may be incomplete, misleading, or wrong. It lacks finesse in the end result due to the lack of documentation and the existence of multiple versions of the same data.
Spider web environment:
The Spider web environment ended up as a dilemma to end-users, IT professionals due to its unfavorable nature and complexity. This environment is called spider web environment because there are many lines connecting which reminded the lines of a spider web. Though data can be retrieved, the efficiency and accuracy are very less. These severe drawbacks called the need for building information architecture centering data warehouse.
Evolution of Data warehouse environment:
As the corporation shifted from spider web to data warehouse environment, it created a major change in the usual techniques in which data is stored. Before the introduction of data warehouse, it was thought that a database must aid all the purposes of data. After the advent of data warehouse, it is evident that there are different types of databases that serve for different purposes.
A data warehouse is a place where information is processed into bot integrated and granular forms of data and history. Though not all the warehouses are integrated, integrated data warehouses has its benefit of providing the enterprise view of a company. The granular data has the benefit of looking same data in different ways. A set of data can be looked in marketing way or it can be looked in finance way. The same data can be used to look in accounting way too. Data warehouses are used to store historical data of many years.
Challenges of data warehouse:
- First is integration of data which is the most difficult and time consuming process as one need to touch the root of old legacies of corporates to derive useful integrated data. It is a painful step, but it is worthful.
- High volume of data created by data warehousing techniques which makes the process tedious. So, there comes a need to get rid of old data. But, for analyses of data they are so valuable and can't be ignored.
- Data warehouses can't be created all at once like other operational applications. It must be developed iteratively, like one step at a time.
Reasons for the development of Data Warehouse 2.0 Environment (DW 2.0):
The earlier techniques have evolved so much and ended as DW 2.0. We need to travel back and forth to understand the forces that initiated the architecture of DW 2.0. Some of them are given below.
- The end-user's demand for new system or architecture.
- Financially economic
- Online processing techniques
- High storage capacity
- Need for integrated data
- The need to included unstructured data for analytics purpose in the data mixture.
Data Warehouse evolution (from business perspective):
- The output of the earlier techniques is in unrefined format. For e.g. It is a hectic process to read all those hexadecimal inputs just to find a small piece of information from the cryptic codes.
- Now, the end-users have become more futuristic. So, they demand the need for more sophisticated output and instantaneous source of output.
- For online processing techniques to be done, the data need to be integrated. Also, it needs historical data for analysis.
- First generation data warehouse came into existence due to the end-user's thirst for corporate data.
Mutated forms of data warehouse:
Due to the appealing features of data warehouse, the business consultants have mutated the concept of data warehouse in accordance to their corporate needs. Some variations of the data warehouses are:
- The Active data warehouse: The online processing and updates are taken place in this warehouse. The major feature of this warehouse is that the transaction has a very high performance rate. The demerits of this mutated warehouse are the uprightness of the transaction is questioned, hefty statistical processing, large capacities are wasted which in turn increases the operational cost.
- The Federated data warehouse: In this type of approach, due to the high complexity in the integration of data, they skip this process. Technically, a warehouse doesn't exist in this approach. The scheme behind that is to construct a data warehouse magically by merging the old legacies of the corporate to fetch and process data simultaneously. This approach seems attractive with less work but it is just a delusion rather than a solution itself. It has numerous pitfalls like bad performance, limited history, absence of data integration, complexity, inherited granularity which provides poor performance to the end-user when he requests data from different level of granularity from the federal warehouse.
- The Star schema data warehouse: The outlook used in this data warehousing needs the construction of dimension tables and fact tables. It provides lot of benefits as a data warehouse but has its limitations. It is designed only for limited requirements and when the requirements change, the data warehouse becomes brittle. The level of granularity keeps changing due to multiple schema formation which questions the integrity of data. It cannot be extended more than a certain limit and it is designed only for one-audience type.
- Data Mart data warehouse: The consultants of the online application processing first build a data mart which gives the chance to know the sales of the product without any complications of building an actual data warehouse. The demerits include non- extensibility, high error occurrences, reconciliation of data is not possible and extract proliferation which makes extraction of legacy data difficult. Another fact about this approach is there is no way a data mart can be converted into a data warehouse. It's like the core of each is different and they can't be mutated to change into warehouse.
Similar Reads
Data Warehousing Tutorial Data warehousing refers to the process of collecting, storing, and managing data from different sources in a centralized repository. It allows businesses to analyze historical data and make informed decisions. The data is structured in a way that makes it easy to query and generate reports.A data wa
2 min read
Basics of Data Warehousing
Data WarehousingA data warehouse is a centralized system used for storing and managing large volumes of data from various sources. It is designed to help businesses analyze historical data and make informed decisions. Data from different operational systems is collected, cleaned, and stored in a structured way, ena
10 min read
History of Data WarehousingThe data warehouse is a core repository that performs aggregation to collect and group data from various sources into a central integrated unit. The data from the warehouse can be retrieved and analyzed to generate reports or relations between the datasets of the database which enhances the growth o
7 min read
Data Warehouse ArchitectureA Data Warehouse is a system that combine data from multiple sources, organizes it under a single architecture, and helps organizations make better decisions. It simplifies data handling, storage, and reporting, making analysis more efficient. Data Warehouse Architecture uses a structured framework
10 min read
Difference between Data Mart, Data Lake, and Data WarehouseA Data Mart, Data Lake, and Data Warehouse are all used for storing and analyzing data, but they serve different purposes. A Data Warehouse stores structured, processed data for reporting, a Data Lake holds raw, unstructured data for flexible analysis, and a Data Mart is a smaller, focused version o
5 min read
Data Loading in Data warehouseThe data warehouse is structured by the integration of data from different sources. Several factors separate the data warehouse from the operational database. Since the two systems provide vastly different functionality and require different types of data, it is necessary to keep the data database s
5 min read
OLAP Technology
Data Warehousing Model
Data Modeling Techniques For Data WarehouseData modeling is the process of designing a visual representation of a system or database to establish how data will be stored, accessed, and managed. In the context of a data warehouse, data modeling involves defining how different data elements interact and how they are organized for efficient ret
5 min read
Difference between Fact Table and Dimension TableIn information warehousing, fact tables and Dimension tables are major parts of a star or snowflake composition. Fact tables store quantitative information and measurements, for example, income or request amounts, which are commonly accumulated for examination. These tables are described by their nu
8 min read
Data Modeling Techniques For Data WarehouseData modeling is the process of designing a visual representation of a system or database to establish how data will be stored, accessed, and managed. In the context of a data warehouse, data modeling involves defining how different data elements interact and how they are organized for efficient ret
5 min read
Concept Hierarchy in Data MiningPrerequisites: Data Mining, Data Warehousing Data mining refers to the process of discovering insights, patterns, and knowledge from large data. It involves using techniques from fields such as statistics, machine learning, and artificial intelligence to extract insights and knowledge from data. Dat
7 min read
Data Transformation
What is Data Transformation?Data transformation is an important step in data analysis process that involves the conversion, cleaning, and organizing of data into accessible formats. It ensures that the information is accessible, consistent, secure, and finally recognized by the intended business users. This process is undertak
6 min read
Data Normalization in Data MiningData normalization is a technique used in data mining to transform the values of a dataset into a common scale. This is important because many machine learning algorithms are sensitive to the scale of the input features and can produce better results when the data is normalized. Normalization is use
5 min read
Aggregation in Data MiningAggregation in data mining is the process of finding, collecting, and presenting the data in a summarized format to perform statistical analysis of business schemes or analysis of human patterns. When numerous data is collected from various datasets, it's important to gather accurate data to provide
7 min read
DiscretizationDiscretization is the process of converting continuous data or numerical values into discrete categories or bins. This technique is often used in data analysis and machine learning to simplify complex data and make it easier to analyze and work with. Instead of dealing with exact values, discretizat
3 min read
What is Data Sampling - Types, Importance, Best PracticesData Sampling is a statistical method that is used to analyze and observe a subset of data from a larger piece of dataset and configure all the required meaningful information from the subset that helps in gaining information or drawing conclusion for the larger dataset or it's parent dataset. Sampl
9 min read
Difference Between Feature Selection and Feature ExtractionFeature selection and feature extraction are two key techniques used in machine learning to improve model performance by handling irrelevant or redundant features. While both works on data preprocessing, feature selection uses a subset of existing features whereas feature extraction transforms data
2 min read
Introduction to Dimensionality ReductionWhen working with machine learning models, datasets with too many features can cause issues like slow computation and overfitting. Dimensionality reduction helps to reduce the number of features while retaining key information. Techniques like principal component analysis (PCA), singular value decom
4 min read
Advanced Data Warehousing
Measures in Data Mining - Categorization and ComputationIn data mining, Measures are quantitative tools used to extract meaningful information from large sets of data. They help in summarizing, describing, and analyzing data to facilitate decision-making and predictive analytics. Measures assess various aspects of data, such as central tendency, variabil
5 min read
Rules For Data Warehouse ImplementationA data warehouse is a central system where businesses store and organize data from various sources, making it easier to analyze and extract valuable insights. It plays a vital role in business intelligence, helping companies make informed decisions based on accurate, historical data. Proper implemen
5 min read
How To Maximize Data Warehouse PerformanceData warehouse performance plays a crucial role in ensuring that businesses can efficiently store, manage and analyze large volumes of data. Optimizing the performance of a data warehouse is essential for enhancing business intelligence (BI) capabilities, enabling faster decision-making and providin
6 min read
Top 15 Popular Data Warehouse ToolsA data warehouse is a data management system that is used for storing, reporting and data analysis. It is the primary component of business intelligence and is also known as an enterprise data warehouse. Data Warehouses are central repositories that store data from one or more heterogeneous sources.
11 min read
Data Warehousing SecurityData warehousing is the act of gathering, compiling, and analyzing massive volumes of data from multiple sources to assist commercial decision-making processes is known as data warehousing. The data warehouse acts as a central store for data, giving decision-makers access to real-time data analysis
7 min read
Practice