Database Server : A Data Warehouse Server

Real-time data warehousing creates some special issues that need to be solved by data warehouse management. These can create issues because of the extensive technicality that is involved for not only planning the system, but also managing problems as they arise. Two aspects of the BI system that need to be organized in order to elude any technical problems are: the architecture design and query workload balancing.

Current trend in the world of information technology is that relatively every organization is managing tens of petabyte of data. There are large proportion of data which need to be store and managed in database. So there is immense requirement of efficient and reliable database management system. Database systems need to be constructed in high reliability methods and techniques in terms of their functionalities and design. System Performance is an analytical metric that must need great output for an effective database system. Complex database system is outrageous and difficult to analyze so performance evaluation is very important concern since databases are one of the most compelling affair in today’s business revolution.

Data Warehousing also known in many industries as an Enterprise Data Warehouse is a system that contains a central repository of integrated data, often collected from multiple sources and is used to perform data analysis enabling the creation of detailed reports that contribute significantly to a corporation’s business intelligence. Data Warehousing emerged as a result of advances in the field of information systems over the last several decades. There are two major factors that drive the need for data warehousing in most organizations. First and foremost, businesses require an integrated, company-wide view of high-quality information to maintain and improve upon their strategic position. Secondly, information systems departments must separate information from operational systems to improve performance dramatically in managing company data. Critical to the success of a Data Warehousing system, Data mining allows for companies to create customer profiles, manipulate information easily, and provide knowledgeable access to the current state of their company. However, a reality that many companies often find out the hard way is that data mining and data warehousing does not work for them. As with many new tools or technology, companies may jump on the bandwagon without fully contemplating its potential weaknesses. In order to remain competitive in today’s business world, companies should consider implementing data warehouses, but only with

Furthermore, the Gartner website argues that “BI has become a strategic initiative and is now recognised by chief information officers (CIOs) and business leaders as instrumental in driving business effectiveness and innovation,” (Anon., 2007). Gartner also argues that “BI projects were the number one technology priority for 2007” (Anon., 2007). According to the Bill Inmon, data warehouse is “a subject-oriented, integrated, time variant and non-volatile collection of data used in strategic decision making”. Hammergen & Simon, (2009) define data warehouse more simpler by saying that “ Data warehousing is therefore the process of creating an architected information management solution to enable analytical and information processing despite platform, application, organizational, and other barriers.“ It is important to note that data warehouse system is different from relational database. The reasons of that are: (1) In the data warehouse data is stored for long term; (2) DW is designed for high performance for analytical queries; (3) its OLAP (Online Analytical Processing) technology enables to view data in various form; (4) linking between tables are simple (Tushman, 2014). Databases, in contrast, have a low performance regarding data analysis; joins between tables are

A Big Data system is comprised of a number of functional blocks that provide the system the capability for acquiring data from diverse sources, pre-processing (e.g. cleansing and validating) this data, storing the data, processing and analyzing this stored data, and finally presenting and visualizing the summarized and aggregated results. The rest of this article describes various performance considerations for each of the components shown in Fig 1. Refer: Appendices

In this section we descus the expected properties of a system designed for performing data analysis at the cloud environment and how parallel database systems and MapReduce-based systems achieve these properties.

Data has always been analyzed within companies and used to help benefit the future of businesses. However, the evolution of how the data stored, combined, analyzed and used to predict the pattern and tendencies of consumers has evolved as technology has seen numerous advancements throughout the past century. In the 1900s databases began as “computer hard disks” and in 1965, after many other discoveries including voice recognition, “the US Government plans the world’s first data center to store 742 million tax returns and 175 million sets of fingerprints on magnetic tape.” The evolution of data and how it evolved into forming large databases continues in 1991 when the internet began to pop up and “digital storage became more cost effective than paper. And with the constant increase of the data supplied digitally, Hadoop was created in 2005 and from that point forward there was “14.7 Exabytes of new information are produced this year" and this number is rapidly increasing with a lot of mobile devices the people in our society have today (Marr). The evolution of the internet and then the expansion of the number of mobile devices society has access to today led data to evolve and companies now need large central Database management systems in order to run an efficient and a successful business.

The purpose of a data warehouse is to make the company’s information accessible and consistent. They need to have the information immediately available and in the same format. Warehousing is of no benefit to a company if they have to wait any length of time to receive the data. A warehouse has to be an adaptive and durable source of information for the business. The warehouse has to be flexible to meet the needed changes of a business, as the business grows; it is possible that additional information will need to be collected. The warehouse needs to have the ability to expand to meet the needs of the business. Warehousing would not be beneficial to a business if they have to seek a new warehouse source each time a change was needed; it would be costly for a business. A data warehouse must be a secure stronghold that protects the information, which is regarded as an asset to the business. In today’s society it this the utmost concerned of a business to make sure that their systems are not easily hacked by outsiders and their customer’s data is secured. Lastly, a warehouse is considered the foundation for decision making. It is the data that is retrieved from the system that is compiled for presentation to the decision makers of the company.

A Data WareHouse is a type of database normally used by large companies to store large amounts of data in and have the data be easily accessible. They are normally set up in one of three set-ups. The basic model that takes data straight from it sources, such as operational systems and flat files. The Staging Mode that has a staging area that takes the data, from the systems and files before moving it to data warehouse. The Final type adds data marts, a small database that takes specific information from the data warehouse, to the previous model between the data warehouse and the end users. Data Warehouses are also really useful because they make it easy to pull data from either queries or data mining. Data warehouses are a useful tool when dealing with large amounts of data.

Recent advancements in internet communication and in parallel computing grabbed the attention of a large number of commercial organizations and industries to adapt the recent changes in storage and retrieval methods. This includes the new data retrieval and mining schemas which enable the firms to provide their clients a wide space for carrying their job processing and storing of the personal data. Although the new storage innovations made the user data to accommodate the petabyte scale in size, the storing schemas are still on the research desk to compete with this adaptation. Some of the new research outcomes which gained a high popularity and become the need of the hour is the Hadoop. Hadoop is developed by Apache based on the papers of

Abstract—Parallel databases are the high performance databases in RDBMS world that can used for setting up data intensive enterprise data warehouse but they lack scalability whereas, MapReduce paradigm highly supports scalability, nevertheless cannot perform as good as parallel databases. Deriving an architectural hybrid model of best of both worlds that can support high performance and scalability at the same time.

Especially with the growth of web usage, there has come a time to relook into the

However, databases have to meet some challenges. Nowadays, the purpose of databases is to serve the demands of large scale companies. These

The organization has decided to keep only 500 GB of data in the on-premises data warehouse, seeing the increase of 10 GB of data per day. Rest of the data is all moved to Amazon Redshift. This has saved lots of costs from buying expensive on-premises systems and there was a significant improvement in the performance of the systems.

Before discussing the current data warehouse architecture in place at ICICI Bank, issues associated with it, especially due to immense data growth and different modalities of data sources, it would be appropriate to have a quick look at the data warehouse history and architectural framework and how ICICI Bank’s data warehouse has evolved over the years. Back in 2008 ICICI Bank used Teradata and was dependent on Teradata for its data warehouse. Back in those days the size of the data warehouse was 3TB. Because of the dramatic growth in the amount of data, user population and the source stations coupled with cost of scaling and maintenance as well as system availability,posed a problem for the bank in using their legacy data warehouse solution. The bank felt that its legacy data warehouse solution posed scalability issues and one of the major issues that bank faced was with their current