Tryangled Dev

Data warehouse is a repository of an organization’s electronically stored data. Data warehouses are designed to facilitate reporting and analysis.

This definition of the data warehouse focuses on data storage. However, the means to retrieve and analyze data, to extract, transform and load data, and to manage the data dictionary are also considered essential components of a data warehousing system. Many references to data warehousing use this broader context. Thus, an expanded definition for data warehousing includes business intelligence tools, tools to extract, transform, and load data into the repository, and tools to manage and retrieve metadata.

In contrast to data warehouses are operational systems that perform day-to-day transaction processing.
History of data warehousing

The concept of data warehousing dates back to the late 1980s [2] when IBM researchers Barry Devlin and Paul Murphy developed the “business data warehouse”. In essence, the data warehousing concept was intended to provide an architectural model for the flow of data from operational systems to decision support environments. The concept attempted to address the various problems associated with this flow - mainly, the high costs associated with it. In the absence of a data warehousing architecture, an enormous amount of redundancy of information was required to support the multiple decision support environments that usually existed. In larger corporations it was typical for multiple decision support environments to operate independently. Each environment served different users but often required much of the same data. The process of gathering, cleaning and integrating data from various sources, usually long existing operational systems (usually referred to as legacy systems), was typically in part replicated for each environment. Moreover, the operational systems were frequently reexamined as new decision support requirements emerged. Often new requirements necessitated gathering, cleaning and integrating new data from the operational systems that were logically related to prior gathered data.

Based on analogies with real-life warehouses, data warehouses were intended as large-scale collection/storage/staging areas for corporate data. Data could be retrieved from one central point or data could be distributed to “retail stores” or “data marts” that were tailored for ready access by users.

Key developments in early years of data warehousing were:

* 1960s - General Mills and Dartmouth College, in a joint research project, develop the terms dimensions and facts.[3]
* 1970s - ACNielsen and IRI provide dimensional data marts for retail sales.[3]
* 1983 - Teradata introduces a database management system specifically designed for decision support.
* 1988 - Barry Devlin and Paul Murphy publish the article An architecture for a business and information systems in IBM Systems Journal where they introduce the term “business data warehouse”.
* 1990 - Red Brick Systems introduces Red Brick Warehouse, a database management system specifically for data warehousing.
* 1991 - Prism Solutions introduces Prism Warehouse Manager, software for developing a data warehouse.
* 1991 - Bill Inmon publishes the book Building the Data Warehouse.
* 1995 - The Data Warehousing Institute, a for-profit organization that promotes data warehousing, is founded.
* 1996 - Ralph Kimball publishes the book The Data Warehouse Toolkit.
* 1997 - Oracle 8, with support for star queries, is released.

Data warehouse architecture

Architecture, in the context of an organization’s data warehousing efforts, is a conceptualization of how the data warehouse is built. There is no right or wrong architecture. The worthiness of the architecture can be judged in how the conceptualization aids in the building, maintenance, and usage of the data warehouse.

One possible simple conceptualization of a data warehouse architecture consists of the following interconnected layers:

Operational database layer
The source data for the data warehouse - An organization’s ERP systems fall into this layer.
Informational access layer
The data accessed for reporting and analyzing and the tools for reporting and analyzing data - Business intelligence tools fall into this layer. And the Inmon-Kimball differences about design methodology, discussed later in this article, have to do with this layer.
Data access layer
The interface between the operational and informational access layer - Tools to extract, transform, load data into the warehouse fall into this layer.
Metadata layer
The data directory - This is usually more detailed than an operational system data directory. There are dictionaries for the entire warehouse and sometimes dictionaries for the data that can be accessed by a particular reporting and analysis tool.

Normalized versus dimensional approach for storage of data

There are two leading approaches to storing data in a data warehouse - the dimensional approach and the normalized approach.

In the dimensional approach, transaction data are partitioned into either “facts”, which are generally numeric transaction data, or “dimensions”, which are the reference information that gives context to the facts. For example, a sales transaction can be broken up into facts such as the number of products ordered and the price paid for the products, and into dimensions such as order date, customer name, product number, order ship-to and bill-to locations, and salesperson responsible for receiving the order. A key advantage of a dimensional approach is that the data warehouse is easier for the user to understand and to use. Also, the retrieval of data from the data warehouse tends to operate very quickly. The main disadvantages of the dimensional approach are: 1) In order to maintain the integrity of facts and dimensions, loading the data warehouse with data from different operational systems is complicated, and 2) It is difficult to modify the data warehouse structure if the organization adopting the dimensional approach changes the way in which it does business.

In the normalized approach, the data in the data warehouse are stored following, to a degree, database normalization rules. Tables are grouped together by subject areas that reflect general data categories (e.g., data on customers, products, finance, etc.) The main advantage of this approach is that it is straightforward to add information into the database. A disadvantage of this approach is that, because of the number of tables involved, it can be difficult for users both to 1) join data from different sources into meaningful information and then 2) access the information without a precise understanding of the sources of data and of the data structure of the data warehouse.

These approaches are not mutually exclusive. Dimensional approaches can involve normalizing data to a degree.

Conforming information

Another important facet in designing a data warehouse is which data to conform and how to conform the data. For example, one operational system feeding data into the data warehouse may use “M” and “F” to denote sex of an employee while another operational system may use “Male” and “Female”. Though this is a simple example, much of the work in implementing a data warehouse is devoted to making similar meaning data consistent when they are stored in the data warehouse. Typically, extract, transform, load tools are used in this work.

Master Data Management has the aim of conforming data that could be considered “dimensions”.

Top-down versus bottom-up design methodologies

Bottom-up design

Ralph Kimball, a well-known author on data warehousing, [4] is a proponent of an approach frequently considered as bottom-up [5], to data warehouse design. In the so-called bottom-up approach data marts are first created to provide reporting and analytical capabilities for specific business processes. Data marts contain atomic data and, if necessary, summarized data. These data marts can eventually be unioned together to create a comprehensive data warehouse. The combination of data marts is managed through the implementation of what Kimball calls “a data warehouse bus architecture”.[6]

Business value can be returned as quickly as the first data marts can be created. Maintaining tight management over the data warehouse bus architecture is fundamental to maintaining the integrity of the data warehouse. The most important management task is making sure dimensions among data marts are consistent. In Kimball words, this means that the dimensions “conform”.

Top-down design

Bill Inmon, one of the first authors on the subject of data warehousing, has defined a data warehouse as a centralized repository for the entire enterprise.[6] Inmon is one of the leading proponents of the top-down approach to data warehouse design, in which the data warehouse is designed using a normalized enterprise data model. “Atomic” data, that is, data at the lowest level of detail, are stored in the data warehouse. Dimensional data marts containing data needed for specific business processes or specific departments are created from the data warehouse. In the Inmon vision the data warehouse is at the center of the “Corporate Information Factory” (CIF), which provides a logical framework for delivering business intelligence (BI) and business management capabilities. The CIF is driven by data provided from business operations

Inmon states that the data warehouse is:

Subject-oriented
The data in the data warehouse is organized so that all the data elements relating to the same real-world event or object are linked together.
Time-variant
The changes to the data in the data warehouse are tracked and recorded so that reports can be produced showing changes over time.
Non-volatile
Data in the data warehouse is never over-written or deleted - once committed, the data is static, read-only, and retained for future reporting.
Integrated
The data warehouse contains data from most or all of an organization’s operational systems and this data is made consistent.

The top-down design methodology generates highly consistent dimensional views of data across data marts since all data marts are loaded from the centralized repository. Top-down design has also proven to be robust against business changes. Generating new dimensional data marts against the data stored in the data warehouse is a relatively simple task. The main disadvantage to the top-down methodology is that it represents a very large project with a very broad scope. The up-front cost for implementing a data warehouse using the top-down methodology is significant, and the duration of time from the start of project to the point that end users experience initial benefits can be substantial. In addition, the top-down methodology can be inflexible and unresponsive to changing departmental needs during the implementation phases.[6]

Hybrid design

Over time it has become apparent to proponents of bottom-up and top-down data warehouse design that both methodologies have benefits and risks. Hybrid methodologies have evolved to take advantage of the fast turn-around time of bottom-up design and the enterprise-wide data consistency of top-down design.

Data warehouses versus operational systems

Operational systems are optimized for preservation of data integrity and speed of recording of business transactions through use of database normalization and an entity-relationship model. Operational system designers generally follow the Codd rules of data normalization in order to ensure data integrity. Codd defined five increasingly stringent rules of normalization. Fully normalized database designs (that is, those satisfying all five Codd rules) often result in information from a business transaction being stored in dozens to hundreds of tables. Relational databases are efficient at managing the relationships between these tables. The databases have very fast insert/update performance because only a small amount of data in those tables is affected each time a transaction is processed. Finally, in order to improve performance, older data are usually periodically purged from operational systems.

Data warehouses are optimized for speed of data retrieval. Frequently data in data warehouses are denormalised via a dimension-based model. Also, to speed data retrieval, data warehouse data are often stored multiple times - in their most granular form and in summarized forms called aggregates. Data warehouse data are gathered from the operational systems and held in the data warehouse even after the data has been purged from the operational systems.

Evolution in organization use of data warehouses

Organizations generally start off with relatively simple use of data warehousing. Over time, more sophisticated use of data warehousing evolves. The following general stages of use of the data warehouse can be distinguished:

Off line Operational Database
Data warehouses in this initial stage are developed by simply copying the data of an operational system to another server where the processing load of reporting against the copied data does not impact the operational system’s performance.
Off line Data Warehouse
Data warehouses at this stage are updated from data in the operational systems on a regular basis and the data warehouse data is stored in a data structure designed to facilitate reporting.
Real Time Data Warehouse
Data warehouses at this stage are updated every time an operational system performs a transaction (e.g., an order or a delivery or a booking.)
Integrated Data Warehouse
Data warehouses at this stage are updated every time an operational system performs a transaction. The data warehouses then generate transactions that are passed back into the operational systems.

Benefits of data warehousing

Some of the benefits that a data warehouse provides are as follows: [7][8]

* A data warehouse provides a common data model for all data of interest regardless of the data’s source. This makes it easier to report and analyze information than it would be if multiple data models were used to retrieve information such as sales invoices, order receipts, general ledger charges, etc.
* Prior to loading data into the data warehouse, inconsistencies are identified and resolved. This greatly simplifies reporting and analysis.
* Information in the data warehouse is under the control of data warehouse users so that, even if the source system data is purged over time, the information in the warehouse can be stored safely for extended periods of time.
* Because they are separate from operational systems, data warehouses provide retrieval of data without slowing down operational systems.
* Data warehouses can work in conjunction with and, hence, enhance the value of operational business applications, notably customer relationship management (CRM) systems.
* Data warehouses facilitate decision support system applications such as trend reports (e.g., the items with the most sales in a particular area within the last two years), exception reports, and reports that show actual performance versus goals.

Disadvantages of data warehouses

There are also disadvantages to using a data warehouse. Some of them are:

* Over their life, data warehouses can have high costs. The data warehouse is usually not static. Maintenance costs are high.
* Data warehouses can get outdated relatively quickly. There is a cost of delivering suboptimal information to the organization.
* There is often a fine line between data warehouses and operational systems. Duplicate, expensive functionality may be developed. Or, functionality may be developed in the data warehouse that, in retrospect, should have been developed in the operational systems and vice versa..

Sample Applications

Some of the applications data warehousing can be used for are:

* Credit card churn analysis
* Insurance fraud analysis
* Call record analysis
* Logistics management.

The future of data warehousing

Data warehousing, like any technology niche, has a history of innovations that did not receive market acceptance.[9]

A 2009 Gartner Group paper predicted these developments in business intelligence/data warehousing market .[10]

* Because of lack of information, processes, and tools, through 2012, more than 35 per cent of the top 5,000 global companies will regularly fail to make insightful decisions about significant changes in their business and markets.
* By 2012, business units will control at least 40 per cent of the total budget for business intelligence.
* By 2010, 20 per cent of organizations will have an industry-specific analytic application delivered via software as a service as a standard component of their business intelligence portfolio.
* In 2009, collaborative decision making will emerge as a new product category that combines social software with business intelligence platform capabilities.
* By 2012, one-third of analytic applications applied to business processes will be delivered through coarse-grained application mashups.

Unix Commands
How to use the unix commands listed below.
For more documentation on a command, consult a good book, or use the man pages. For example, for more information on grep, use the command man grep.

Contents

cat — for creating and displaying short files
chmod — change permissions
cd — change directory
cp — for copying files
date — display date
echo — echo argument
ftp — connect to a remote machine to download or upload files
grep — search file
head — display first part of file
ls — see what files you have
lpr — standard print command (see also print )
more — use to read files
mkdir — create directory
mv — for moving and renaming files
ncftp — especially good for downloading files via anonymous ftp.
print — custom print command (see also lpr )
pwd — find out what directory you are in
rm — remove a file
rmdir — remove directory
rsh — remote shell
setenv — set an environment variable
sort — sort file
tail — display last part of file
tar — create an archive, add or extract files
telnet — log in to another machine
wc — count characters, words, lines

cat

This is one of the most flexible Unix commands. We can use to create, view and concatenate files. For our first example we create a three-item English-Spanish dictionary in a file called “dict.”

% cat >dict
red rojo
green verde
blue azul
<control-D>
%

<control-D> stands for “hold the control key down, then tap ‘d’”. The symbol > tells the computer that what is typed is to be put into the file dict. To view a file we use cat in a different way:

% cat dict
red rojo
green verde
blue azul
%

If we wish to add text to an existing file we do this:
% cat >>dict
white blanco
black negro
<control-D>
%

Now suppose that we have another file tmp that looks like this:

% cat tmp
cat gato
dog perro
%

Then we can join dict and tmp like this:
% cat dict tmp >dict2

We could check the number of lines in the new file like this:

% wc -l dict2
8

The command wc counts things — the number of characters, words, and line in a file.
chmod

This command is used to change the permissions of a file or directory. For example to make a file essay.001 readable by everyone, we do this:

% chmod a+r essay.001

To make a file, e.g., a shell script mycommand executable, we do this

% chmod +x mycommand

Now we can run mycommand as a command.
To check the permissions of a file, use ls -l . For more information on chmod, use man chmod.
cd
Use cd to change directory. Use pwd to see what directory you are in.

% cd english
% pwd
% /u/ma/jeremy/english
% ls
novel poems
% cd novel
% pwd
% /u/ma/jeremy/english/novel
% ls
ch1 ch2 ch3 journal scrapbook
% cd ..
% pwd
% /u/ma/jeremy/english
% cd poems
% cd
% /u/ma/jeremy

Jeremy began in his home directory, then went to his english subdirectory. He listed this directory using ls , found that it contained two entries, both of which happen to be diretories. He cd’d to the diretory novel, and found that he had gotten only as far as chapter 3 in his writing. Then he used cd .. to jump back one level. If had wanted to jump back one level, then go to poems he could have said cd ../poems. Finally he used cd with no argument to jump back to his home directory.
cp
Use cp to copy files or directories.
% cp foo foo.2

This makes a copy of the file foo.
% cp ~/poems/jabber .

This copies the file jabber in the directory poems to the current directory. The symbol “.” stands for the current directory. The symbol “~” stands for the home directory.
date
Use this command to check the date and time.
% date
Fri Jan 6 08:52:42 MST 1995
echo
The echo command echoes its arguments. Here are some examples:

% echo this
this
% echo $EDITOR
/usr/local/bin/emacs
% echo $PRINTER
b129lab1

Things like PRINTER are so-called environment variables. This one stores the name of the default printer — the one that print jobs will go to unless you take some action to change things. The dollar sign before an environment variable is needed to get the value in the variable. Try the following to verify this:

% echo PRINTER
PRINTER

ftp
Use ftp to connect to a remote machine, then upload or download files. See also: ncftp

Example 1: We’ll connect to the machine fubar.net, then change director to mystuff, then download the file homework11:

% ftp solitude
Connected to fubar.net.
220 fubar.net FTP server (Version wu-2.4(11) Mon Apr 18 17:26:33 MDT 1994) ready.
Name (solitude:carlson): jeremy
331 Password required for jeremy.
Password:
230 User jeremy logged in.
ftp> cd mystuff
250 CWD command successful.
ftp> get homework11
ftp> quit

grep
Use this command to search for information in a file or files. For example, suppose that we have a file dict whose contents are

red rojo
green verde
blue azul
white blanco
black negro

Then we can look up items in our file like this;
% grep red dict
red rojo
% grep blanco dict
white blanco
% grep brown dict
%

Notice that no output was returned by grep brown. This is because “brown” is not in our dictionary file.

Grep can also be combined with other commands. For example, if one had a file of phone numbers named “ph”, one entry per line, then the following command would give an alphabetical list of all persons whose name contains the string “Fred”.

% grep Fred ph | sort
Alpha, Fred: 333-6565
Beta, Freddie: 656-0099
Frederickson, Molly: 444-0981
Gamma, Fred-George: 111-7676
Zeta, Frederick: 431-0987

The symbol “|” is called “pipe.” It pipes the output of the grep command into the input of the sort command.
For more information on grep, consult

% man grep
——————————————————————————–

head
Use this command to look at the head of a file. For example,

% head essay.001

displays the first 10 lines of the file essay.001 To see a specific number of lines, do this:

% head -n 20 essay.001

This displays the first 20 lines of the file.
ls
Use ls to see what files you have. Your files are kept in something called a directory.

% ls
foo letter2
foobar letter3
letter1 maple-assignment1
%

Note that you have six files. There are some useful variants of the ls command:

% ls l*
letter1 letter2 letter3
%

Note what happened: all the files whose name begins with “l” are listed. The asterisk (*) is the ” wildcard” character. It matches any string.
——————————————————————————–

lpr
This is the standard Unix command for printing a file. It stands for the ancient “line printer.” See

% man lpr

for information on how it works. See print for information on our local intelligent print command.
mkdir
Use this command to create a directory.
% mkdir essays

To get “into” this directory, do
% cd essays

To see what files are in essays, do this:
% ls

There shouldn’t be any files there yet, since you just made it. To create files, see cat or emacs.
more
More is a command used to read text files. For example, we could do this:

% more poems

The effect of this to let you read the file “poems “. It probably will not fit in one screen, so you need to know how to “turn pages”. Here are the basic commands:

q — quit more
spacebar — read next page
return key — read next line
b — go back one page
For still more information, use the command man more.

mv
Use this command to change the name of file and directories.

% mv foo foobar

The file that was named foo is now named foobar

ncftp
Use ncftp for anonymous ftp — that means you don’t have to have a password.

% ncftp ftp.fubar.net
Connected to ftp.fubar.net
> get jokes.txt

The file jokes.txt is downloaded from the machine ftp.fubar.net.

print
This is a moderately intelligent print command.
% print foo
% print notes.ps
% print manuscript.dvi

In each case print does the right thing, regardless of whether the file is a text file (like foo ), a postcript file (like notes.ps, or a dvi file (like manuscript.dvi. In these examples the file is printed on the default printer. To see what this is, do

% print

and read the message displayed. To print on a specific printer, do this:
% print foo jwb321
% print notes.ps jwb321
% print manuscript.dvi jwb321

To change the default printer, do this:
% setenv PRINTER jwb321
pwd
Use this command to find out what directory you are working in.
% pwd
/u/ma/jeremy
% cd homework
% pwd
/u/ma/jeremy/homework
% ls
assign-1 assign-2 assign-3
% cd
% pwd
/u/ma/jeremy
%

Jeremy began by working in his “home” directory. Then he cd ‘d into his homework subdirectory. Cd means ” change directory”. He used pwd to check to make sure he was in the right place, then used ls to see if all his homework files were there. (They were). Then he cd’d back to his home directory.
rm
Use rm to remove files from your directory.
% rm foo
remove foo? y
% rm letter*
remove letter1? y
remove letter2? y
remove letter3? n
%

The first command removed a single file. The second command was intended to remove all files beginning with the string “letter.” However, our user (Jeremy?) decided not to remove letter3.
rmdir
Use this command to remove a directory. For example, to remove a directory called “essays”, do this:

% rmdir essays

A directory must be empty before it can be removed. To empty a directory, use rm.
rsh
Use this command if you want to work on a computer different from the one you are currently working on. One reason to do this is that the remote machine might be faster. For example, the command

% rsh solitude

connects you to the machine solitude. This is one of our public workstations and is fairly fast.

setenv
% echo $PRINTER
labprinter
% setenv PRINTER myprinter
% echo $PRINTER
myprinter
sort
Use this commmand to sort a file. For example, suppose we have a file dict with contents
red rojo
green verde
blue azul
white blanco
black negro

Then we can do this:
% sort dict
black negro
blue azul
green verde
red rojo
white blanco

Here the output of sort went to the screen. To store the output in file we do this:
% sort dict >dict.sorted

You can check the contents of the file dict.sorted using cat , more , or emacs .
tail
Use this command to look at the tail of a file. For example,

% tail essay.001

displays the last 10 lines of the file essay.001 To see a specific number of lines, do this:

% tail -n 20 essay.001

This displays the last 20 lines of the file.
tar
Use create compressed archives of directories and files, and also to extract directories and files from an archive. Example:

% tar -tvzf foo.tar.gz

displays the file names in the compressed archive foo.tar.gz while

% tar -xvzf foo.tar.gz

extracts the files.
telnet
Use this command to log in to another machine from the machine you are currently working on. For example, to log in to the machine “solitude”, do this:

% telnet solitude

See also: rsh.
wc
Use this command to count the number of characters, words, and lines in a file. Suppose, for example, that we have a file dict with contents

red rojo
green verde
blue azul
white blanco
black negro

Then we can do this
% wc dict
5 10 56 tmp

This shows that dict has 5 lines, 10 words, and 56 characters.

The word count command has several options, as illustrated below:

% wc -l dict
5 tmp
% wc -w dict
10 tmp
% wc -c dict
56 tmp
dummy
Under construction

This guide shows how to create a stylish rollover effect for a menu which might normally be done with Javascript or DHTML. CSS is used to do the menu highlighting.

http://www.ssi-developer.net/css/menu-rollover-effect.shtml