1. Who are the top report users?
2. Who are the top report creators?
3. Which users generate the most errors?
4. What are the most common errors?
5. What are the most popular reports?
6. Which reports take the longest to run?
7. Which database tables are used most frequently?
8. What is the row count of the most frequently used tables?
9. What table columns are used the most?

Let me first set a typical scenario that I have found during my consulting career. After several years of development, a large corporation has finally managed to implement a corporate-wide data warehouse (DW). A dedicated IT development group has built and deployed a reporting solution on top of the DW. Unfortunately, the users of the DW, typically on the business side of the corporation, are very unhappy with the reporting environment. They claim that performance is extremely slow. Reports take forever to run. Many reports return cryptic errors that only IT can decipher. The IT group is bombarded with support requests, and must allocate most of its time to answering support calls. Furthermore, since the corporation’s business is changing constantly, business users demand that the DW evolve to reflect how the business works. However, since IT spends most of its time fixing problems with the existing reporting environment, it cannot implement new features in the DW.

The scenario described above creates a dynamic whereby business users grow increasingly disillusioned with the DW and the IT group in charge of it, while the IT team grows frustrated with the business users and their insatiable demand for support and feature improvements. Both groups become frustrated with the reporting software, which is the most visible part of the DW.

When the situation becomes critical enough, the IT team decides to bring an external consultant to “fix the problem”.

In my next post I will list the high-level steps that a consultant must follow to solve the problem.

$ db2 “get snapshot for all on DBNAME” > allsnap.txt

What are aggregate tables?

Aggregate tables, also know as summary tables, are fact tables which contain data that has been summarized up to a different level of detail. For example, let’s say that your data warehouse contains a transaction table with the following characteristics (I’ll use a banking example):

Table dimensionality: account id, transaction type, day id, transaction amount
Average number of transactions per day: 30 million
Number of days stored in the transaction table: 30
Approximate number of rows: 900 million rows

Let’s pretend that half of the daily transactions are deposits, so there are approximately 450 million rows that represent deposit transactions. The other half are withdrawals.

Suppose a DW user wants to know how much money was deposited into the bank during the past month. The user, through the reporting software, will issue a query similar to:

select sum(transaction_amount)
from transaction_fact
where transaction_type=’deposit’

Pretend that your DW platform can scan 10 million rows per second; therefore, the approximate time to complete the query will be:

query time = number of rows / scan rate

which in our example translates into:

query time = 900 million rows / 10 million rows/second

query time = 90 seconds

Waiting 90 seconds for such a simple query is simply unacceptable, so here is where an aggregate table can help you. To answer our hypothetical question, we will build an aggregate table which summarizes the transaction table by transaction type. The aggregate may be defined as follows:

create table fact_transaction_aggregate as
select day_id, transaction_type, sum(transaction_amount) as transaction_amount
from transaction_fact
group by day_id, transaction_type

We said before that there are only two transaction types and thirty days of data. Using the simplifying assumption that half of the daily transactions are deposits and half are withdrawals, the size of the new table will be only 60 rows! (table size = 30 days * 2 transaction types)

The SQL needed to get the answer is:

select sum(transaction_amount)
from transaction_aggregate
where transaction_type=’deposit’

And the answer will come back in 0.000006 seconds (60 rows / 10 million rows /second). The result: happy users!

It discusses:

• Guidelines on building the large database and determining the number of partitions
• Bulk load using the new multipartition load
• Performance enhancements using MultiDimensional Clustering and Materialized Query Tables.
• Availability through the new online utilities
• Self Managing And Resource Tuning features
• Migration scenarios