The data warehousing methodology we follow at Cambridge Technology Partners advocates beginning each project with a scoping phase. This scoping phase, or simply "scope," usually lasts from two to five weeks, depending on the complexity of the project (that is, how broad a set of functionality you need to support, how many data sources you'll have, how many organizations represent the data warehouse's user constituency, and similar factors). After the scope, a design phase occurs, which is followed by a development phase, rollout, and deployment.
It sounds pretty straightforward, right? Though some data warehousing analysts and consultants maintain that data warehouse projects cannot be based on well-defined user requirements because such requirements cannot--or should not--be fully determined before the warehouse is used operationally (a subject I'll address in a future column), let's assume for the purposes of this column that it is possible to define functional requirements from which a data warehouse should be built. What steps should you follow to do so?
In this column, I will present an overview of what you should accomplish during the scope, and in my next column, we'll discuss some of the techniques you should use.
The goals of the data warehouse scope are straightforward and include:
•To build and validate the business case for proceeding with the subsequent data warehouse project phases (design, development, and deployment)
•To explore, evaluate, and decide upon the boundaries of your data warehouse: what business functionality will be supported, what facts--groups of data--you need for the functionality, what data sources you will use to create the facts, who the users will be, what business units will be served, and so on
•To conduct preliminary explorations into the candidate technologies of the data warehouse, particularly front-end tools but also "data warehouse middleware" products (those that support the extraction, transformation, and movement of data) and the database management system that will host the data warehouse.
Simply stated, by the conclusion of the scoping phase of activity, there should be--make that must be--general consensus from all stakeholders as to the reasons for constructing the data warehouse, what it will contain, and who will use it. Furthermore, there must be widespread agreement that:
•You can successfully build the data warehouse in the allotted time and within the allotted budget.
•The design and development team assigned has the right base of skills to succeed in its assignment.
•Adequate support exists at the executive levels of the company, both within the user community and within the IT areas, to see the project through.
•The business and technology risks discovered during the scope are manageable; the answers might not be immediately known, but there is confidence that solutions will be found during subsequent phases of the data warehouse project.
Though the expected accomplishments listed above seem like a lot to accomplish during the short period of time allocated for the scope, there is a method to this madness. A team that not only successfully determines the properties that define the scope of the data warehouse but also works through issues of organizational politics, identify, and categorize risk, and quickly evaluates technology and products for suitability has a significant likelihood of success as the data warehouse project moves into subsequent phases. Conversely, a team that fails to accomplish the above will likely not succeed in building a data warehouse. (Read: "Stop right here and regroup; do not move forward, because you'll probably do little else than waste time and money.")
The team must achieve significant results by the end of the first week of the scope, whether the entire scoping phase will last two weeks or five weeks. This progress includes two items often overlooked on data warehousing projects: agreement on the mission statement and agreement on the business case.
The mission statement. It's often tempting to declare the mission of a data warehouse project to be exactly that: "to build a customer analysis (or credit risk, or some other subject) data warehouse." A data warehouse project that proceeds under such an ill-defined charter has a significant chance of failure due to the lack of a clear-cut business mission.
The first order of business during the scoping phase is to reach agreement on a mission statement--a paragraph or two that:
•Is business-focused (that is, that refers to functionality or processes that are accomplished by the organization's users)
•Starts to establish boundaries of the data warehouse (such as which organization will be supported by the warehouse).
At the same time, the data warehouse's mission statement should not:
•Contain references to specific data sources
•Mention specific technologies or products.
Consider the following data warehouse mission statement:
Mission: to develop a data warehouse that will support the following bankwide customer management and support functions: customer acquisition, customer support, customer retention, cross-selling, and customer reacquisition.
Once agreed upon, the mission statement presented above will guide all subsequent discussion about the detailed functionality: the names and roles of those who will use the data warehouse, the specific data elements that will be acquired from all the data sources, and so on.
The mission statement should be:
•Prepared in advance by the team leader, using the data warehouse's project charter, background research, and other appropriate material as input
•Presented as part of the kick-off meeting (more on this later)
•Thoroughly discussed until consensus is reached among all participants.
A common mistake is to assume that a one- or two-paragraph mission statement, such as the previous example, can be presented, discussed, and agreed to in a matter of minutes. The thinking is often, "How could anyone possibly disagree with this? The mission statement is simply a formalized statement of the project charter, and by default, the project charter is acceptable because the project has been funded."
However, as most people who have participated in a mission statement discussion know, seemingly innocuous wording often raises passions among the diverse audience of data warehouse scope participants. Organizational politics, cultural issues, and even basic scoping and boundary issues (for example, should the customer support data warehouse really be bankwide or just for the retail side of the bank?) can lead to prolonged, animated discussions from the outset of the effort. Therefore, you should allot sufficient time in the schedule for the presentation, discussion, and adoption of the mission statement. One of the worst things that can happen is to adopt a mission statement without consensus simply because of time pressures; if that occurs, the entire data warehouse project is already severely at risk.
Expected business benefits and functionality. While creation of and consensus about the mission statement is critical, this is only the beginning of the work you must accomplish during the data warehouse scope. You need to generate significantly more detail than a paragraph or two's worth of words; you must develop an entire business case.
The business case is a consolidated picture of the functionality that the data warehouse will support. If you look at the mission statement as a high-level, business-oriented description of why time, money, and energy should be devoted toward building a data warehouse, the business case is a more detailed version of why the organization should proceed.
A common mistake with respect to data warehouses is to focus exclusively on data when creating the business case. A document that relies on such statements as "the customer support data warehouse must contain the following: the previous two years' sales by product, customer, and channel" as the reason to build a data warehouse is missing an important element: why the warehouse should contain that information and the business value that will be gained. It may turn out that the data warehouse should contain the information listed above, but the surest way to promote indecisiveness and even cause concern among the data warehouse project's sponsors is to focus exclusively on the data.
Where, then, should you begin when building the business case? The key is to focus on functionality--concise, action-oriented statements describing what must be done as part of the course of business activity. Examples of functionality statements include:
•Track cross-selling performance by channel (e-commerce; call center driven; direct marketing; and so on).
•Analyze customer churn and provide alerts when levels rise above acceptable thresholds.
•Provide targeted direct marketing support to meet the company's monthly customer reacquisition goals.
It is important to remember that the statements of proposed data warehouse functionality must be in concert with the mission statement that your group previously presented, discussed, and adopted. That is, every statement of functionality that is considered a candidate for data warehouse support must:
•Apply to the proposed user and organizational community as presented in the mission statement
•Be part of the overall sphere of functionality (in our example, customer management and support).
You should support the following types of functionality in your data warehouse:
•Activities that cannot be done at all today, but should be
•Activities that are accomplished today by manually synthesizing data from reports, file extracts, and other sources, which often requires a great deal of time and energy.
It is the responsibility of each business community member of the data warehouse to present accurately and thoroughly the functionality of his or her job that fits within the sphere of influence of this initiative. At the same time, it is the facilitator's job to keep the discussion on track and not let proposed functionality stray outside the bounds of the mission statement and the team's charter.
The "now or future" balancing act. Inevitably, someone offering a point of functionality as part of the business case for the data warehouse raises this sort of question:
"I do (some function) this way now, but if we're going to be implementing this new system, it would be better done like this: ...."
There needs to be an understanding from the outset of the data warehouse project as to what the philosophy of, and objectives for, process change and improvement are in the context of the data warehouse itself. On one side of the spectrum, the charter may be that business functionality, and the processes that make up that functionality, will not change as a result of the implementation of the data warehouse.
On the other side of the spectrum, there may be situations where the business processes are so "broken"--inefficient, ineffective, disjointed, and so forth--that implementing a data warehouse in support of those processes would be a serious mistake. Rather, an effort to explore process improvement is not only desirable but mandatory. Ideally, this need for process improvement has already been identified; the process improvement teams have already met, done whatever needed to be done, and presented their results; and the mandate of "this is the way we will do things from now on" has already been issued.
Sometimes, however, it is during the data warehouse scope--indeed, during the first days of the scope--that the first inklings of serious business process problems are raised. You could argue that a healthy organizational culture would have fostered the raising of these issues long before the data warehouse scope has begun, but that's really not the issue.
In other words, do not waste time and money proceeding; you may succeed in developing and deploying some type data warehouse in an expedient manner, but the chances are high that little, if any, enhanced business value will come out of this effort.
(Portions of this column appear in 90 Days to the Data Mart, Chapter WK1.)
Alan Simon is vice president of worldwide data warehousing solutions at
Cambridge Technology Partners. He is the author of 22 books, including Data Warehousing for
Dummies (IDG Books, 1997), 90 Days to the Data Mart (John E. Wiley & Sons, 1998), and a
new expanded edition of How to be a Successful Computer Consultant (McGraw-Hill, 1998).
You can reach him at asimon@ctp.com.