E-Commerce Bedrock

By Amjad Umar

Electronic commerce (EC)--that is, buying and selling over a network--has been with us for almost two decades in the form of Electronic Data Interchange (EDI), emails, file transfers, fax communications, and bar-code systems. While traditional EC, conducted over privately owned and managed value-added networks (VANs), has been growing at a respectable 5 to 10 percent a year, this expansion is minuscule compared to the explosive growth of Internet-based EC. VAN- and Internet-based EC, however, each have their pros and cons.

At a basic level, EC providers and users need highly secure, highly reliable, and well-managed networks with extensive facilities to support core EC activities, such as advertising, browsing/selection, purchasing, billing/invoicing, and payments. In the past, VAN providers such as GEIS, AT&T, and Harbinger have supported some of these activities through private data networks and proprietary technologies. These networks support purchasing and billing quite well (purchasing by sending purchase orders and billing by supporting invoices); the existence of the well-known X850 (for purchase orders) and X810 (for invoices) EDI standards are evidence of it. VANs, however, are not designed for advertising and browsing. In addition, they're too expensive and require too many proprietary interfaces and software modules to provide the services I've described.

On the other side of the coin, the public Internet is ideal for advertising and browsing, but it's too unreliable and unsecure to satisfy the security, reliability, and manageability required for purchasing, billing and cash transfers. Consequently, EC suppliers are turning to extranets (enterprise networks that use Internet technology but restrict access to recognized partners). But to be successful, these extranets must support high-quality EC services that meet intense security and management requirements.

The Role of Transaction Databases

As EC evolves, it will expand beyond EDI and increasingly rely on higher level services such as transactions and workflow among organizations. Transaction databases, or TDBs (also known as transaction-oriented databases and transaction-based databases), will play an increasingly important role in this area.

Views differ as to the definition of a TDB. One view, held in financial-industry segments such as banking and insurance, is that a TDB is little more than a master log of transactions. For example, in the service industries, TDBs contain requests submitted by customers; the analysis and processing of these requests are considered mission-critical activities. A broader view--the one adopted in this article--is that a TDB is simply a database that supports transactions; it allows you to store, analyze, and manipulate information needed in transaction processing. To perform these tasks, TDBs must provide high availability, integrity, and performance.

TDBs support organizational activities; they store the information required to conduct basic but essential business operations. TDBs also support the analysis of an organization's activities for improving and/or streamlining performance.

The main properties of TDBs are:

• Data is in a state of change.
• Changes occur as transactions.
• Data can be found quickly and is not duplicated.
• Interrelationships among data items are stored in the database.
• Properties and methods of data storage are internal to the database.

Historically, TDBs have been constructed using flat files, hierarchical databases, network databases, and relational databases, but object-oriented and active databases are the technologies of the future for TDB deployment. Not surprisingly, most TDBs today are based on relational DBMSs from major vendors such as Oracle, Sybase, and Informix. In an April 1996 survey described by SunWorld Online (www.tritec.de/sunworldonline), 35 percent of respondents identified Oracle as their primary engine for transaction databases, with Sybase and Informix following at 21 percent and 19 percent respectively. The same survey indicated that the largest transaction databases developed or managed by most respondents (40 percent) are in the range of one to five gigabytes; a smaller number (20 percent) manage larger TDBs in the range of 20 to 100 gigabytes.

In EC, TDBs are used for storing shopping carts, purchase orders, payments, income receipts, and invoices. In addition, stored customer and inventory information is used to support order-processing activities. These databases, as we will see, are quickly becoming the foundation for transaction processing and workflow in EC.

Before proceeding, let's briefly revisit the classical definition of a transaction and see how it maps to EC. Simply stated, a transaction is a group of statements that must be executed as a unit and that satisfies the following four properties, known collectively as ACID:

• Atomicity, also known as the "all-or-nothing property." A transaction is treated as a single unit of operation; either all the transaction's actions are completed, or none of them are. If a transaction completes all its actions successfully, it has "committed."
• Consistency. A transaction maps one consistent (correct) state of the database to another. Informally, consistency involves the correct reflection of reality in the database. For example, if an EC merchant has 50 televisions in stock, a database consistent with this reality should also show 50 televisions. The notion of a consistent state is highly dependent on the semantics of the database. A set of constraints, called semantic integrity constraints, is used to verify database consistency.
• Isolation. A transaction cannot reveal its results to other concurrent transactions before commitment. Isolation assures that transactions do not access data being updated (temporarily inconsistent and incomplete during the execution of a transaction).
• Durability. When committed (completed successfully), the results of a transaction are permanent and cannot be erased from the database. The DBMS ensures that the results of a transaction are not altered due to system failures (transactions endure failures).

Should EC conform to all the ACID properties? One view is that EC is on a dangerous path because it ignores traditional transaction processing concepts and technologies. For example, at least one analysis of digital cash products has shown that many of them do not adhere to atomicity.¹ Another view is that EC activities should be as lightweight as possible and not be saddled with ACID overhead. Let's work through some details.

EC Transactions in TDBs
EC is a mixture of decision-support and transaction processing activities; some core EC activities involve transaction processing and some do not. The ACID requirements for these activities can be easily analyzed (see Table 1). Furthermore, EC transactions typically fall into two participant categories:

• EC transactions among trusted business partners (suppliers and corporations that enter business agreements and contracts to buy and sell products, for example). These transactions typically involve large amounts, introduce medium traffic, and require rigorous security. Such transactions are perfect candidates for extranets.
• EC transactions between suppliers and the general public (such as in Internet shopping malls). These transactions typically involve small amounts, introduce heavy traffic, and require medium security.

Given that different transaction semantics must be supported for each of these categories, how should you process transactions so that the needed ACID properties are satisfied? For centralized mainframe systems, transaction processing for ACID transactions--in the form of online transaction processing (OLTP)--has been the backbone of commercial data processing since the early 1970s. Mainframe-based transaction-processing monitors such as CICS and IMS-DC/IMS-TM have matured over the years to provide high performance and reliable services, but the situation is dramatically different in the distributed environments characterizing EC. In these environments, the possible approaches fall into three categories: TP-less (do not use any transaction management facilities), TP-lite (use database procedures to handle updates), and TP-heavy (use a distributed transaction manager to perform updates).²

Which approach works best depends on the type of EC activities in question. Each has benefits and drawbacks, so the following questions should be asked before making a decision:

• In what format is the data stored (databases or flat files)? If the data is stored in multiple databases and flat files, TP-lite is not an option because database procedures are supported only in RDBMS environments.
• On how many servers does the data reside? If the application needs to update and commit data stored on multiple servers, then the TP-heavy approach should be used because database procedures cannot participate with other database procedures in a distributed transaction.
• What is the requirement for data synchronization? If the data synchronization interval is periodic, then a TP-lite solution combined with a data replication server may be useful for managing updates against replicated data.
• What are the requirements for performance and load balancing? TP-less works well when you do not need any transaction processing capabilities, and TP-lite solutions with database procedures are much faster than the TP-heavy solutions that require synchronization among sites. But TP-heavy solutions provide many sophisticated procedures for dynamic load balancing, priority scheduling, process restarts, and prestarted servers that are especially useful for large-scale production environments. These features are the main strength of TP-heavy products because many of these products have been used over the years to handle thousands of transactions in production OLTP environments.

While the debate between the proponents of TP-lite and TP-heavy continues, many EC projects are ignoring the whole issue by focusing primarily on EDI, and thus, in some sense, reinventing the wheel. EDI, at its best, is a TP-less approach. Some EC applications are being deployed as TP-lite, while large mission-critical EC applications use TP-heavy only at back-end mainframe systems. Meanwhile, many small EC applications appear to work quite well as TP-less.

At this point, the TDBs supporting EC transactions can be conventional RDBMs because EC does not put many TP demands on the database. But this picture, as we will see later, will change.

Object-Oriented Transactions
Object-oriented transaction management has been an area of considerable academic and industrial investigation. There are several interesting applications of object technology in EC; for example, customers, merchants, and commodities can be viewed as objects with interesting class hierarchies, methods, and messages. Broadly speaking, object-oriented transaction management takes three forms: object-oriented transaction programming, transaction processing with distributed objects, and transaction processing with object-oriented databases.

Object-oriented transaction programming involves the use of programs written in an object-oriented programming language to perform transaction operations (for example, commit or rollback). This approach is the same in principle as that of all other kinds of transaction programming and is currently the state of the practice; after all, C++ programs can be used in Encina and Tuxedo environments. From a transaction-processing point of view, no complications are introduced. Object-oriented transaction programming is naturally the most common object-oriented approach being used in EC.

Transaction processing with distributed objects involves performing ACID operations among objects across machines. It introduces many complex issues. In particular, messages exchanged among objects can carry transaction semantics. For example, if an object sends a message to a remote object, then operations such as concurrency control and recovery may be needed among the distributed objects. Traditional object models do not take this type of processing into account; specialized middleware is required.

The Object Management Group (OMG) is specifying a transaction processing distributed-objects service for its Common Object Broker Architecture (CORBA). CORBA 2.0 (see sidebar, "CORBA Services for EC"), the current operational version at press time, already specifies an Object Transaction Service (OTS) and Object Concurrency Control Service (OCCS). These two services are expected to boost industrial products to support transaction management with distributed objects. CORBA, however, is not the only technology for transaction processing with distributed objects. Microsoft Transaction Server (MTS) also supports this paradigm but uses the Microsoft Distributed Component Object Model (DCOM) instead of CORBA (more on MTS later).

Transaction processing with object-oriented databases involves the introduction of TP-heavy-type operations on top of OODBMSs. Most of these operations are also currently available on relational and hierarchical databases. The fundamental difference is that unlike older DBMSs, which provide read and write operations, OODBMSs support methods that are rich in semantics. Most of the work in OODBMS has focused on query and modeling aspects. Transaction management in object-oriented multidatabase systems in particular is receiving serious attention. We'll have to wait and see how this technology finds its way in EC.

Within the EC context, objects may represent business entities such as customers, items, and orders; and the transactions can involve purchasing and payment activities. Let's examine how the two main approaches, CORBA and DCOM, support this paradigm.

CORBA-based OTSs. CORBA 2.0 OTSs are based on the X/Open Distributed Transaction Processing Reference Model. This model, however, has been extended to manage distributed objects through two improvements: The XA interface (specifies the APIs among the transaction managers and the database mangers) and the TX interface (specifies the APIs among the transaction managers and applications) have been specified using a set of CORBA interfaces defined in IDL. All intercomponent communication is mandated to comprise CORBA method calls on instances of these interfaces.

In addition, OTSs are fully compatible with X/Open compliant software. OMG enforces this compliance by requiring an OTS to import and export transactions to and from XA compliant resource managers and TX compliant transaction managers, respectively. To manage concurrency control, OMG has introduced the OCCS specification to provide locking support for resource sharing among concurrently executing objects. In addition, OMG has specified optional nested transactions to insulate a global transaction from partial failure of some of its constituent operations.

The OTS mediates among the CORBA applications to ensure that the updates to databases are performed atomically within the scope of a transaction. For example, if applications A and B were part of a transaction that needed to update a database, the OTS is called by the participating applications to initiate a transaction, register the databases and other resources that participate in the transaction, coordinate the updates, and roll-out any changes made in case of a failure.

OTS products are beginning to appear. One example is OrbixWeb 2.0 from Iona, which combines OTS with Java and Internet-Inter-ORB Protocol (IIOP) support for the Internet and intranets. Because IIOP is currently accepted as a de facto standard for interactions among objects over the Internet, OrbixWeb is a candidate for EC transactions. The OTS technology in OrbixWeb is provided by Transarc (the IBM company that built the Encina transaction processor) and Groupe Bull. The OTS operates on top of Iona's ORB. TPBroker from Visigenic is another example of an OTS.

DCOM-based Object Transaction Services (MTS). Microsoft has introduced MTS (see Figure 1) as a component-based programming environment for developing, deploying, and managing Internet/intranet transactionýbased applications. MTS 1.0, released in December 1996, runs on Windows NT 4.0 and accommodates Windows NTý and Windows 95ýbased clients (with DCOM support) for transaction processing.

Because of its component-based application development model, MTS allows you to breakdown application logic into ActiveX components that can be built using standard development tools such as Visual Basic, Visual C++, Visual J++, Cobol, and PowerBuilder. When installed in the MTS environment, these components can be invoked from HTML browser clients and Win32 clients. A browser client invokes the MTS ActiveX components using any Internet server (including the Microsoft Internet Information Server), and a Win32 client invokes MTS ActiveX components through DCOM. MTS itself provides the infrastructure for transaction coordination across systems and system resources (such as processes and database connections) initiates and controls transactions, enforces security, and provides administrative tools. In addition, MTS provides a set of dispatchers that interact with IBM mainframe CICS and IMS transactions, OLE transactions, and X/Open XA (which could be used as an interface with the CORBA-based OTS).

Putting the Pieces Together
EC platforms combine TDBs and transaction processing services with other services (such as EDI, email, security, and administration) to support interactions among business-to-business and business-to-customer entities. The platform software (sometimes called EC middleware) typically resides above the network services to support the core activities of advertising, browsing, purchasing, billing, and payments. The middleware needed for the contemporary EC applications comprises EDI, faxes, file transfers, and email. New middleware services, however, are needed to support transaction and workflow semantics because many EC activities (including billing and payment) require such support. In addition, the mediators that link the Web world with the legacy world are important in EC because many billing, invoicing, and order-processing systems are traditional mainframe-based legacies. Naturally, the use of distributed object middleware (CORBA, ActiveX/DCOM) is an interesting area for EC.

At this point, the major vendors (IBM, Microsoft, Oracle, NCR, Netscape) are scrambling to introduce their own EC platforms. Most of them consist of various servers--Web servers, commerce servers, transaction servers, database servers, and so on.

IBM Net.Commerce. IBM Net.Commerce (see Figure 2) and its companion products provide the tools to develop, host, and operate informational and transactional Web sites on the Internet and intranets. Net.Commerce supports a range of APIs for linking electronic commerce activities with other applications such as order processing, inventory, pricing, shipping, and tax calculations. At the core of this architecture is the Net.Commerce server, which provides APIs, command support, macros, daemons for dynamic Web page generation, and multihosting for accommodating multiple virtual stores on the same machine. The Net.Commerce database houses the transactional (order information, shipping information) as well as nontransactional (tax calculation tables) catalogs. These catalogs include native support for DB2 and can be optionally stored in any other ODBC compliant databases. Internet Connection Secure Server establishes secure sessions across the Internet between the user's browser and Net.Commerce applications, Net.Commerce administrator provides a suite of tools used to create and administer Net.Commerce sites and virtual storefronts, and Net.Commerce utilities support import of data into catalogs from external sources such as supplier stores and legacy data interfaces for access to business applications ("Merchant legacy systems") such as inventory and billing. Finally, Net.Commerce companion products include support for Secure Electronic Transaction (SET) payments, a catalog workbench for creating and maintaining catalogs, data mining tools, and Lotus Notes Domino for document flow. Transactional services, eventually including the OTS, are expected to be integrated in Net.Commerce.

Microsoft Internet Commerce Strategy. Microsoft has entered the EC market with its Site Server Enterprise Edition (EE) family of products. The Site Server EE includes Microsoft Commerce Server (previously known as Microsoft Merchant Server), which supports, through wizards and other GUI utilities, dynamic catalog management, online order processing, creation of dynamic product and price promotion, and support for special functionality through "pipelines." For example, the order processing pipeline provides data structures for managing orders, payment, integration with existing systems, and third-party software packages. (This pipeline can be extended to support taxes, EDI, and payment processing.) Payments are supported through Microsoft Wallet, while an open API is offered for integrating independent pay technologies such as credit cards, debit cards, and electronic checks. The Site Server EE runs on the Windows NT Server and interworks with other Microsoft servers such as the Internet Information Server and Microsoft Transaction Server, as shown in Figure 3.

Oracle E-Commerce Platform. Oracle's Network Computing Architecture (NCA) is the foundation for the company's E-Commerce platform. NCA supports several servers (Oracle Web Server, Web Request Broker, and Oracle Universal Server), which communicate through HTTP and CORBA protocols. NCA cartridges plug into this architecture to provide specialized functionalities. Figure 4 shows how the Oracle E-Commerce platform employs application-specific cartridges to support EC.

The Oracle Web Application Server manages all communications among the different Oracle product components. The core of Oracle E-Commerce is the Oracle Universal Server, which includes a search engine, APIs for integrating with back-end (including legacy) systems, and facilities to generate on-the-fly content such as price and product descriptions. (Naturally, Oracle7 is used for transactional and nontransactional database services.) The Oracle Internet Commerce Server provides the core functionalities of setting up and configuring the stores and contents, shopping carts, interfaces with inventory control systems, and logging for analysis. The Oracle Payment Server supports a variety of third-party add-ons such as Verifone and CyberCash for payment and credit card handling, Taxware for tax calculations, and TanData for shipping and handling management.

Making Transactions Internet-Aware. As you'd expect, Internet transaction processing (ITP) is vital to EC, especially for customer-to-business commerce. For example, Syntellect provides an Internet-based movie subscription service that receives orders from customers and then processes these orders using Tuxedo transaction processing capabilities. Many of these ITPs are, and will continue to be, written using Java applets.

However, ITP must resolve the philosophical differences between the Internet and corporate transaction processing: While the Internet is free-wheeling and unsupervised, OLTPs are proprietary and corporate-controlled. To strike a balance between the two, corporations are using the firewall to partition ITP services. For example, in this approach, the clients use the public Internet outside the firewall to submit requests and receive answers, but the actual transaction processing is performed inside the firewall by the application services with the protection and integrity controls typically needed for transaction processing.

As an example, Tandem introduced its Internet Transaction Processing (iTP) initiative in October 1996 to extend its OLTP products into the EC arena. The iTP strategy involves six servers: Tandem iTP Commerce Server (enables business-to-commerce transactions on the Internet), Tandem iTP Media Server (allows distribution of videos, electronic catalogs, music, and large databases over the Internet), Tandem iTP Messaging Server (lets businesses communicate through X.400 and SMTP), Tandem iTP CTI Server (supports computer telephony integrated applications), Tandem iTP Intranet Server (facilitates collaboration by geographically distant team members), and the Tandem iTP Matrix Server (enables integration of legacy and other custom applications). Progress Software has also built an ITP architecture, and many other initiatives have been announced and are on the drawing board.

The Upshot
Internet EC is an infant technology and will evolve through various stages. The main growth will be in the middleware needed to support the increased interorganizational, transactional, and workflow activities.

TDBs will play a strong role in this stage as we move from EDI and email to complex interorganizational EC transactions. TDBs will increasingly be used for storing shopping carts, purchase orders, payments, income receipts, and invoices; in addition, customer and inventory information will be stored in databases to support order-processing activities.

In the near future, TDBs will support business transactions through a range of database technologies (RDBMS, universal databases, OODBMSs, and active databases) in combination with new techniques such as intelligent agents, distributed objects, legacy mediators, and workflows. In addition, as the current products for transaction processing with distributed objects mature, we'll see more EC platforms relying on this new technology.

References
1. Tygar, J. "Atomicity in Electronic Commerce." Proceedings of ACM Symposium on Principles of Distributed Computing, May 23-26, 1996. 2. Umar, A. Object-Oriented Client/Server Internet Environments. Prentice Hall, 1997.

Further Reading
Caruso, V. "The Evolution of Transaction Based Databases." www.ee.uts.edu.au/ise/homepages/hmpgs95a/vcaruso/dbhist.html. Kalakota, R., and Whinston, A. Frontiers of Electronic Commerce. Addison-Wesley, 1996.

Amjad Umar, Ph.D., is a principal engineer for electronic commerce and distributed systems at Bellcore and an adjunct professor at Rutgers University. He is the author of three Prentice Hall books on distributed systems. You can reach Amjad at aumar@notes.cc.bellcore.com.

 

Transaction Databases as Data Warehouses

Although TDBs are mainly intended to support transactional activities, these databases can also be useful for populating data warehouses. For example, in an insurance company, a transaction database storing all processed insurance claims can be a rich source for a claims data mart. This data mart could then be mined to uncover interrelationships among types of claims, user populations, and geographical areas. In the same vein, TDBs for EC can be used for data warehouses. For example, the purchase log that shows what items were purchased by whom is an excellent data source for decision support. A purchase-capturing TDB can be easily created by tracking when a customer clicks on the "purchase" icon after filling the shopping cart. This TDB, if loaded in an RDBMS, can be used for sophisticated analysis of user buying habits and for negotiating prices with merchants. The purchase log could also be mined to discover associations among customers, projects, items, and suppliers. The catalogs that maintain product information are, in effect, data warehouses that support purchasing decisions. These databases are not typically TDBs unless you store inventory information in the catalogs (inventory update can be treated as a transaction). A catalog can be viewed as a data warehouse especially if it consolidates catalogs from many suppliers to provide a consistent look and feel. As for choosing the best approach for supporting data warehouses in EC, as a starting point, an RDBMS should do the job. But because EC applications increasingly use graphics, and, in some cases, may need to store multimedia information, object and object/relational databases are good technology candidates for the future.