TPC photo
The TPC defines transaction processing and database benchmarks and delivers trusted results to the industry
    Document Search         Member Login    
     Home
About the TPC
Benchmarks
      Newsletter
      Join the TPC
      Downloads
      Technical Articles
      TPCTC
      Performance-Pulse

TPC Benchmark Status
February 2004

Overview

The TPC held a General Council meeting on December 11 in Dana Point, California. The main focus of the work was on refining existing benchmarks and laying the foundation for new benchmarks. On the OLTP front, work continues on the new TPC-E specification with a focus in two general areas: TPC-provided source code and new/modified features of the benchmark. The TPC-C benchmark has been updated to Version 5.2. With respect to decision support, progress continues on the new DS benchmark on dbgen output modifications, benchmark metrics, data maintenance, and pricing. For the TPC-H and TPC-R benchmarks, solutions are being explored to address dbgen issues with 64 bit architectures. As for the transactional web e-Commerce benchmark, a draft specification of the next version of TPC-W is now available for public review.

 

Current Benchmarks

TPC-C

The TPC-C Maintenance Subcommittee completed work on Version 5.2 of the TPC-C Specification. This version will become effective on February 9, 2004. The changes in Version 5.2 include:

  • Wording to clarify certain clauses in the specification
    • Definition of a third-party
    • Revising component availability dates
    • Line item pricing requirements
  • Report performance (tpmC) as a whole number (previously shown with 2 digits after the decimal point)

Results obtained with TPC-C Version 5.2 are comparable with those obtained with TPC-C Version 5.1

 

TPC-H and R

The TPC-H/R subcommittees identified a problem in dbgen when compiled for 64 bit architectures: Due to a data type overflow in dbgen, customer and supplier address fields are generated differently on 32 and 64 bit architectures. The code causing this problem is in rnd.c. A simple workaround allows users to generate the correct address fields. The subcommittee has proposed a code change which is currently being implemented and scheduled for vote during the February meeting. It is believed that there is no performance impact due to this behavior. A detailed investigation is under way.

 

New Benchmarks

TPC-E (OLTP)

The TPC-E Development Subcommittee continued development of a new OLTP workload. The workload is designed to challenge computing environments of today and the future. The subcommittee reviewed several presentations from member companies regarding the current design of the benchmark environment and ideas to expand functionality. Early prototype data were presented and work items have been identified to address concerns raised by the data. An updated version of the source code was delivered and work items for the next version have been identified. Also, an updated version of the draft specification was approved along with ongoing changes and development of the specification.

 

TPC-DS

The subcommittee continued its work on developing TPC-DS in the areas of dbgen output modifications, metric, data maintenance and pricing.

Dbgen output can be modified. The resources (time, systems) to generate and modify dbgen output are not included in any metric. The subcommittee agreed on a basic throughput based metric. It further developed a catalog of criteria as a framework to finalize the metric. A group of members have been tasked to finalize the metric and present a proposal at the next meeting. The data maintenance portion of the specification has been overhauled according to motions made at the last meeting. Pricing in TPC-DS will follow closely the general pricing document that the TPC is currently developing with minimal TPC-DS specific clauses. In this process the subcommittee also identified some minor issues with the pricing document, which will be forwarded to the pricing subcommittee.

 

TPC-W

The TPC-W subcommittee continued its development of the next version of TPC-W. Version 2 is a transactional web service benchmark consisting of database interactions displaying ACID properties running against a commercial application server. The primary metrics are SIPS (Service Interactions Per Second), Price Performance which is $/SIPS, and the system availability date.

Components that must be commercially available have been enumerated and defined. The subcommittee has also adopted additional requirements such as adhering to the WS-I basic profile (BP1.0) for web service interoperability. Strict restrictions have been placed on user written/modified code regarding operations resulting in caching of data. All caching of durable data must maintain full atomicity, consistency, and isolation (ACI).

All benchmark clauses have been defined and accepted. This includes processing requirements, database schema and population, test run requirements, and steady state requirements for the test run. It also includes ACID tests which demonstrate compliance of the database system as well as that of distributed transactions and message queues. Reporting requirements, and audit requirements for the web service interactions have also been defined and accepted.

Two prototype efforts have thus far yielded three initial sets of prototype data (2 .NET, 1 Java). Additional prototype data is expected as time proceeds. Prototype performance is trending upward as the implementations evolve. The prototypes demonstrate the specified workload can be implemented using both Java and .NET.

The subcommittee presented the draft specification to the TPC General Council and received unanimous approval to proceed to Company and Public review with comments accepted until February 20, 2004.

 

Other TPC Activities

Public Relations Committee

The PR Committee continues its work on the campaign to promote the TPC and the TPC benchmarks to the industry. In response to the increasing publication of TPC results in currencies other than US dollars, the TPC web site has been enhanced to display results grouped by currency.

 

Pricing

The TPC formed a pricing workgroup on October 9, 2002 to recommend revisions to the existing pricing methodology so that prices used in published TPC results are clear, consistent and verifiable for all the TPC benchmarks. The revisions do not change what is priced. Its adoption is expected to produce results comparable with current results. The requirements for reporting and ordering are enhanced and strengthened, allowing improved verifiability of pricing. The pricing specification would be referenced by all existing and future TPC specifications for a consistent methodology.

The draft TPC pricing specification is now available for both public and member company review, with comments accepted until February 29, 2004.

 

All Benchmark Status Reports
 

Valid XHTML 1.0 Transitional Valid CSS!