TPC photo
The TPC defines transaction processing and database benchmarks and delivers trusted results to the industry
 Home
 Results
 Benchmarks
 Technical Articles
 Related Links
 What's New
 About the TPC
 Who We Are
 Contact Us
 Join the TPC
 Useful Links
 TPCTC
 Document Search


 Member Login

History and Overview of the TPC
by Kim Shanley, Chief Operating Officer, Transaction Processing Performance Council

February, 1998
Preface
In my view, the TPC's history can be best understood by focusing on two of its major organizational activities: 1) creating good benchmarks; 2) creating a good process for reviewing and monitoring those benchmarks. Good benchmarks are like good laws. They lay the foundation for civilized (fair) competition. But if we have good benchmarks, why do we need all the overhead process for reviewing and monitoring the benchmark results? Similarly, you might ask if we have good laws, why do we need police, lawyers and judges? The answer to both questions is the same. Laws and benchmarks are not, in of themselves, enough. And by this I don't mean to imply that it's simply human nature to break or bend the rules. The TPC has found that no matter how clear-cut the rules appear to be when the benchmark specifications are written, there are always gray areas, and yes, loopholes left in the benchmark law. There must be a way of addressing and resolving these gray areas and loopholes in a fair manner. And yes, even "good laws," said Aristotle, "if they are not obeyed, do not constitute good government." Therefore, there must be a means for stopping those who would break or bend the rules. While this book is primarily a technical overview of the industry's benchmarks, the TPC's history is about both benchmark law and benchmark order.

The State Of Nature

In writing this early history of the TPC, I've drawn heavily upon the account by Omri Serlin published in the second edition of this handbook. It was through Omri's initiative and leadership that the TPC was founded.

In the early 1980's, the industry began a race that has accelerated over time: automation of daily end-user business transactions. The first application that received wide-spread focus was automated teller transactions (ATM), but we've seen this automation trend ripple through almost every area of business, from grocery stores to gas stations. As opposed to the batch-computing model that dominated the industry in the 1960's and 1970's, this new online model of computing had relatively unsophisticated clerks and consumers directly conducting simple update transactions against an on-line database system. Thus, the on-line transaction processing industry was born, an industry that now represents billions of dollars in annual sales.

Given the stakes--even at this point in the race--over who could claim the best OLTP system, the competition among computer vendors was intense. But, how to prove who was the best? The answer, of course, was a test--or a benchmark. Beginning in the mid-1980's, computer system and database vendors began to make performance claims based upon the TP1 benchmark, a benchmark originally developed within IBM that then found its way into the public domain. This benchmark purported to measure the performance of a system handling ATM transactions in a batch mode without the network or user interaction (think-time) components of the system workload (similar in design to what later turned out to be TPC-B). The TP1 benchmark had two major flaws. First, by ignoring the network and user interaction components of an OLTP workload, the system under test (SUT) could generate inflated performance numbers. Secondly, the benchmark was poorly defined and there was no supervision or control of the benchmark process. As a result, the TP1 marketing claims, not surprisingly, had little credibility with the press, market researchers (among them Omri Serlin), or users. The situation also deeply frustrated vendors who felt was their competitors' marketing claims, based upon flawed benchmark implementations, were ruining every vendor's credibility.

Early Attempts at Civilized Competition

In the April 1, 1985 issue of Datamation, Jim Gray in collaboration with 24 others from academy and industry, published (anonymously) an article titled, "A Measure of Transaction Processing Power." This article outlined a test for on-line transaction processing which was given the title of "DebitCredit." Unlike the TP1 benchmark, Gray's DebitCredit benchmark specified a true system-level benchmark where the network and user interaction components of the workload were included. In addition, it outlined several other key features of the benchmarking process that were later incorporated into the TPC process:

  • Total system cost published with the performance rating. Total system cost included all hardware and software used to successfully run the benchmark, including 5 years maintenance costs. Until this concept became law in the TPC process, vendors often quote only part of the overall system cost that generated a given performance rating.
  • Test specified in terms of high-level functional requirements rather than specifying any given hardware or software platform or code-level requirements. This allowed any company to run this benchmark if they could meet the functional requirements of the benchmark.
  • The benchmark workload scaleup rules -- the number of users and size of the database tables -- increased proportionally with the increasing power of the system to produce higher transaction rates. The scaling prevented the workload from being overwhelmed by the rapidly increasing power of OLTP systems.
  • The overall transaction rate would be constrained by a response time requirement. In DebitCredit, 95 percent of all transactions had to be completed in less than 1 second.
The TPC Lays Down the Law

While Gray's DebitCredit ideas were widely praised by industry opinion makers, the DebitCredit benchmark had the same success in curbing bad benchmarking as the prohibition did in stopping excessive drinking. In fact, according to industry analysts like Omri Serlin, the situation only got worse. Without a standards body to supervise the testing and publishing, vendors began to publish extraordinary marketing claims on both TP1 and DebitCredit. They often deleted key requirements in DebitCredit to improve their performance results.

From 1985 through 1988, vendors used TP1 and DebitCredit--or their own interpretation of these benchmarks--to muddy the already murky performance waters. Omri Serlin had had enough. He spearheaded a campaign to see if this mess could be straightened out. On August 10, 1988, Serlin had successfully convinced eight companies to form the Transaction Processing Performance Council (TPC).

to top of page

TPC-A
Using the model and the consensus that had already developed around the DebitCredit benchmark, the TPC published its first benchmark, TPC Benchmark A (TPC-A) within one year (November 1989). TPC-A differed from DebitCredit in the following respects:

  • The requirement that 95 percent of all transactions must complete in less than 1 second was altered to 90 percent of transactions must complete in less than 2 seconds.
  • The number of emulated terminals interacting with the SUT was reduced to a requirement of 10 terminals per tps and the cost of the terminals was included in the system price.
  • TPC-A could be run in a local or wide-area network configuration (DebitCredit has specified only WANs).
  • The production-oriented requirements of the benchmark were strengthened to prevent the reporting of peak, unsustainable performance ratings. Specifically, the ACID requirements (atomicity, consistency, isolation, and durability) were bolstered and specific tests added to ensure ACID viability.
Finally, TPC-A specified that all benchmark testing data should be publicly disclosed in a Full Disclosure Report.

The first TPC-A results were announced in July 1990. Four years later, at the peak of its popularity, 33 companies were publishing on TPC benchmarks and 115 different systems had published TPC-A results. In total, about 300 TPC-A benchmark results were published.

The first TPC-A result was 33 tpsA at a cost of $25,500 per transaction or tpsA. The highest TPC-A result ever recorded was 3,692 tpsA with a cost of $4,873 per tpsA. In summary, the highest tpsA rating had increased by a whopping factor of 111 times and the price/performance had improved by a factor of five. Does this increase in the top tpsA ratings correspond to an identical increase in the real world performance of OLTP systems during this period? Even in keeping in mind that this is a comparison of peak, not average benchmark ratings, the answer is no. The increase in tpsA ratings is just too great. The increase can be attributed to four major reasons: 1) the first benchmark test is usually run for bragging rights and is grossly unoptimized compared to later results; 2) real performance increases of hardware and software products; 3) vendors improving their products to eliminate performance bugs exposed by the benchmark, 4) vendors playing the benchmarking game effectively--learning from each other on how best to run the benchmark. So yes, there is a gamesmanship aspect to the TPC benchmark competition, but it should not obscure the fact that TPC benchmarks have provided an objective measure of a truly vast increase in computing power of hardware and software during this period. Indeed, the benchmarks have accelerated some of these software improvements. And all the marketing gamesmanship should also not invalidate the legacy achievement of TPC-A that for the first time, it provided the industry with an objective and standard means of comparing the performance of a vast number of systems.

TPC-B
While TPC-A leveraged the industry consensus built up over DebitCredit, the TPC's was much more ambiguous about publishing a benchmark around the TP1 model. The ambiguity about what was to turn into the TPC's second benchmark, the TPC-B benchmark, lasted throughout the life of the TPC-B. Everyone within the TPC organization believed that one of TPC-A's principal strengths was that it was an end-to-end system benchmark that exercised all aspects of an OLTP system. Furthermore, this OLTP system representing users working on terminals, conducting simple transactions over a LAN connected to a database server, was a model of computing that everyone could intuitively understand.

As described earlier, TP1 (and later TPC-B) was the batch version of DebitCredit, without the network and user interaction (terminals) figured into the workload. A strong block of companies within the TPC, including hardware companies who sold "servers" (as opposed to end-to-end system solutions) and database software companies, felt that the TPC-B model was more representative of the customer environments they sold into. The anti-TPC-B crowd, on the other hand, argued that the partial-system model that TPC-B represented would reduced the stress on key system resources and would therefore produce artificially high transaction rates. In addition, while the tps rates would be artificially high, the total system cost since the network and terminal pricing would be eliminated would be artificially low, thereby artificially boosting TPC-B's price/performance ratings. Finally, the anti-TPC-B spokespeople argued, since TPC-A and TPC-B were so much alike, using identical a "tps" throughput rating system, the users and press would be confused. Whether they won the argument or not can be debated, but what cannot is that TPC-B proponents eventually won the day and in August 1990, TPC-B was published as the second TPC benchmark standard. TPC-B used the same TPC-A transaction type (banking transaction) but it cut out the network and user interaction component of the TPC-A workload. What was left was a batch transaction processing benchmark.

The first TPC-B results were published in mid-1991 and by June 1994, at the peak of its popularity, TPC-B results had been published on 73 systems. In total, about 130 TPC-B tests were published. In all, there were about 2.5 times more TPC-A results published. The first TPC-B result was 102.94 tpsB with a cost of $4,167 per tpsB. The highest TPC-B rating was a 2,025 tpsB result, and the best price/performance number was $254 per tpsB. In summary, the top TPC-B ratings increased by a factor of 19 and the price performance rating improved by a factor of 16.

It's more difficult to give the legacy of TPC-B the unconditional stamp of success. It never received the user or market analyst acceptance that TPC-A did, but many within the Council and the industry perceived a real value in this "database server" benchmark model. This belief fueled a later failed TPC efforts around TPC-S (more on this later), and continues to influence the development of TPC benchmarks today. In January 1998, the TPC announced the formation of a Web Commerce benchmark (TPC-W) which will measure OLTP and browsing performance only of the web server, excluding the network and human interaction components of the overall system. So, TPC-B proponents may not have won the public debate on the merits of TPC-A versus TPC-B, but they may take some measure of satisfaction that the back-end model of benchmarking lives on.

Political Reform Begins Immediately

The TPC was a major improvement over the state of nature that existed previously. However, as the Council was to learn, most of the work of building a successful benchmarking organization and process was ahead of them. In this sense, the early TPC was not unlike the early American colonies who idealistically believed that they could eliminate the endless political and legal conflict of the Old Word by passing laws abolishing lawyers. While such a law still appeals to a significant minority, in our more judicious moments, most would agree that it's not possible.

As soon as vendors began to publish TPC results, complaints from rival vendors began to surface. Every TPC result had to be accompanied by a Full Disclosure Report (FDR). But, what happened when people reviewed the FDR and didn't like what they read? How could protest be registered and how would it be adjudicated? Even if a member of the public or a vendor representative were, so to speak, make a citizen's arrest of a benchmark violator, there was no police or court system to turn the perpetrator over to for further investigation or if need be, prosecution. It became apparent to the Council that without an active process for reviewing and challenging benchmark compliance, there was no way that the TPC could guarantee the level playing field that the TPC had promised the industry.

Throughout 1990 and 1991, the TPC embarked on a political journey to fix this hole in its process. The Technical Advisory Board (TAB), which was originally constituted as just an advisory board, became the arm of the TPC where the public or companies could challenge published TPC benchmarks. The TAB process, which remains in place today, established a fair, deliberative mechanism for reviewing benchmark compliance challenges. Once the TAB has thoroughly researched and reviewed a challenge, the TAB makes a recommendation to the full Council. The full Council then hears the TAB's report, discusses and debates the challenge, and then votes on the challenge. If the Council finds the result non-compliant in a significant or major way, the result is immediately removed as an official TPC result.

Benchmarking Versus Benchmarketing

By the spring of 1991, the TPC was clearly a success. Dozens of companies were running multiple TPC-A and TPC-B results. Not surprisingly, these companies wanted to capitalize on the TPC's cachet and leverage the investment they had made in TPC benchmarking. Several companies launched aggressive advertising and public relations campaigns based around their TPC results. In many ways, this was exactly why the TPC was created: to provide objective measures of performance. What was wrong, therefore, with companies wanting to brag about their good results? What was wrong is that there was often a large gap between the objective benchmark results and their benchmark marketing claims--this gap, over the years, has been dubbed "benchmarketing."

So the TPC was faced with an ironic situation. It had poured an enormous amount of time and energy into creating good benchmark and even a good benchmark review process. However, the TPC had no means to control how those results were used once they were approved. The resulting problems generated intense debates within the TPC.

Out of these Council debates emerged the TPC's Fair Use policies adopted in June, 1991.

  • When TPC results are used in publicity, the use is expected to adhere to basic standards of fidelity, candor, and due diligence, the qualities that together add up to, and define, Fair Use of TPC Results.
  • Fidelity: Adherence to facts; accuracy
  • Candor: Above-boardness; needful completeness
  • Due Diligence: Care for integrity of TPC results
Have the TPC's Fair Use policies worked? By and large, they have been effective in stopping blatantly misuse or misappropriation of the TPC's trademark and good name. In other words, very few companies claim TPC results when, in fact, they don't have them. In general, TPC member companies have done a fair job in policing themselves to stop or correct fair use violations that have occurred. At times, the TPC has acted strongly, issuing cease and retraction orders, or levying fines for major violations.

It must be said, however, that there remains today among the press, market researchers, and users, a sense that the TPC hasn't gone far enough in stamping out benchmarketing. This issue has two sides. On the one hand, companies spend hundreds of thousands of dollars, even millions of dollars, running TPC benchmarks to demonstrate objective performance results. It's quite legitimate, therefore, for these companies to market the results of these tests and compare them with the results of their competitors. On the other hand, no company has a right to misrepresent or mislead the public, regardless of how legitimate the benchmark tests may be. So where does the "war" on benchmarketing stand today? Much like the war on crime, the war on benchmarketing persists, and the TPC continues to wage an active campaign to eliminate it.

Codifying the Spirit of the Law

With the creation of a good review and fair use process, and with dozens of companies publishing regularly on the TPC-A and TPC-B benchmarks, the TPC may be forgiven for lapsing into a self-satisfied belief that the road ahead was smooth. That sense of well-being was torpedoed in April, 1993 when the Standish Group, a Massachusetts-based consulting firm, charged that Oracle had added a special option (discrete transactions) to its database software, with the sole purpose of inflating Oracle's TPC-A results. The Standish Group claimed that Oracle had "violated the spirit of the TPC" because the discrete transaction option was something a typical customer wouldn't use and was, therefore, a benchmark special. Oracle vehemently rejected the accusation, stating, with some justification, that they had followed the letter of the law in the benchmark specifications. Oracle argued that since benchmark specials, much less the spirit of the TPC, were not addressed in the TPC benchmark specifications, it was unfair to accuse them of violating anything.

The benchmarking process, which sprang from the discredited TP1 and DebitCredit days, has always been treated with a fair degree of skepticism by the press. So the Standish Group's charges against Oracle and the TPC attracted broad press coverage. Headlines like the May 17, 1993 issue of Network World were not uncommon, "Report Finds Oracle TPC results to be misleading; says option discredits TPC-A as benchmark."

Whether Oracle's discrete transition option was truly a benchmark special was never formally discussed or decided by the TPC. The historical relevance of this incident was that it spurred the TPC into instituting several major changes to its benchmark review process.

to top of page

New Anti-Benchmark Special Prohibition
TPC benchmark rules had always required companies to run the benchmark tests commercially available software. However, after the Standish Group charges, the Council realized that it had no real protection from companies that purposely designed a benchmark special component into their commercially available software. In other words, this special component could be buried in some obscure corner of overall product code and only be used when the vendor wanted to run a TPC test. If the TPC was formed to create fair, relevant measures of performance, then yes, the benchmark special was a violation of the TPC's spirit and thus had to be prohibited. In September, 1993, the Council passed Clause 0.2, which contains the sweeping prohibition against benchmark specials that become a bedrock of the TPC process to ensure, relevant benchmarks.

The Council drew a line in the sand with the passing of the Clause 0.2 anti-benchmark prohibition that has become part of the bedrock of the TPC process to ensure fair, relevant benchmarks:

  • Specifically prohibited are benchmark systems, products, technologies or pricing...whose primary purpose is performance optimization of TPC benchmark results without any corresponding applicability to real-world applications and environments. In other words, all "benchmark special" implementations that improve benchmark results but not real-world performance or pricing, are prohibited.
Clause 0.2 in TPC-A and TPC-B went into effect in June, 1994. Oracle decided not to test its discrete transaction option against the new anti-benchmark special rules in the specifications and withdrew all of its results by October, 1994. Let it also be noted that Oracle remains a TPC member and strong supporter of the organization.

New TPC Auditing Process
As a result of the 1993 controversies, the TPC realized that the millions of dollars being invested in the running of TPC benchmarks would be completely wasted if the credibility of the results were challenged. The TPC's process of FDR review was fine, but it only was invoked after a result was published and publicized. Yes, the TPC could yank a result from the official results list after it was found to be non-compliant, and even fine a company for violating the specifications, but the damage to the company's competitors and TPC's credibility would already have been done. In summary, it wasn't enough to catch the bad horse after it had left the barn. The goal was to stop the bad horse from ever getting out of the barn.

The result of these discussions, passed in September and December, 1993, was the creation of a group of TPC certified auditors who would review and approve every TPC benchmark test and result before it was even submitted to the TPC as an official benchmark or publicized. While TPC benchmarks are still reviewed and challenged on a regular basis, the TPC auditing system has been very effective in preventing most of the bad horses from ever leaving the barn.

New and Better Benchmarks
From the outset, I have said that the TPC's history is both about benchmark law and benchmark order. From the last sections of this chapter, the reader might have received the false impression that the TPC is exclusively a political organization endlessly embroiled in public controversies and institutional reform. The "benchmark order" activity of the TPC is certainly important, but the TPC's day-to-day focus is to build better benchmarks.

TPC-A was a major accomplishment in bringing order out of chaos, but TPC-A was primarily a codification of the simplistic TP1 and DebitCredit workloads created in the mid-1980's. However, what was very clear even as the TPC members approved TPC-A in late 1989 was that better, more robust and realistic workloads would be required for the 1990's.

Two benchmark activities were launched in 1990, the development of TPC-C, the next generation OLTP benchmark, and TPC-D, a decision support benchmark.

Both TPC-C, which was approved as a new benchmark in July, 1992 and TPC-D, which was approved in April, 1994, are covered in other chapters of this book and therefore I'll only add a few comments on these benchmarks.

The first TPC-C result published in September, 1992 was 54 tpmC result with a cost per tpmC of $188,562. As of this date (January 1998) more than six years later, the top result is a 52,871 tpmC with a cost per tpmC of $135. We have witnessed the same tremendous improvement in the top TPC-C numbers as we did for TPC-A and for the same reasons: 1) real world performance and cost improvements and 2) an increased knowledge about how to run the benchmark. (Again, keep in mind that just by looking at peak numbers, and not averages, we're seeing an exaggerated inflationary effect). Currently, there 143 official TPC-C results, a higher total than TPC-A at the peak of its popularity.

The first TPC-D result was a 100 GB result in December, 1995 with a throughput performance rating of 84 QthD and a price/performance rating of $52,170 QphD. Today, the top 100 GB throughput result that has been produced is 1205 QthD and $1877 QphD. Currently, there are 28 official TPC-D results. Why so few? TPC-D is only 2.5 years old, compared to TPC-C's venerable 6 years, and TPC-D is more expensive and complex to run.

Both TPC-C and TPC-D have gained widespread acceptance as the industry's premier benchmarks in their respective fields (OLTP and Decision Support). But the increase in the power of computing systems is relentless, and benchmark workload must continually be enhanced to keep them relevant to real world performance. Currently, a new major revision of TPC-C is being planned for release in early 1999. A new major revision of TPC-D is being planned for mid-1998 and another one in 1999.

to top of page

Aborted Benchmark Efforts
TPC Server Benchmark (TPC-S)

The goal of the Server Benchmark effort, initiated in early 1995, was to create a server version of TPC-C (as opposed to TPC's end-to-end system model) that would be both easier and less expensive to run. The strategy was to remove TPC-C's front-end and remote terminal emulation (RTE) requirements. In the end, TPC-S did not garner enough Council support. The reasons for this go back to issues raised in the TP1 versus DebitCredit and TPC-A versus TPC-B debates. Many felt that both TP1 and TPC-B were less credible than the end-to-end DebitCredit and TPC-A benchmarks. They argued removing the human interaction and network elements from the workload would lead to artificially high performance ratings. Those opposed to TPC-S cited the historical record that TPC-B results, on average, were almost twice as high as TPC-A results. Furthermore, some companies within the TPC felt two OLTP benchmarks both TPC-C and TPC-S would only cause confusion while forcing vendors to run another expensive benchmark without any compensating technical value. Again, those opposed to the TPC-S effort cited the TPC-B experience, where TPC-B results had been confused with TPC-A results. The TPC-S effort failed to secure enough Council votes to be sent out for mail ballot approval at the December, 1995 TPC General Council meeting, and the TPC-S effort ceased. As was noted earlier, the concept of a back-end or database server benchmark continues to find life in the new TPC-W benchmark being proposed.

TPC-E, the "Enterprise" Benchmark
The TPC-E or Enterprise Benchmark effort was initiated shortly after TPC-C was approved in July, 1992. Proponents of TPC-E argued that while TPC-C was significantly more complex and robust that the earlier TPC-A benchmark, but it still wasn't complex enough to stress very large, enterprise-class systems. TPC-E would demand that systems provide faster response times, higher I/O bandwidth, and concurrent batch processing, as well as fast recovery times which are not required of smaller systems. The TPC spent three years formalizing the TPC-E benchmark, and in the spring of 1996, submitted it for full Council approval. While a majority of TPC members supported the adoption of TPC-E (on two separate votes), TPC-E failed to secure the two-thirds approval required to approve any TPC benchmark as an industry standard. TPC-E failed to garner enough support for three reasons: 1) the TPC already had one OLTP benchmark (TPC-C), and two benchmarks in this space would only generate industry/user confusion; 2) another benchmark would force vendors to expend precious resources to run another benchmark even more expensive than TPC-C; 3) as an enterprise benchmark, TPC-E was only relevant to a relatively small number of companies competing in that space.

As is the case with the other "failed" TPC benchmark efforts, TPC-E continues to have an influence in the TPC benchmark development process. Many TPC-E concepts, such as lowering the transaction response time requirements, adding new and more complex transactions, as well as defining additional metrics beyond the primary throughput and price/performance metrics, are being incorporated into the revisions of TPC-C and TPC-D.

TPC Client/Server
Beginning in late 1994 and extending until mid-1996, the TPC worked on a new client/server benchmark (TPC-C/S) based around the core TPC-C workload. TPC-C/S was designed to model the more complex client/server environment with a wider range of applications and data. Three new transactions were to be added to TPC-C, including an order analysis, market analysis and an image transaction. The Council never voted on a TPC-C/S benchmark as it became apparent by mid-1996 that the World Wide Web was changing the computing paradigm. A TPC Web benchmark group formed out of the core of the TPC-C/S group. This eventually led to the TPC-W effort, described next. In addition, the legacy of the TPC-C/S effort can be seen in some of the new transactions and complexity being added to the Revision 4.0 work of the TPC-C Subcommittee.

The TPC's Newest Benchmark: TPC-W

The TPC broke new ground in 1998 with the announcement of a new Web Commerce Benchmark, called TPC-W. TPC-W, due to be released in late 1998, is designed to represent a business (retail store, software distribution, airline reservation, etc.) that markets and sells a product or service over the Internet. The benchmark will measure the performance of systems supporting users browsing and processing orders on a business web site. Since the network connection and user think time components of an overall Web OLTP environment are so radically variable, TPC-W will not include these components in its workload. TPC-W, then, could be viewed as a server benchmark (although not a batch benchmark).

to top of page

Summary
Early Legacy: Order Out of Chaos

Over its ten-year history, the TPC has weathered its share of criticism some fair, some not of its benchmarks and its benchmarking process. Nevertheless, the major achievement that TPC represents is often overlooked. Simply put, the TPC brought order out of chaos and thereby dramatically improved the business life of both users and vendors. Publishing the TPC benchmarks permanently raised the bar for what the industry expects in terms of benchmarks. The industry now expects benchmarks to be relevant to the real world and expects them to be backed by a stringent and independent review process. After the TPC, anything less just won't stand.

The TPC's Continuing Mission

The TPC continues to provide users and vendors with a number of benefits:

An objective means of comparing performance of different systems. Users and vendors alike need a common yardstick of computer system performance. Users need a standard yardstick of performance to make informed buying decisions; vendors need it to market effectively to an increasingly sophisticated base of users. TPC benchmarks provide a way for both groups to quantify transaction processing and decision support performance.

An objective means of comparing price performance of different systems. The TPC has been the most successful benchmarking group in developing a standard means of comparing the price/performance of different systems (most benchmarking groups just give up on this issue). The TPC's requirements for test sponsors to detail all hardware and software components and their associated costs, along with five years of maintenance costs, provides the industry's best price/performance metric.

A standard yet customizable benchmark for corporate and governmental acquisitions. Not all TPC benchmark results are published via the TPC and enter the public domain. Over the years, hundreds of corporate and governmental (both domestic and international) "users" those who purchase computer systems have employed TPC benchmarks as the foundation for their computer systems acquisition process. Worldwide, these sales total in the billions of dollars. Typically, as corporations and governments develop their request for proposals (RFPs) or request for quotes (RFQs), they scramble about to find an objective means of evaluating the performance of a bewildering array of different vendor architectures, technologies, and products. It's truly a daunting task. Prior to the TPC, these users would spend enormous time and resources trying to define a custom benchmark and trying to convince the vendor community that their custom benchmark was technically sound and fair to all parties. Using a TPC benchmark already accepted by the user and vendor communities eliminates much of this wasted time and resources. Typically, these users will take a standard TPC benchmark and customize it to match their unique processing requirements. For example, rather than running the standard 17 TPC-D queries, a user might request that vendors run the 17 TPC-D queries plus 5 queries that more closely represent their decision support environment. These users often settled on a particular hardware or software platform prior to the acquisition process. They specify their chosen hardware or software platform and then request that vendor products achieve a certain transaction level or query processing performance level. In almost all cases, the RFPs and RFQs calling for TPC benchmark results are publicized in some manner, but the TPC benchmark results are kept confidential. Therefore, in addition to the hundreds of published TPC benchmark results, there are hundreds of private (or quasi-private) TPC user benchmarks.

Complete system evaluation versus subsystem or processor evaluation. The TPC benchmarking model has been the most successful in modeling and benchmarking a complete end-to-end business computing environment, and this has helped the TPC benchmarks gain the recognition as credible, realistic workloads. Most past and many current benchmarks only measure the hardware performance (processor and memory subsystem). Increasingly, the complexity of business systems in how the software performs, and the TPC benchmarks have led the way in developing a benchmark model that most fully incorporates robust software testing.

Objective engineering tools which spurs real hardware and software improvements. TPC benchmarks are well-understood, stable workloads that engineers use on a continuous basis to eliminate hardware and software bottlenecks that lead to real world performance improvements for users. Granted, there is no linear relation between the increase in TPC performance numbers and the increase in real world performance. However, the absence of a linear relationship should not be used to dismiss the fact that significant real world performance gains are being generated via the TPC competitive process.


New, more sophisticated workloads. In the final analysis, benchmarks because they must be run in days and not months can only represent, not duplicate, a more complex reality. The real question to be asked of any benchmark is how well it represents or models that complex reality. As technological change continues to shift the ground beneath the static model, benchmarks must adapt and TPC's benchmarks are no exception.

The TPC is in the process of a major revision of its two mainstream benchmarks, TPC-C and TPC-D, to keep pace with changes in the OLTP and decision support arenas. It's also breaking new ground with its new Web Commerce Benchmark (TPC-W) effort. The mission, in short, continues.
 


to top of page
Home   Results   Benchmarks   Technical Articles   Related Links   What's New   About the TPC   Who We Are   Privacy Policy   About Pricing   Contact Us  
 

Valid XHTML 1.0 Transitional Valid CSS!