Percentile Responses – What Are They?

Transaction processing – for example, a simple update of a customer’s bank account with a deposit – is the lifeblood of many businesses. There is often a need to specify what response time is required, depending on the needs of the application. This is where much confusion can arise, with talk of averages, percentiles, means, and other terms.

It will be evident that response times for individual transactions, even of the same type, will vary. In attempting to compare the capacities of two or more transaction processing systems, we need some way of quoting these times meaningfully rather than a long list of each response time. One way is to quote the average and some number representing the variation of response times around this number.

We will examine this number, called a percentile, in more detail a little later.

TC Figure 1

Figure 1: Sample Response Time Curve

Since all transactions, even of the same type and execution parameters, do not have the same response time, a useful form of measure for transaction response times is a graph of the numbers of transactions and their associated response times, the response curve. A schematic response curve is shown in the figure above.

Although the graph shows a continuous curve, it really represents a smoothed out histogram for measurements on a finite number of transactions and their response times. In the figure shown:

Related:IT Availability: The Whole Truth

For measurements on larger and larger numbers of transactions, the histogram would approach a smooth curve as the intervals, (t₁-t₀), (t₂-t₁) and so on, approach zero.

The response curve we have just discussed was illustrative only and the types of curves arising in real situations are covered later. However, a practical aspect of response times and response curves should be noted here.

Response time variations

Experience has shown that users prefer a system which has an average response time (T_R), made up of a narrow range of individual response times to one with a lower average composed of widely varying response times. For example, an average response time of 2.5 seconds derived from response times between 1.7 and 3.1 seconds is preferable to an average of 2.2 seconds made up of responses between 0.1 and 12.1 seconds. Why? 2.2 is less than 2.5.

The latter often results in irritation, errors and loss of concentration by the user. This highlights the idea of a 'distribution' of response times instead of a single number for every transaction entered. The latter case has a good average response but a wide distribution of component responses whereas the former has a worse response time but a narrower spread.

Related:In-depth: The IT Performance Concept

Another trick of the trade is also relevant to this discussion. It is often the case that a new computer system has much spare capacity when initially installed but fills up over a period as the planned user base increases. In the early days, the users will probably see a very good response time which will deteriorate as more users come onto the system. Since their expectations will have been set by this early experience, they will become dissatisfied with the later level of service.

Response time percentiles: Detail

The X^th percentile of the value V of a variable means that X% of the relevant population is below the value V. The formula for percentiles in queuing theory usually relates to response time (latency) and depends on the arrival distribution of the queuing system. The formula for the X^th percentile of response time T_R for an exponential distribution is given by:

TC Formula 1_0

Equation 1: The Xth Percentile of an Exponential Distribution T_R

where 'ln' is the logarithm to base 'e'.

This gives, for example, the value of the 90th percentile of 2.3 T_R.

The table below shows some value characteristics for three distribution types.

TC Table 1

Table 1: Response Time 90th Percentile for Different Distributions

As a corollary (add-on fact), the formula for wait time, T_W, percentiles (exponential distribution) is:

TC Formula 2

Equation 2: The Xth Percentile of Wait Time

The percentile specification is on often used in Service Level Agreement (SLAs) when talking about transaction (in its broadest sense) response times. Thus, an SLA might specify a 90th %ile for payroll transactions of 1.9 seconds but for complex stock transaction, it may specify 3.5 seconds as the 90th %ile.

The following figure shows three percentiles on a typical response time curve for an exponential arrival distribution.

TC Figure 2_0

Figure 2: Examples of Response Time Percentiles

Other specifications might be means or averages of the times of responses but they don't give any information about the spread of responses, an important factor, which we discussed above under 'Response Time Variations'.