Guerrilla Manual Online

Hit-and-run mantras you can use on your boss or colleagues in team meetings

Updated on May 3, 2013

1. Updates to the print version of The Guerrilla Manual provided as a pull-out booklet in the original Guerrilla Capacity Planning book.
   
   
2. New aphorisms encapsulated as Guerrilla Mantras or GM that automatically appear on Twitter.
   
   

Contents

1  WEAPONS OF MASS INSTRUCTION

1.1  Why Go Guerrilla?

1.2  Best Practices

Copying someone else's apparent success is like cheating on a test. You may make the grade this time, but how far will the bluff take you into the future?

A recent NPR program discussed how this approach has backfired in the arena of medicine and healthcare. The analogy is appropriate because many aspects of performance analysis are not too different from the science of medical diagnosis.

• 11. Avoid Prod acceptance testing like the plague.
  
  
• 12. Develop on a platform totally unrelated to the Prod system.
  
  
• 13. Never compare load test requirements with actual remote user-based measurements from Keynote, Gomez, etc.
  
  

1.3  Virtualization

Translation: We performance weenies need more whistles and less bells. In other words, virtualization vendors need to make sure they provide us with backdoors and peepholes so we can measure how resources are actually being consumed in a virtualized environment.

1.4  An Ounce of Prevention

Then, explain the multi-billion dollar dietary-supplements industry?

1.5  Why is Performance Analysis Hard?

1.6  Brisk Risk Management

"I can understand people being worked up about safety and quality with the welds," said Steve Heminger, executive director ... "But we're concerned about being on schedule because we are racing against the next earthquake."

1.7  Failing On Time

Until you read and heed this statement, you will probably have a very frustrating time getting your perforance analysis conclusions across to managment.

1.8  Performance Homunculus

• Security management
  
  
• Software distribution
  
  
• Backup management
  
  
• Chargeback management
  
  
• Cap and Perf management
  
  

Cap and Perf management can rightly be regarded as just a subset of systems management, but the infrastructure requirements for successful capacity planning (both the tools and knowledgeable humans to use them) are necessarily out of proportion with the requirements for simpler systems management tasks like software distribution, security, backup, etc. It's self-defeating to try doing capacity planning on the cheap.

1.9  Self Tuning Applications

Think about it. Performance analysis is a lot like a medical examination, and medical Expert Systems were heavily touted in the mid 1980's. You don't hear about them anymore. And you know that if it worked, HMO's would be all over it. It's a laudable goal but if you lose your job, it won't be because of some expert performance robot.

1.10  Squeezing Capacity

Today, there is a serious need to squeeze more out of your current capital equipment.

1.11  When Wrong is Right

The planning part of capacity planning requires making predictions. Even a wrong prediction is useful because it can serve as a warning that either:

1. the understanding which underlies your prediction is wrong
   
   
2. the measurement process is broken and is producing wrong data
   
   

1.12  Throw More Hardware at It

Single-threadedness can wreck you

1.13  Network Performance

If your local network is out of bandwidth, has interminable latencies, or is otherwise glitching, don't bitch about the performance of your application. Get the LAN fixed, first. Then we'll talk about the performance of your application.

1.14  Not even wrong!

In case you're wondering, those are REAL data and yes, the axes are correctly labeled. I'll let you ponder what is wrong with these measurements. Here's a hint: They're so broken, they're not even wrong! Only if you don't understand basic queueing theory, would you accept measurements like these (which the original performance engineer did).

1.15  Checked your measurements lately?

Most people remain blissfully unaware of the fact that ALL measurements come with errors; both systematic and random. An important capacity planning task is to determine and track the magnitude of the errors in your performance data. Every datum should come with a `±' attached (which will then force you to put a number after it).

1.16  Data Are Not Divine

Data comes from the Devil, only models come from God.

Corollary: That means it's helpful to be able to talk to God. But God, she does babel, sometimes. :)

1.17  Busy work

Western culture too often glorifies hours clocked as productive work. If you don't take time off to come up for air and reflect on what you're doing, how are you going to know when you're wrong?

1.18  Little's Law

1. Eqn.(1) tells us the average number of customers or requests in residence.
   
   
2. Eqn.(2) tells us the average number of customers or requests in service.
   
   

I use it almost daily to cross-check that throughput and delay data are consistent, no matter whether those data come from measurements or models. More details about Little's law can be found in Analyzing Computer System Performance with Perl::PDQ. Another use of Little's law is calculating service times, which are notoriously difficult to measure directly. See also Mantras 2.10 and 2.12.

1.19  Bigger is Not Always Better

The bigger the symmetric multiprocessor (SMP) configuration you purchase, the busier you need to run it. But only to the point where the average run-queue begins to grow. Any busier and the user's response time will rapidly start to climb through the roof.

1.20  Bottlenecks

In any collection of processes (whether manufacturing or computing) the bottleneck is the process with the longest service demand. In any set of processes with inhomogeneous service time requirements, one of them will require the maximum service demand. That's the bottleneck. If you reduce that service demand (e.g., via performance tuning), then another process will have the new maximum service demand. These are sometimes referred to as the primary and the secondary bottlenecks. The performance tuning effort has simply re-ranked the bottleneck.

1.21  Benchmarks

The purpose of competitive benchmarking a computer system is to beat everyone else on performance, so you can say "mine is bigger than yours." It's the IT equivalent of war! Benchmark run-rules were made to be broken or at least bent; just don't get caught. This must be true because industrial benchmark organizations like SPEC.org and TPC.org have technical review committees that look for cheating in submitted results.

For capacity planning and system sizing, you need to be aware of this fact of life and look for the loopholes in published benchmark results.

Here, competitive refers to industrial benchmarks, as opposed to benchmarking that you might do for purely internal comparisons or diagnostic purposes.

1.22  Failure to Communicate

"What we've got here is failure to communicate." —Prison warden in the movie Cool Hand Luke.

Question:

How long should you spend creating your capacity planning presentation?

Rule of thumb:

You should spend as much time refining the presentation of your capacity planning results as you did reaching them.

"Je n'ai fait celle-ci plus longue que parce que je n'ai pas eu le loisir de la faire plus courte." —Blaise Pascal (Dec 4, 1656).

Translation: I have made it [this letter] longer because I have not had time to make it shorter.

1.23  Consolidation

When consolidating applications, it's not the migration that's the problem, it's the often mistaken motivation. The fallacious reasoning goes like this. Since there are a slew of servers that are only running at 10% busy, migrate those apps onto fewer (virtualized) servers. Naturally, the target server capacity will be better utilized but the application response time (SLA) has now gone out the window.

1.24  Control Freaks Unite!

Control over the performance of hardware resources e.g., CPUs and disks, is progressively being eroded as these things simply become commodity black boxes viz., multicore processors and disk arrays. This situation will only be exacerbated with the advent of Internet-based application services. Software developers will therefore have to understand more about the performance and capacity planning implications of their designs running on these black boxes. (See Sect. 3)

1.25  Productivity

Thanks to the Puritans, presumably, American corporate culture is obsessed with the false notion that being busy is being productive. Wrong! Europeans (especially the Mediterraneans) understand the power of the cat nap. After nearly 400 years, it's time for America to get over it.

"Creativity is the residue of time wasted." —A. Einstein

Who works the longest hours?

Data come from the OECD (Organisation for Economic Co-operation and Development). Developing countries often work longer hours, but working longer doesn't necessarily mean working better. Interestingly, the USA is in the middle along with Australia.

1.26  Art vs. Science

A number of recent books and presentations on performance analysis and capacity planning have appeared with "The Art of..." in the title. In itself, this is not new. The title of Raj Jain's excellent 1991 book is The Art of Computer Systems Performance Analysis. Nonetheless, they all resort to various scientific techniques to analyze performance data. The application of science inevitably involves some art.

1.27  ITIL for Guerrillas

The ITIL framework is all about defining IT processes to satisfy business needs, not their implementation. That's what makes GCaP training an excellent IT-business solution.

1.28  Vanishing Paradox

If you do performance management perfectly, you run the risk of becoming invisible and therefore expendable to your management.

In other words, having successfully supported performance management, a manager could eventually feel justified in asking: "Why is my budget paying for performance management when everything is performing perfectly?" AKA a career-limiter.

Compare this with software development, for example. If a developer does their job perfectly, they risk being overburdened with more work than you can handle (AKA job security). In contrast to performance management, a manager might be heard to say: "We must have this new functionality in the next release!"

1.29  Dumb Question

1.30  Quantum Leap

Quantum transitions in energy are only associated with the discrete spectrum of atomic or molecular bound states.

1.31  Don't Be a Sheeple

1.32  Good Communication

That last one was tweeted and ended up as entry #136 in David Pogue's Twitter Book:

1.33  Performance Gatekeepers

A common situation in big organizations is for various groups to be responsible for the performance evaluation of the software or hardware widgets they create. In principle, this is a good thing, but there is a downside. Over-zealous performance optimization of any subsystem can deoptimize the entire system. To avoid this side-effect, a separate central group should be responsible for oversight of total system performance. They should act as both reviewers and gatekeepers for the performance analyses produced by all the other groups in the organization.

1.34  Performance Analysis is a Money Sink

• Performance evaluation
  
  
• Performance by design
  
  
• Performance engineering
  
  
• Performance testing
  
  

1.35  Tyranny of the 9s

downt <- function(nines,tunit=c('s','m','h')) {
	ds <- 10^(-nines) * 365.25*24*60*60
	if(tunit == 's') { ts <- 1; tu <- "seconds" }
	if(tunit == 'm') { ts <- 60; tu <- "minutes" }
	if(tunit == 'h') { ts <- 3600; tu <- "hours" }
	return(sprintf("Downtime per year at %d nines: %g %s", nines, ds/ts,tu))
}

> downt(5,'m')
[1] "Downtime per year at 5 nines: 5.2596 minutes"
> downt(8,'s')
[1] "Downtime per year at 8 nines: 0.315576 seconds"

library(NORMT3)
sigp <- function(sigma) {
	sigma <- as.integer(sigma)
	apc <- erf(sigma/sqrt(2))
	return(sprintf("%d-sigma bell area: %10.8f%%; Prob(chance): %e", 
		sigma, apc*100, 1-apc))
}

> sigp(2)
[1] "2-sigma bell area: 95.44997361%; Prob(chance): 4.550026e-02"
> sigp(5)
[1] "5-sigma bell area: 99.99994267%; Prob(chance): 5.733031e-07"

• 3σ corresponds roughly to a probability of 1 in 1,000 that four 9s availability occurred by chance.
  
  
• 5σ is roughly a 1 in a million chance, which is the same as flipping a fair coin and getting 20 heads in a row.
  
  
• 6σ is roughly a 1 in a billion chance.
  
  

1.36  Time Scale Intuition

Because of the great diversity of time scales that exist in modern computer systems, it's a good idea to try and get a more intuitive feel for some of them. My first attempt at helping you to do that was in 1998, where I included the table shown at the left in The Practical Performance Analyst. I updated that as Table 3.1 in my Perl PDQ book. Here, I've rendered those quantities as a data frame in R: Subsystem nanoSeconds secondZ Rescaling SIunit 1 4 GHz CPU clock 2.50e-01 0.25 1.00 s 2 L1 cache access 5.00e-01 0.50 1.00 s 3 L2 cache access 1.25e+00 1.25 1.00 s 4 Memory bus cycle 2.00e+00 2.00 1.00 s 5 DRAM chip access 6.00e+01 60.00 1.00 min 6 Physical disk seek 3.50e+06 3500000.00 1.33 month 7 Network NFS read 3.20e+07 32000000.00 1.01 yr 8 Database SQL update 5.00e+08 500000000.00 15.84 yr 9 Magnetic tape read 5.00e+09 5000000000.00 1.58 century To get a better impression of the vast range of time scales, actual nanoseconds are rescaled to be units measured in seconds. In other words, a multiplier of a gigasecond has been applied to the actual time values to make them human scale. The key point is that since a CPU processor operates on a nanosecond time-scale in the real world or ``seconds'' on the rescaled human level, RAM accesses take on the order of minutes, disk accesses take on the order of a month, database accesses take years, and magetic-tape IOs take on the order of a century. Looked at from the standpoint of the CPU processor clock, this gives a somewhat different slant to the expression: Hurry up and wait!

1.37  Throughput vs. Latency

1. Bandwidth (or throughput) corresponds to the diameter of the pipe.
   
   
2. Latency corresponds to the length of the pipe (implicit time to flow through it at a constant rate).
   
   

On closer scrutiny, one finds many problems with the conventional wisdom:

1. Bandwidth and latency are ill-defined terms.
   
   
2. Latency refers to some kind of (heavily context-dependent) delay, e.g., disk latency means something completely different from packet latency.
   
   
3. Queueing theory is very specific about different latencies: waiting time, service time, response time, etc.
   
   
4. Bandwidth usually refers to a special case of throughput, i.e., maximum throughput.
   
   
5. Throughput and latency only appear to be unrelated in clocked systems; like flow through a pipe (see above).
   
   
6. If you apply the conventional wisdom to database performance, for example, you will be surprised to find that latency (e.g., response time) increases with throughput.
   
   

1. Bandwidth and Latency are Related Like Overclocked Chocolates
   
   
2. Bandwidth vs. Latency - The World is Curved
   
   
3. Little's law is a curved surface in 3-D
   
   

2  PERFORMANCE MODELING RULES OF THUMB

2.1  What is Performance Modeling?

"90% of coding is debugging. The other 10% is writing bugs."

2.2  Monitoring vs. Modeling

2.3  Keep It Simple

A performance model should be as simple as possible, but no simpler!

"A designer knows that he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away." —Antoine de St-Expurey

I now tell people in my Guerrilla classes, despite the fact that I repeat this rule of thumb several times, you will throw the kitchen sink into your performance models; at least, early on as you first learn how to create them. It's almost axiomatic: the more you know about the system architecture, the more detail you will try to throw into the model. The goal, in fact, is the opposite.

2.4  More Like The Map Than The Metro

The Tube map is pure abstraction that has very little to do with the physical railway system. It encodes only sufficient detail to enable transit on the underground from point A to point B. It does not include a lot of irrelevant details such as altitude of the stations, or even their actual geographical proximity. A performance model is a similar kind of abstraction.
Despite several attempts, the original Tube map has hardly been improved upon since its conception in 1933. Apparently, it already met the requirement of being as simple as possible, but no simpler. The fact that it was designed by an electrical draughtsman, probably helped.

2.5  The Big Picture

2.6  A Point of Principle

As an example, consider a classic multi-user time-share OS, e.g., Unix. Its PoP can be stated as follows:

Time-share scheduling gives every user the illusion that they are the only user on the system. [17 words]

All the thousands of lines of code in the OS that implement time-slicing, priority queues, and so forth, are there merely to support that PoP. The performance goal in that case is user response time.

Being able to see things in this unconventional way can really help in the deciding on the simplest kind of performance model.

1. Your laptop is also a time-share OS, e.g., Mac OS X, Ubuntu Linux, Windows. Is the PoP for that single-user OS the same or different than it is for a multi-user OS?
   
   
2. Virtualization environments, like VMware or Parallels hypervisors are all the rage, but these hypervisors are nothing more than a kind of global OS with each time-share OS treated as an embedded guest OS. Is the hypervisor PoP the same or different than it is for a time-share OS?
   
   

2.7  Guilt is Golden

You, as the performance analyst or planner, only have to shine the light in the right place and then stand back while others flock to fix it.

2.8  What is a Queue?

1. A queue is a line of customers waiting to be severed—as in " Off with their heads!" (*)
   
   
2. Hardware version: The buffer or queue is implemented as a register, e.g., a memory register.
   
   
3. Software version: The buffer or queue is implemented as a list. In some computer languages it is a separate data type, e.g., Lisp, Mathematica, Perl, etc.
   
   

2.9  Where to Start?

One place to start constructing a PDQ model is by drawing a functional block diagram. The objective is to identify where time is spent at each stage in processing the workload of interest. Ultimately, each functional block is converted to a queueing subsystem like those shown above. This includes the ability to distinguish sequential and parallel processing. Other diagrammatic techniques e.g., UML diagrams, may also be useful but I don't understand that stuff and never tried it. See Chap. 6 "Pretty Damn Quick(PDQ) - A Slow Introduction" of Analyzing Computer System Performance with Perl::PDQ.

2.10  Inputs and Outputs

Take Little's law Q = X R for example. It is a performance model; albeit a simple equation or operational law, but a model nonetheless. All the variables on the RIGHT side of the equation (X and R) are INPUTS, and the single variable on the LEFT is the OUTPUT. A more detailed discussion of this point is presented in Chap. 6 "Pretty Damn Quick(PDQ) - A Slow Introduction" of Analyzing Computer System Performance with Perl::PDQ.

2.11  No Service, No Queues

If the measurements of the real system do not include the service time for a queueing node that you think ought to be in your PDQ model, then that PDQ node cannot be defined.

2.12  Estimating Service Times

Suppose, for example, you had requests coming into an HTTP server and you could measure its CPU utilization with some UNIX tool like vmstat, and you would like to know the service time of the HTTP Gets. UNIX won't tell you, but you can use Little's law (U = X S) to figure it out. If you can measure the arrival rate of requests in Gets/sec (X) and the CPU %utilization (U), then the average service time (S) for a Get is easily calculated from the quotient U/X.

2.13  Change the Data

2.14  Closed or Open Queue?

2.15  Opening a Closed Queue

An open queueing model assumes an infinite population of requesters initiating requests at an arrival rate λ (lambda). In a closed model, λ (lambda) is approximated by the ratio N/Z. Treat the thinktime Z as a free parameter, and choose a value (by trial and error) that keeps N/Z constant as you make N larger in your PDQ model. Eventually, at some value of N, the OUTPUTS of both the closed and open models will agree to some reasonable approximation.

2.16  Steady-State Measurements

2.17  Transcribing Data

2.18  Workloads Come in Threes

Apart from making an unwieldy PDQ report to read, generally you are only interested in the interaction of 2 workloads (pairwise comparison). Everything else goes in the third (AKA "the background"). If you can't see how to do this, you're probably not ready to create the PDQ model.

2.19  Better Than a Crystal Ball

Predicting the future is not the same thing as "seeing" the future. A performance model is just a means for evaluating the data that are provided to it. The purpose of the model is to transform those data into information. The amount of information that can be extracted is intimately dependent on the values of those data. Change the input data and you change the output information. An oft-quoted example is: garbage in, garbage out.

That's an extreme example. More commonly, you may see unexpected modeling results. In that case, the new data do not meet the expectations set by the prior meaurements. But that doesn't necessarily imply the model is wrong. More likely something has changed in the meaurement system so that it has failed to remain consistent with the initial information contained in the previous measurements.

Having a performance model forces you to ask why that unexpected change has occurred and can anything be done to remove it. Without a performance model, you don't have any expectations or context for asking such questions.

2.20  Patterns and Anti-Patterns

Visual example:

Textual example:

Colorless green ideas sleep furiously. —N. Chomsky (1957)

New research indicates that if a person is not in control of the situation, they are more likely to see patterns where none exist, suffer illusions and believe in conspiracy theories.

2.21  Interpreting Data

The conventional view of performance models is that they are useful for:

1. Predicting the future performance of an extant system
   
   
2. Exploring what-if scenarios that may or may not be realistic
   
   

A performance model can also be used for interpretating performance measurements. Both the measurements and the model must be consistent or something is wrong and needs to be explained.

2.22  Intuition and Modeling

Ulysses is the the Greek hero in Homer's Odyssey. On his way back from the Trojan wars, Ulysses orders his men to tie him to the mast of his ship and to plug their own ears so that they will not succumb to the beautiful song of the sirens and be diverted to their deaths. Ulysses, being a typical manager, chooses to be bound and to keep his ears unplugged because he cannot bear the idea of not hearing the sirens' music.

2.23  Load Average

For a more complete discussion, see:

1. See Chapter 4 in 1st edn. of my Perl::PDQ book
   
   
2. See Chapter 6 in 2nd edn. of my Perl::PDQ book
   
   
3. Read the original online articles (as PDFs):
   1. How the Load Average Works
      
      
   2. Not Your Average Average
      
      
   3. Addendum on Hz versus HZ
      
      
   
   
   
4. How to convert the absolute load average to the relative stretch factor metric 
   
   

2.24  VAMOOS Your Data Angst

1. Visualize: Make a plot of your data without making any assumptions about how it should look. This is where tools like scatter plots come in.
   
   
2. Analyze: Look for patterns or other significant features in the data and possibly quantify them, e.g., trends in the distribution of data points or periodically repeating features, such as spikes or peaks.
   
   
3. Modelize: Consider different types of models: SWAG, statistical regression, queue-theoretic, simulation, etc. If you are using the Chart>Add Trendline feature in Excel, this is where you choose your model from the Excel dialog box.
   
   
   Don't fret over whether or not it's the "right" choice: there's no such thing, at this point. Whatever your choice, try to make it consistent with steps 1 and 2. If it doesn't work out it doesn't matter because, based on the next step, you're about to come around again. :)
   
   
4. Over and Over: None of this is likely to converge in a single pass.
   
   
5. Satisfied: Iterate your way to success. Repeat until you are satisfied that all your assumptions are consistent.
   
   

2.25  Measurement Errors

Measurements are expressed using numbers, but measurement is not the same as mathematics. Mathematics involves exact quantities, e.g., the prime numbers. They are exact by definition. The number 11 is provably a prime number. Exactly. It's not approximately a prime number.

Measured numbers, on the other hand, are the result of a process involving comparisons. It's an estimation procedure and therefore cannot produce exact numbers; despite the fact that measurements are often displayed as though they were exact numbers, e.g., 11 seconds or 23% busy. More on that shortly. The measurement process tries to determine the closeness of the comparison. Since the assessed closeness is always an approximation, it can never be exact (like a prime number) and therefore, must contain errors.

In that sense, measured numbers look more like the mathematical representation of irrational numbers, e.g., π. By definition, π = C/D is the ratio of the circumference of a circle C to its diameter D. We can think of the diameter as being like a ruler or tape measure where the ratio C/D is like asking, how many diameters there are in the circumference? One possible result is the ratio 3/1, i.e., there are 3 diameters in the circumference. But you already know that the exact number 3 is less than the value of π. If the circumference was π meters long, there would be a small gap remaining after you stepped your meter-ruler three times along the circumference.

On the other hand, you might be willing to accept π = 3 as a rough approximation, and live with the gap, depending on how you plan to use that value. For example, it might be good enough to do a quick calculation in your head. Successively better approximations could involve ratios like: 7/2, 22/7, 333/106, 355/113. These ratios are analogous to reading off graduation marks on your ruler to improve the precision of your measurement. But none of these ratios or fractions are exactly equal to π. That's why π is declared to be ir-ratio-nal: unable to be written exactly as a ratio of whole numbers. Nor is a more familiar number, like 3.1415926, any better. That's just an inexact decimal fraction. They are all approximations, with attendant errors.

The more important question is, how big is the error? A secondary question is, can you tolerate that level of error?

It's a bit of an indictment that all performance measurement tools do not display errors. That omission leads people to believe (inadvertently) that a reported number, like 23% CPU busy, is the same number as the 9th prime number (viz., 23). Since all performance metrics are measured (indeed, those two words have the same root word), it should really be displayed as 23 ± 5% (for example) to remind us that there is always error or "wrongness" in the measurement.

2.26  Modeling Errors

Predicting the future is not the same thing as "seeing" the future, in the sense of seeing what things might look like. See GMantra 2.19 for more on this point. All predictions are merely estimates based on (input) data (or other estimates) and therefore predictions come with errors. The only real question is, how much error can you tolerate?

2.27  Data Ain't Information

A Google search returns a lot of data, not information. Google admits that: Are you feeling lucky? It should be a certainty, but it's not. Sneakily, Google also knows that your brain will quickly decide what is information in all those links, such that you don't even realize you're doing most of the work which you unconsciously attribute to Google. They know your brain craves patterns.

2.28  Data Science

Anything that has to call itself a "science," usually isn't. (Social Science?) How about Information Science?

3  UNIVERSAL SCALABILITY LAW (USL)