Guerrilla Manual Online

Hit-and-run mantras you can use on your boss or in a tiger team meeting

Updated on Jan 14, 2012

Contents

1  WEAPONS OF MASS INSTRUCTION

1.1  Why Go Guerrilla?

1.2  Best Practices

Copying someone else's apparent success is like cheating on a test. You may make the grade this time, but how far will the bluff take you into the future?

A recent NPR program discussed how this approach has backfired in the arena of medicine and healthcare. The analogy is appropriate because many aspects of performance analysis are not too different from the science of medical diagnosis.

1.3  Virtualization

Translation: We performance weenies need more whistles and less bells. In other words, virtualization vendors need to make sure they provide us with backdoors and peepholes so we can measure how resources are actually being consumed in a virtualized environment.

1.4  An Ounce of Prevention

Then, explain the multi-billion dollar dietary-supplements industry?

1.5  Why are CaP and Perf Hard?

1.6  Brisk Risk Management

"I can understand people being worked up about safety and quality with the welds," said Steve Heminger, executive director ... "But we're concerned about being on schedule because we are racing against the next earthquake."

1.7  Failing On Time

Until you read and heed this statement, you will probably have a very frustrating time getting your perforance analysis conclusions across to managment.

1.8  Performance Homunculus

• Security management
  
  
• Software distribution
  
  
• Backup management
  
  
• Chargeback management
  
  
• Cap and Perf management
  
  

Cap and Perf management can rightly be regarded as just a subset of systems management, but the infrastructure requirements for successful capacity planning (both the tools and knowledgeable humans to use them) are necessarily out of proportion with the requirements for simpler systems management tasks like software distribution, security, backup, etc. It's self-defeating to try doing capacity planning on the cheap.

1.9  Self Tuning Applications

Think about it. Performance analysis is a lot like a medical examination, and medical Expert Systems were heavily touted in the mid 1980's. You don't hear about them anymore. And you know that if it worked, HMO's would be all over it. It's a laudable goal but if you lose your job, it won't be because of some expert performance robot.

1.10  Squeezing Capacity

Today, there is a serious need to squeeze more out of your current capital equipment.

1.11  When Wrong is Right

The planning part of capacity planning requires making predictions. Even a wrong prediction is useful because it can serve as a warning that either:

1. the understanding that underlies your prediction is wrong
   
   
2. the measurement process is broken and is producing wrong data
   
   

Either way, something needs to be corrected, but you wouldn't realize that without making a prediction in the first place. If you aren't iteratively correcting predictions throughout a project life-cycle, you will only know things are amiss when it's too late! GCaP says you can do better than that.

1.12  Throw More Hardware at It

Single-threadedness can wreck you

1.13  Network Performance

If the network is out of bandwidth, has interminable latencies or is otherwise glitching, fix it! Then we'll talk about the performance of your application.

1.14  Not even wrong!

In case you're wondering, those are REAL data and yes, the axes are correctly labeled. I'll let you ponder what is wrong with these measurements. Here's a hint: They're so broken, they're not even wrong! Only if you don't understand basic queueing theory, would you accept measurements like these (which the original performance engineer did).

1.15  Checked your measurements lately?

Most people remain blissfully unaware of the fact that ALL measurements come with errors; both systematic and random. An important capacity planning task is to determine and track the magnitude of the errors in your performance data. Every datum should come with a `±' attached (which will then force you to put a number after it).

1.16  Data Are Not Divine

Data comes from the Devil, only models come from God.

Corollary: That means it's helpful to be able to talk to God. But God, she does babel, sometimes. :)

1.17  Busy work

Western culture too often glorifies hours clocked as productive work. If you don't take time off to come up for air and reflect on what you're doing, how are you going to know when you're wrong?

1.18  Little's Law

1. Eqn.(1) tells us the average number of customers or requests in residence.
   
   
2. Eqn.(2) tells us the average number of customers or requests in service.
   
   

I use it almost daily to cross-check that throughput and delay data are consistent, no matter whether those data come from measurements or models. More details about Little's law can be found in Analyzing Computer System Performance with Perl::PDQ. Another use of Little's law is calculating service times, which are notoriously difficult to measure directly. See also Mantras 2.10 and 2.12.

1.19  Bigger is Not Always Better

The bigger the symmetric multiprocessor (SMP) configuration you purchase, the busier you need to run it. But only to the point where the average run-queue begins to grow. Any busier and the user's response time will rapidly start to climb through the roof.

1.20  Bottlenecks

1.21  Benchmarks

The purpose of competitive benchmarking a computer system is to beat everyone else on performance, so you can say "mine is bigger than yours." It's the IT equivalent of war! Benchmark run-rules were made to be broken or at least bent; just don't get caught. This must be true because industrial benchmark organizations like SPEC.org and TPC.org have technical review committees that look for cheating in submitted results.

For capacity planning and system sizing, you need to be aware of this fact of life and look for the loopholes in published benchmark results.

Here, competitive refers to industrial benchmarks, as opposed to benchmarking that you might do for purely internal comparisons or diagnostic purposes.

1.22  Failure to Communicate

If your audience is missing the point or you don't really have one because you didn't spend enough time on developing it, you just wasted a lot more than your presentation time-slot.

1.23  Consolidation

1.24  Control Freaks Unite!

Control over the performance of hardware resources e.g., CPUs and disks, is progressively being eroded as these things simply become commodity black boxes viz., multicore processors and disk arrays. This situation will only be exacerbated with the advent of Internet-based application services. Software developers will therefore have to understand more about the performance and capacity planning implications of their designs running on these black boxes. (See Sect. 3)

1.25  Productivity

Thanks to the Puritans presumably, American corporate culture is obsessed with the false notion that being busy is being productive. Wrong! Europeans (especially the Mediterraneans) understand the power of the cat nap. After nearly 400 years, it's time for America to get over it.

1.26  Art vs. Science

A number of recent books and presentations on performance analysis and capacity planning have appeared with "The Art of..." in the title. In itself, this is not new. The title of Raj Jain's excellent 1991 book is The Art of Computer Systems Performance Analysis. Nonetheless, they all resort to various scientific techniques to analyze performance data. The application of science inevitably involves some art.

1.27  ITIL for Guerrillas

The ITIL framework is all about defining IT processes to satisfy business needs, not their implementation. That's what makes GCaP training an excellent IT-business solution.

1.28  Vanishing Paradox

If you do performance management perfectly, you run the risk of becoming invisible and therefore expendable to your management.

In other words, having successfully supported performance management, a manager could eventually feel justified in asking: "Why is my budget paying for performance management when everything is performing perfectly?" AKA a career-limiter.

Compare this with software development, for example. If a developer does their job perfectly, they risk being overburdened with more work than you can handle (AKA job security). In contrast to performance management, a manager might be heard to say: "We must have this new functionality in the next release!"

1.29  Dumb Question

1.30  Quantum Leap

Quantum transitions in energy are only associated with the discrete spectrum of atomic or molecular bound states.

1.31  Don't Be a Sheeple

That last one was tweeted and ended up as entry #136 in David Pogue's Twitter Book:

1.32  Performance Gatekeepers

A common situation in big organizations is for various groups to be responsible for the performance evaluation of the software or hardware widgets they create. In principle, this is a good thing, but there is a downside. Over-zealous performance optimization of any subsystem can deoptimize the entire system. To avoid this side-effect, a separate central group should be responsible for oversight of total system performance. They should act as both reviewers and gatekeepers for the performance analyses produced by all the other groups in the organization.

1.33  Performance Analysis is a Money Sink

• Performance evaluation
  
  
• Performance by design
  
  
• Performance engineering
  
  
• Performance testing
  
  

1.34  Tyranny of the 9s

downt <- function(nines,tunit=c('s','m','h')) {
	ds <- 10^(-nines) * 365.25*24*60*60
	if(tunit == 's') { ts <- 1; tu <- "seconds" }
	if(tunit == 'm') { ts <- 60; tu <- "minutes" }
	if(tunit == 'h') { ts <- 3600; tu <- "hours" }
	return(sprintf("Downtime per year at %d nines: %g %s", nines, ds/ts,tu))
}

> downt(5,'m')
[1] "Downtime per year at 5 nines: 5.2596 minutes"
> downt(8,'s')
[1] "Downtime per year at 8 nines: 0.315576 seconds"

library(NORMT3)
sigp <- function(sigma) {
	sigma <- as.integer(sigma)
	apc <- erf(sigma/sqrt(2))
	return(sprintf("%d-sigma bell area: %10.8f%%; Prob(chance): %e", 
		sigma, apc*100, 1-apc))
}

> sigp(2)
[1] "2-sigma bell area: 95.44997361%; Prob(chance): 4.550026e-02"
> sigp(5)
[1] "5-sigma bell area: 99.99994267%; Prob(chance): 5.733031e-07"

• 3σ corresponds roughly to a probability of 1 in 1,000 that four 9s availability occurred by chance.
  
  
• 5σ is roughly a 1 in a million chance, which is the same as flipping a fair coin and getting 20 heads in a row.
  
  
• 6σ is roughly a 1 in a billion chance.
  
  

2  PERFORMANCE MODELING RULES OF THUMB

2.1  What is Performance Modeling?

"90% of coding is debugging. The other 10% is writing bugs."

2.2  Monitoring vs. Modeling

2.3  Keep It Simple

A performance model should be as simple as possible, but no simpler!

"A designer knows that he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away." —Antoine de St-Expurey

I now tell people in my Guerrilla classes, despite the fact that I repeat this rule of thumb several times, you will throw the kitchen sink into your performance models; at least, early on as you first learn how to create them. It's almost axiomatic: the more you know about the system architecture, the more detail you will try to throw into the model. The goal, in fact, is the opposite.

2.4  More Like The Map Than The Metro

The Tube map is pure abstraction that has very little to do with the physical railway system. It encodes only sufficient detail to enable transit on the underground from point A to point B. It does not include a lot of irrelevant details such as altitude of the stations, or even their actual geographical proximity. A performance model is a similar kind of abstraction.
Despite several attempts, the original Tube map has hardly been improved upon since its conception in 1933. Apparently, it already met the requirement of being as simple as possible, but no simpler. The fact that it was designed by an electrical draughtsman, probably helped.

2.5  The Big Picture

2.6  Point of Principle

As an example, consider a time-share computer system. Its principle of operation can be stated as follows. Time-share scheduling gives every user the illusion that they are the only user on the system. All the thousands of lines of code in the operating system that implement time-slicing, priority queues, and so forth, are there merely to support that illusion.

2.7  Guilt is Golden

You, as the performance analyst or planner, only have to shine the light in the right place and then stand back while others flock to fix it.

2.8  What is a Queue?

1. A queue is a line of customers waiting to be severed—as in " Off with their heads!" (*)
   
   
2. Hardware version: Think of a queue as a register, e.g., a memory register.
   
   
3. Software version: Think of a queue as a list. In some languages it is a data type, e.g., Lisp, Mathematica, Perl, etc.
   
   

2.9  Where to Start?

One place to start constructing a PDQ model is by drawing a functional block diagram. The objective is to identify where time is spent at each stage in processing the workload of interest. Ultimately, each functional block is converted to a queueing subsystem like those shown above. This includes the ability to distinguish sequential and parallel processing. Other diagrammatic techniques e.g., UML diagrams, may also be useful but I don't understand that stuff and never tried it. See Chap. 6 "Pretty Damn Quick(PDQ) - A Slow Introduction" of Analyzing Computer System Performance with Perl::PDQ.

2.10  Inputs and Outputs

Take Little's law Q = X R for example. It is a performance model; albeit a simple equation or operational law, but a model nonetheless. All the variables on the RIGHT side of the equation (X and R) are INPUTS, and the single variable on the LEFT is the OUTPUT. A more detailed discussion of this point is presented in Chap. 6 "Pretty Damn Quick(PDQ) - A Slow Introduction" of Analyzing Computer System Performance with Perl::PDQ.

2.11  No Service, No Queues

If the measurements of the real system do not include the service time for a queueing node that you think ought to be in your PDQ model, then that PDQ node cannot be defined.

2.12  Estimating Service Times

Suppose, for example, you had requests coming into an HTTP server and you could measure its CPU utilization with some UNIX tool like vmstat, and you would like to know the service time of the HTTP Gets. UNIX won't tell you, but you can use Little's law (U = X S) to figure it out. If you can measure the arrival rate of requests in Gets/sec (X) and the CPU %utilization (U), then the average service time (S) for a Get is easily calculated from the quotient U/X.

2.13  Change the Data

2.14  Closed or Open Queue?

2.15  Opening a Closed Queue

An open queueing model assumes an infinite population of requesters initiating requests at an arrival rate λ (lambda). In a closed model, λ (lambda) is approximated by the ratio N/Z. Treat the thinktime Z as a free parameter, and choose a value (by trial and error) that keeps N/Z constant as you make N larger in your PDQ model. Eventually, at some value of N, the OUTPUTS of both the closed and open models will agree to some reasonable approximation.

2.16  Steady-State Measurements

2.17  Transcribing Data

2.18  Workloads Come in Threes

Apart from making an unwieldy PDQ report to read, generally you are only interested in the interaction of 2 workloads (pairwise comparison). Everything else goes in the third (AKA "the background"). If you can't see how to do this, you're probably not ready to create the PDQ model.

2.19  Better Than a Crystal Ball

Predicting the future is not the same thing as "seeing" the future. A performance model is just a means for evaluating the data that are provided to it. The model transforms those data into information. The information that can be extracted is intimately dependent on the values of those data—change the data and you change the information. For example:

1. Garbage in, garbage out
   
   
2. Unexpected behaviour
   
   

In case 2 the new data are not meeting the expectations set by previous meaurements. But it's the system that is "wrong," not the model, because the system has failed to follow the initial trend contained in the earlier measurements. The performance model forces you to ask why that happened and can anything be done to improve system performance. On the other hand, without a performance model, you don't have any context for such questions.

2.20  Patterns and Anti-Patterns

Visual example:

Textual example:

Colorless green ideas sleep furiously. —N. Chomsky (1957)

New research indicates that if a person is not in control of the situation, they are more likely to see patterns where none exist, suffer illusions and believe in conspiracy theories.

2.21  Interpreting Data

The conventional view of performance models is that they are useful for:

1. Predicting the future performance of an extant system
   
   
2. Exploring what-if scenarios that may or may not be realistic
   
   

A performance model can also be used for interpretating performance measurements. Both the measurements and the model must be consistent or something is wrong and needs to be explained.

2.22  Intuition and Modeling

Ulysses is the the Greek hero in Homer's Odyssey. On his way back from the Trojan wars, Ulysses orders his men to tie him to the mast of his ship and to plug their own ears so that they will not succumb to the beautiful song of the sirens and be diverted to their deaths. Ulysses, being a typical manager, chooses to be bound and to keep his ears unplugged because he cannot bear the idea of not hearing the sirens' music.

2.23  Load Average

For a more complete discussion, see:

1. See Chap. 4 of my Perl::PDQ book
   
   
2. Read the original online articles
   
   
3. How to convert the absolute load average to the relative stretch factor metric 
   
   

2.24  VAMOOS Your Data Analysis Hesitations

1. Visualize: Make a plot of your data without making any assumptions about how it should look. This is where tools like scatter plots come in.
   
   
2. Analyze: Look for patterns or other significant features in the data and possibly quantify them, e.g., trends in the distribution of data points or periodically repeating features, such as spikes or peaks.
   
   
3. Modelize: Consider different types of models: SWAG, statistical regression, queue-theoretic, simulation, etc. If you are using the Chart>Add Trendline feature in Excel, this is where you choose your model from the Excel dialog box.
   
   
   Don't fret over whether or not it's the "right" choice: there's no such thing, at this point. Whatever your choice, try to make it consistent with steps 1 and 2. If it doesn't work out it doesn't matter because, based on the next step, you're about to come around again. :)
   
   
4. Over and Over: None of this is likely to converge in a single pass.
   
   
5. Satisfied: Iterate your way to success. Repeat until you are satisfied that all your assumptions are consistent.
   
   

2.25  Measurement Errors

Unlike mathematics, measurement is a process (an experimental procedure) and therefore cannot produce pure numbers like the prime numbers. The measurement process always produces errors. The real question is, how big are those errors and can you tolerate them? It's a bit of an indictment that most of our performance analysis tools do not display errors, and that leads to people to think that 23 percent CPU busy is the same as 23 the 9th prime. It should read something like: 23±5% to remind us that there is an associated error range.

2.26  Modeling Errors

Predicting the future is not the same thing as "seeing" the future, in the sense of seeing what things might look like. See GMantra 2.19 for more on this point. All predictions are merely estimates based on (input) data (or other estimates) and therefore predictions come with errors. The only real question is, how much error can you tolerate?

2.27  Data Ain't Information

A Google search returns a lot of data, not information. Google admits that: Are you feeling lucky? It should be a certainty, but it's not. Sneakily, Google also knows that your brain will quickly decide what is information in all those links, such that you don't even realize you're doing most of the work which you unconsciously attribute to Google. They know your brain craves patterns.

2.28  Data Science

Anything that has to call itself a "science," usually isn't. (Social Science?) How about Information Science?

3  UNIVERSAL SCALABILITY LAW (USL)