Metrics

What’s the score today?

It goes back to ancient times. The dreaded question, which you wish was never asked, but at the end of the day, you feel the breathing down your neck, followed by the sound, “How many have ye?”

Of course, a variant of this question has been asked to all of us, asking us to provide an update on the progress we made on the day, week or month. It is a perfectly legitimate question, it is a form of providing information and without adequate information, any formof management can fail.

Yes, I’m talking about metrics here, considered by many as evil and as important as oxygen by others.One of the famous quotes on metrics is ‘If it cannot be measured, it cannot be managed’. I have somewhat ambivalent feelings about this quote; it implies that all the activities need to be measured and subsequent activities are decided by these measurements. In real world scenarios, activities are not run by numbers alone. Imagine a scenario where someone prepares a plan basing only on a set of measurements, without considering the people involved, their abilities or capabilities. It is a recipe for disaster. You can argue that the capabilities can also be measured; however, measuring people’s capabilities by numbers is one of the most controversial metrics in our industry.

Don’t get me wrong, metrics by itself is not bad or evil; some of them are really useful. Take for example the burn down chart; it is the number one tool for the project manager to assess the progress of development. At the same time, poorly defined or documented metrics can be misleading as well. The statement ‘95% of the test cases are passed’ doesn’t convey any useful information to the stakeholder. Without knowing the context, it can convey a very misleading message to the recipient.

Metrics can be evil in the following circumstances.

1. They are misinterpreted without knowing the context.

Imagine a scenario where is someone is trying to assess the performance of the test team members by analysing the number of defects logged by each person per day. This is one of the classic scenarios where metrics are misinterpreted. If person A is logging more defects than person B, it doesn’t necessarily mean that person A is doing a better job than person B. It may be that person A is working on a really buggy piece of code or complex scenarios whereas person B could be working on less complex areas or already tested areas or on a product from a remarkably structured and efficient development effort.

Without knowing the context, results can be gravely misinterpreted.

2. Poorly defined metrics.

Sometimes metrics are defined without analysing the consequences. A task has been assigned to the development team, requiring them to fix a certain number of defects per day. Whatever measure you take to ensure the code quality, the primary aim of the developers will be to meet the numbers, resulting in inadequate quality.

Dilbert has got it right.

Capture

 

 

 

3. Metrics are generated just for creating beautiful charts, not really adding any value.

I have seen reports with n number of sheets, crammed with data, detailing each and every activity each and every person has done for every hour! If it requires a data mining team to get any sense of the report you are sending, the metrics are irrelevant.

4. When they are used for micro-management of individuals.

Sadly this happens, albeit rarely.

All metrics are not bad and they are really useful for making informed business decisions, with the caveat that both the sender and receiver know what they want and all the parameters are clearly understood.

The following article lists a number of useful metrics for agile management.

http://www.scrumshortcuts.com/blog/planning-metrics/scrum-metrics-and-reporting-measure-what-you-manage/

The next article emphasise the importance of proper analysis of the data you send.

http://thetesteye.com/blog/2010/08/ignoratio-elenchi/
Well, how many comments have ye?

By –
Finny Mathews, TCoE, RM ESI
www.rmesi.co.in