[Stoves] ETHOS 2013: Where is the New Data on Stove Performance in the Field?

Sat Nov 24 23:31:59 CST 2012

Dear Tom

"If the variation from testing is greater than the variation required to
elevate a stove between tiers then why bother with tiers to categorize
stoves? By using averages have we just removed the variability? If we don't
keep the outliers in the data set how can we analyze them to determine
sources of variation or error? I haven't seen evidence that anyone has
really questioned the data."

Well Tom, you have nailed that one as well. The WBT 4.1.2 asks the tester to
repeat the test three times and to fill in the form. The individual results
have a coefficient of variation (CoV) on the Results page. Some are using
this or similar tests who say that if the CoV is outside a certain range
(for example 25%) the tests should be discarded and it should be done over.

Now suppose the CoV 10%. What does CoV mean? It means the Standard Deviation
divided by the Mean (average) of the results.

Using the most recent version of the WBT 4.1.2 file dated June 2012 from

http://www.pciaonline.org/files/WBT_data-calculation-sheet_4.1.2_updated_19-
Jun-2012.xls

I entered the values 10, 11 and 12 g per minute into cell AP31 for tests 1,
2, and 3 respectively. That is where the calculated burn rate of fuel for
the low power portion of the test is found, normally called the simmering
phase. Just enter it manually.

The Result page pulls these results into line 30 and it will bring show
three numbers in cells G30, H30 and i30. The numbers will be 10, 11 and 11
respectively. It will not show 10, 11 and 12 because Cells H30 and i30 both
look at Test 2. So correct the formula in i30 to read 

='Test-3'!$AP31 

The CoV is 9%. This is nothing close to the precision so don't get excited
just yet.

A difference of 1 g per minute in the burn rate in 10 gives a Standard
Deviation of 1. For a TLUD gasifier stove with a very constant burn rate
like the Quad 2 (for which the test is publicly available) the result is St
Dev 0.1 and a CoV of 1.2%. Highly replicable.

It has been said that a Cov of 25% is acceptable. In order to get that
result change the test results for AP31 cells to 7.5, 10 and 12.5. The CoV
is 25% because the St Dev is 2.5 and the Mean is 10. 

Consider that the test method is the same and the fuel and everything is the
same for all three tests, but the result varies by 25% on this one section.
This indicates a high variability, yes? Is this result acceptable? Can you
confidently place a stove on tiers separate by 20% if one metric, a portion
of one final metric, the fuel consumption, varies by 25%? The mathematical
answer is 'no'. The total variation has to be less than 1/3 of the gap. And
this is just for one variation.

The graphic error bar for a single test is not a dot. There should be an
error bar and it relates to the precision of the test metric, with all the
errors properly rated and summed using a standard method which I will not
explain here. Suffice to say that if there is an error bar of 10% above and
below the values for each test, that is the result that should be plotted
for a single test, and all subsequent tests. They should not be averaged
first into groups of three. Averaging does not reduce the error for each or
any test. It does not increase the accuracy nor the precision.

What is happening is there are no error bars being shown on stove test
results. The average result is usually plotted and the upper and lower test
numbers are shown with a vertical line as if they are error bars but they
are actually results, not indications of error or precision or imprecision. 

The average of some results is not the result. The results are the results.
The fact that the protocol calls for three results to be averaged does not
make the test method a valid way to provide a rating. 

What is happening now is that when the stove is re-tested three new times, a
second average of results is calculated. In the Berkeley paper the averages
were them compared with each other. For example the average of two sets of
tests were found to be 6% different. So what? That is not a useful number.
It does not indicate anything about the accuracy or precision of the test
method.

The tests stand individually because there is a test method and following it
produced the results with all the procedural and systematic errors that
occurred that day. If repeating the test using the same method does not
produce results that are within 1/3 of the Tier gap, then the test method
(or the person testing) does not show the necessary skill to place stoves on
a tier with confidence. Averaging groups of tests and then comparing the
group results is not a valid way if determining if the 1/3 tier gap has been
met.

When you are performing a health survey in a community you normally have to
submit your method to a review board who will check for errors like that.
Then you correct it, then you are allowed to proceed. It should not be
different for the methods used for testing stoves. In fact the IWA calls for
it to be done properly.

Regards

Crispin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bioenergylists.org/pipermail/stoves_lists.bioenergylists.org/attachments/20121125/751574a7/attachment.html>