[Stoves] ETHOS 2013: Where is the New Data on Stove Performance in the Field?

Crispin Pemberton-Pigott crispinpigott at gmail.com
Fri Nov 23 14:15:58 CST 2012


Dear Tom and Dean

 

Tom asks: >Do the tests mean anything? 

 

The tests give calculated results. Numbers. The interpretation of those
numbers is what matters as much as the numbers themselves. But first the
numbers have to be valid. Each metric in a test has a precision attached to
it. For example you cannot get a 0.1% precision from a gas cell that reads
to 2.5% precision by measuring 25 times. That is basic science. You can nail
down where the centre of the error bar is, but all readings are still ±2.5%.
When all the actions and readings in a test are rated in this manner, you
get an overall precision. According to Penn Taylor the WBT precision is
about ±50%. I have found a number of systematic errors additional to the
ones he pointed out in his Thesis which increase it to about ±80% for most
stoves, ±200% for TLUD gasifiers. So before we go creating averages of final
outputs, we need to be clear that a complicated test with imprecise steps
cannot give precise results.

 

This still says nothing about accuracy. Accuracy asks, Is the result is
close to the ‘real’ result? The real fuel consumption, for example, is
determined by running a series of tests and taking the one in the middle
where the old fuel and the char and the ash are dealt with ‘on a daily
basis’ and the amount of new fuel required to keep the stove going each
cooking session is assessed. This straightforward approach removes all sorts
of systematic errors. Measured carefully, such a method can give a result
within perhaps 2% of the ‘truth’. Now compare the result of a ‘standard’ WBT
result with that result. There is always a large difference. That difference
shows the accuracy or lack of it in the WBT.

 

Tom wrote>
We should be at the point where we have large numbers of tests
for each condition to arrive at performance numbers rather than having to
rely on averages of averages of data generated by using outdated versions of
the testing protocols. 

 

Averages: So you noticed that too? Good. Taking three tests and creating an
average of them gives another number. The error bar on those numbers is not
reduced in size by the averaging process. Comparing one set of averaged
numbers with another set of averaged numbers is an invalid method of
demonstrating the precision or accuracy of the answers given. This is basic
statistics. I was surprised to see Berkeley doing this in the report. They
should get a red X and be sent back to do it properly.

 

Tom wrote: >If there are still flaws in the testing protocol does the whole
stack of cards fall down?  

 

Of course it does. It is well known that the results from the UCB-WBT 3.1
are not a valid reporting of fuel consumption or efficiency of stoves. That
is why the exercise to correct the method was started by ETHOS, and it got
started when Berkeley realised it. I say ‘well known’ because it was
discussed here 4 years ago in detail and hundreds or thousands of people in
the stove community are aware that the WBT 4.1.2 was created to try to
correct the most significant errors in the 3.1 version.

 

How then can any general statement be based on hundreds of 3.1 version
tests? Have the result been corrected even to the 4.1.2 stage? Take for
example the calculation of the fuel used to complete a 5 lite WBT which is
near the bottom of the Results page. It reports a mass of dry fuel which it
indicates is the consumption of fuel needed to boil then simmer 5 litres of
water for 45 minutes. (To be clear, the EPA lab does not use this metric).
The figure is a concatenation of several conceptual errors and one invalid
mathematical step. The final number is often used to compare stove
performance against Aprovecho’s 850 g benchmark for an improved stove. There
are many examples published on line. But the number is completely invalid as
a measure of fuel consumption. Just look at the formulas.  Any small change
in the mass of char remaining after simmering creates a large change in the
final number well in excess of any possible change in the fuel consumed to
create that change in char mass. That is a hint there is something terribly
wrong. These days we are seeing stove test results that are patently
impossible.

 

Tom wrote: >What is the statistical probability of changing the outcome if
we find that we have to change one element the protocol (e.g. assumption
about charcoal) to make it more accurate?

 

The call is I guess to ‘fix it’. A lot of us have had a go at ‘fixing it’ by
pointing out what the conceptual and methodological and statistical problems
are. Who is listening and who is not? 

 

Dean, I want to run this one by you: The WBT says that the quantity of water
boiled at the end of phase 1 and 2 is the water remaining in the pot, and
after the simmering phase it is the mass of water remaining in the pot. You
were always very clear on this approach and always rejected the alternative.
The alternative is that when you put in 5 litres of water and bring it to a
boil, you have boiled 5 litres. If there is 100 cc of water missing, it has
been boiled away otherwise it would not be missing (obviously).

 

So the fuel consumed should be divided by which figure? The 5 litres that
were boiled or the 4.9 that remains? You and I have disagreed about this.
The error created in the WBT 5 liter benchmark number is mostly caused by
choosing to use the final mass instead of the initial mass, three times in a
row. That is what I mean by a concatenation of errors. In order to correct
this basic error (Specific fuel consumption to boil) will require giving up
the concept that the missing 100 g of water was not boiled by the fuel that
was burned and that the volume boiled is always 5 litres.

 

Is there a willingness to accept this? If the correct formula is applied the
5 litre Benchmark number doubles for some stoves and does not change at all
for others. This shows it is a systematic error that cannot be ‘mapped’ to
some other performance assessment method because it applies to each stove to
a different degree. A doubling is a pretty major correction if you were to
apply it to everything in the Berkeley analysis document.

Tom asks: >What is the impact on our understanding of stove performance and
what stoves we need to improve, or abandon?

 

The impact is massive!

 

Tom asks: >Another important question: do failures in stove programs have
anything to do with stove technology or are they due to problems with the
implementing organizations? 

 

Most stove programmes start off assuming the solutions already exist and the
thing to do is to find out which ones are ‘best’ then roll them out. ‘Best’
has often turned out to be a lab evaluation based on some or other WBT
containing all or most of the errors in the UCB-WBT. The fact is that these
WB Tests can’t meaningfully tell the difference between stoves. The CCT is a
simplified field version that as I pointed out earlier today does not
measure the amount of fuel consumed. It calculates the dry fuel equivalent
of the energy used in the thermal efficiency calculation. So the ‘field
test’ can’t tell the difference between stoves either. Good grief! Yet we
still have proclaimed documents lauding both as valid methods to be used for
determining what to promote.

 

Tom asks: > Are there things we can do to strengthen our stove
organizations? 

 

For a start they should start asking for due diligence checks on their
methods! Do you think Health organisations can spend a couple of million $
on a survey using a method that has not been checked for suitability and
precision, accuracy and appropriate statistical method? Why is it
‘different’ with stoves?

 

>This is going to take a lot of beers. . .

 

In the circumstances a tempting prospect even to an abstainer


 

Regards

Crispin

 

Tom   

 

From: Stoves [mailto:stoves-bounces at lists.bioenergylists.org] On Behalf Of
Dean Still
Sent: Friday, November 23, 2012 9:21 AM
To: Discussion of biomass cooking stoves
Cc: ethos at vrac.iastate.edu
Subject: Re: [Stoves] ETHOS 2013: Where is the New Data on Stove Performance
in the Field?

 

Hi Tom,

 

Great suggestions for ETHOS!

 

 A Stove Performance Inventory was released this week with both lab and
field data funded by the Global Alliance.

 

"This Stove Performance Inventory, developed by the Berkeley Air Monitoring
Group in partnership with the Alliance and with funding from Environment
Canada, contains data from over 600 sets of cookstove performance tests.  A
detailed report
<http://www.cleancookstoves.org/resources_files/stove-performance-inventory-
pdf.pdf>  is available in the Resources section of the Alliance’s website."

 

Best,

 

Dean

 

 

 

 

On Fri, Nov 23, 2012 at 8:36 AM, Tom Miles <tmiles at trmiles.com> wrote:

Mark,

 

You have listed a number of standards and protocol topics. Now that GACC
exists have we made any gains on the science side? Where is the data?  We
should be generating good field data by now. I don’t want to just hear
reports about what people have decided in political committees that nobody
seems to be able to attend or vote in.  

 

I would like to see a critical review of the testing protocols and methods
with regard to the key metrics and emissions. For example, the protocols to
date have made assumptions about heating values of fuels and residues ash
and charcoal that I have always felt were just placeholders until someone
with larger research budgets could validate them. Determining the amount of
energy left in the ash or charcoal is a good example. I often get asked what
value to use for the remaining ash/charcoal. I haven’t seen a test where the
remaining charcoal/ash has been directly tested for ash content and heating
value. 

 

Testing the benchmarks. Do the benchmarks that were derived several years
ago make sense now that we have improved tools for measuring stove
performance or do we get the same numbers because we’re using the same
tools? Do the benchmarks tell us anything about stove performance in the
field? Do stoves preform in the field in the same relative way they are
shown to perform in the lab or are some stoves much better than others (or
much worse) when they are used in the field? Are the stoves designed to the
test (e.g. WBT) or to the use in the field? Do some stoves perform best when
they are tested in the lab and fail in the field? Or, are we even testing
for this?

 

Field applications. Are our tools and metrics of any use for improving stove
performance in actual use? If so then how are projects in developing
countries using these tools to improve their fuel use and health? How do
local, nation, or regional stove projects use these tools to improve their
stoves, or do they just ignore them?

 

QA/QC. When a program buys container loads of stoves how do we know that
they perform within the expectations created by the test results? Do any of
the manufactured stove suppliers test the quality and performance of their
stoves on a regular basis? Is there any monitoring?

 

How has testing been used for different fuels? I was inspired this week by a
photo from Mexican which a construction worker was using an LPG burner in an
eCocina stove (Stove Team International) because it substantially reduced
his LG use. Can we compare fuel consumption for different fuels? How good
are our fuel consumption metrics? 

 

Health. We still do not have proven direct correlation between stove
emissions and heath. Most of the data seems to be recycled. Are there new
health studies? Has GACC and the many supporting organizations funded any of
the fundamental health studies that every year Jay Smith tells us are
lacking? 

 

These are some of the questions that I would like to see addressed at ETHOS.


 

Looking forward to another productive ETHOS. 

 

Thanks

 

Tom  

T R Miles Technical Consultants, Inc.

tmiles at trmiles.com

www.trmiles.com

www.stove.bioenergylists.org

 

 

 

 

 

 

 

 

From: ethos-bounces at vrac.iastate.edu [mailto:ethos-bounces at vrac.iastate.edu]
On Behalf Of Bryden, Kenneth [M E]
Sent: Monday, October 22, 2012 2:05 PM
To: Discussion of biomass cooking stoves; ETHOS - Listserve
Subject: [Ethos] ETHOS 2013

 

All, 

 

It's that time of year again! We're starting to get the activities together
together for the ETHOS conference. The conference will be January 25 - 27,
2013 in Seattle. Proposed discussion topics include

 

- Update on the Global Alliance and their activities

- Standardizing Reporting on IWA Indicators

- Stove Performance Inventory, Sharing Public Data, and Establishing Common
Data Formats

- Update on ISO Process

- Updates on Protocol Developments (including possibly charcoal, plancha,
batch-fed, durability, finalizing WBT from public comment period)

 

Let us know what else you would like to talk about. A lot is happening and
I'm sure it will be as exciting as ever. I'd appreciate having your ideas on
topics and panels by November 15.

 

Abstracts for papers and talks are due January 1.

 

For more details, to register, and to submit your abstract the conference
web site is http://www.vrac.iastate.edu/ethos/conference.php.

 

Please send your ideas directly to Dean and I.

 

Best regards

Mark

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bioenergylists.org/pipermail/stoves_lists.bioenergylists.org/attachments/20121123/1bb05135/attachment.html>


More information about the Stoves mailing list