[Stoves] ETHOS 2013: Where is the New Data on Stove Performance in the Field?

Sun Nov 25 16:24:04 CST 2012

Dear Dean

The tests were performed using the WBT 4:

"In this study, twenty-two cookstoves burning six fuel types (wood,
charcoal, pellets, corn cobs, rice hulls, and plant oil) at two fuel
moisture levels were examined under laboratory-controlled operating
conditions as outlined in the Water Boiling Test (WBT) protocol, Version 4."

You said: "You can see that fuel use is measured, not assumed, etc."

I beg to disagree. The fuel use is not measured, it is calculated from the
energy available from the mass of fuel burned, compensated for unburned
char. I have just had confirmation of this from Jim yesterday morning. In
other words, he followed the experimental procedure laid down in the WBT
protocol exactly. The spreadsheet does not have a place to enter the amount
of raw fuel needed to accomplish a cooking task. That is why the results of
the Quad 2 stove test (636 dry g) are so different from the actual fuel
needed to perform a 5-litre WBT (1550 dry g). 

Rocket stoves, which also produce quite a bit of char, are credited with a
fuel consumption number that is not reflective of their actual performance.
That will probably interest quite a number of organisations. It may explain
a lot of the difference between WBT, CCT and KPT results. Maybe not - there
are several other problems with the protocol and calculations.

You have many times referred testers to the CCT for field confirmation of
actual fuel use and warned people not to rely on the WBT results.  The CCT
makes the same type of calculation for 'dry fuel consumed' and does not
determine the amount of fuel the stove actually draws from the forest for
each cooking event. I checked the formulas.  Like the WBT, it too finds out
the approximate amount of heat that could be generated from the fuel and
divides it by the dry heat content of the fuel type used for that test. 

The calculation method assumes that charcoal produced is actually raw fuel
not consumed which leads to a significant misrepresentation of the
performance with certain stove types, generally crediting them with using
much less fuel than they do and over-rating the overall thermal efficiency
(but still giving a proxy for the heat transfer efficiency).

'Overall thermal efficiency' is not the same as the heat transfer
efficiency.  The WBT 4.1.2 tries to measure the HTE (see Fig 2 in the paper)
but the paper calls it the OTE. I understand the 'overall efficiency' as
being the amount of raw fuel needed to feed the stove to accomplish a
heating type task (like heating water). There are many industrial examples
of this.  In the paper it is labelled 'overall thermal efficiency', but is
it actually a proxy for the heat transfer efficiency with several losses
uncounted (as can be seen by the way the number is generated).

>And, as you can imagine, in this EPA peer reviewed study the statistical
significance is good.

Again, I  beg to disagree.  The statistical significance (or rather, the
confidence we can have in the result) is undermined by the use of an
averaging technique that is not accepted by statisticians as valid. How can
a test method (and then the derived test results obtained by using that
method) be 'statistically significant' if the comparison of individual test
results is only made after grouping them first? This demonstrates that the
peer review is only as good as the reviewers. If the reviewers do not
themselves review the test method, or do not ask that the author provide the
results of an independent review of the method, you 'get what you pay for'.
Even then, an independent review might be challenged if it contains its own
mistakes. These things happen all the time.

A peer reviewer might have noticed, for example, that the heat value of rice
hull char was credited with a heat content of 29.5 MJ/kg. This is far from
the actual heat content which is closer to 12-14 MJ (an error of ?100%). 

Because the fuel use number is calculated from the net heat value of fuel
burned (according to both me and Jim) and because the rice hull char
remaining has a far lower heat content than the number applied (according to
any reference you care to find) the stove was credited with a significantly
higher thermal efficiency than it actually has. This resulted (because of
the erroneous manner in which the fuel consumption is determined) in a much
better fuel consumption rating than it actually has for two reasons instead
of the usual one: 1) the error in the heat value for the char remaining, and
2) the error of not measuring the amount of raw fuel the stoves consumes to
perform each test.

If the Journal provides us with the spreadsheets from the actual tests as
supplementary material for the published article (normally required for
publication), we will be able to reproduce the work mathematically and see
what the effect of making the necessary corrections would be. I expect there
would be a significant change to the ratings given to all char-making stoves
and all those burning rice hull. How much? We can't tell until we see the
supporting documentation for the article. My guess? 50-100% increase in fuel
consumed per WBT and a halving of the number for the overall thermal
efficiency.  That supporting documentation will also include the version of
the WBT (might be v.4 or 4.1 or 4.1.1 or 4.1.2) so we can reproduce it
exactly.

The published article system is where the real progress comes in science and
there is no problem with challenging the content of peer-reviewed, published
material. Published articles are a form of discourse.  If something is
published without the supporting material required to replicate the result,
including calculations, it is not normally considered to have entered the
published domain and remains 'grey literature'. 

There are scores of grey literature articles, monographs and pamphlets
claiming that the WBT, CCT and KPT are good tools for developing and
determining the performance of stoves. What we have never seen in a Journal
is a review of the procedures and mathematical methods conducted by an
independent lab to put some confidence intervals on these WBT results.  Dr
Penn Taylor's Masters and PhD theses have not been published. They address
the issues and give a precision of about 50% for the WBT 3.1 (which was
under review at the time).  The accuracy was not determined.  

Berkeley's review of hundreds of old WBT 3.1's may be perfectly valid as a
method, but it's results rest on the platform of the validity of those tests
for which there is yet no independent confirmation and very real suspicion
that the results are not representative of true stove performance from an
efficiency or fuel consumption point of view. The frequent warning by
Aprovecho and others not to rely on WBT results (it is even written into the
IWA is it not?) is itself enough to view with suspicion anything based on a
review of hundreds of them made using an old, uncorrected version of the UCB
test, not so? That is what Tom Miles was asking.  What exactly were they
trying to show?

Regarding confirmation of more recent tests, there is a bit of a problem
with the WBT 4.1.2 because it appears to be getting some undocumented
changes made anonymously. I wrote to several people asking who is actually
controlling this (signing off on suggested changes). I have no name yet.
The reason I asked is because the current 19 Jun 2012 version has at least
two changes that were not present earlier in the month but the version
number is the same. 

One change I noticed is that the order of the test sheets reads Test 2, Test
1, Test3 instead of Test 1, Test 2, Test 3. The other is the introduction of
an error in the formula in cell i30 of the Results tab that materially
affects the CoV for the burn rate which may lead to a set of tests being
considered valid when they are not (as far as some labs are concerned - not
all labs use the CoV). Jim Jetter mentioned another one so that is three
just on that version.

Until the process of correcting the WBT is formalised the sorts of problems
described above and in previous communications will continue to dog the
stove community. In the meantime it is perfectly reasonable for stove
projects to develop and use alternative test methods that are scientifically
sound, properly documented and independently validated.

Regards

Crispin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bioenergylists.org/pipermail/stoves_lists.bioenergylists.org/attachments/20121125/ae1a2b5f/attachment.html>