[Stoves] FW: ETHOS 2013: Where is the New Data on Stove Performance in the Field?

[Jetter.Jim at epamail.epa.gov]

Dear Crispin,

As we’ve discussed (off the list), I think you and I agree that if remaining char is actually discarded in practice, then the test metric should reflect that. If char is sometimes discarded and sometimes used in practice, then two different test metrics could reflect the two cases.

Following is my response to your message (copied below) regarding our recent article published in the journal, Environmental Science and Technology, available at the web site:
http://pubs.acs.org/doi/abs/10.1021/es301693f

Crispin: “The spreadsheet does not have a place to enter the amount of raw fuel needed to accomplish a cooking task.”

Jim: The spreadsheet does have a place to enter the amount of raw fuel.  The spreadsheet is available at: http://www.pciaonline.org/testing
Equivalent dry fuel is calculated (in the spreadsheet) to provide a more fair comparison between stoves tested with fuels with different moisture content.  It would be relatively easy to use the data already entered in the spreadsheet to calculate fuel use assuming char is discarded.

Crispin: “In the paper it is labeled 'overall thermal efficiency', but is it actually a proxy for the heat transfer efficiency…”

Jim: Overall thermal efficiency is not a proxy for heat transfer efficiency.  Please see a discussion of this in the Supporting Information (available on the journal web site), Section 10, Page S19.

Crispin: “The statistical significance (or rather, the confidence we can have in the result) is undermined by the use of an averaging technique that is not accepted by statisticians as valid.”

Jim: We performed a minimum of three replications for each stove/fuel combination we tested, and we reported results as the average and standard deviation of replicates.  The standard deviation indicates the variation between replicates, but does not indicate statistically significant differences between stove/fuel results.  We used the Student’s t-test (assuming unequal variance) to evaluate statistically significant differences for some of the results discussed in the article, but we did not do this evaluation for the entire data set.

Crispin: “A peer reviewer might have noticed, for example, that the heat value of rice hull char was credited with a heat content of 29.5 MJ/kg. This is far from the actual heat content which is closer to 12-14 MJ (an error of ?100%).”

Jim: The heat value of rice hull char was not credited with a heat content of 29.5 MJ/kg. As we described in the article and supporting information, heat of combustion for fuels and remaining char was measured using ASTM Standard Method D4442-07 (bomb calorimeter).  Measured heat of combustion values are shown in Figure S6 in the supporting information.  The heat of combustion for rice hull char from the Mayon Turbo stove seemed high, but we had a second sample analyzed with nearly the same result.

Crispin: “If the Journal provides us with the spreadsheets from the actual tests as supplementary material for the published article (normally required for publication), we will be able to reproduce the work mathematically and see what the effect of making the necessary corrections would be. I expect there would be a significant change to the ratings given to all char-making stoves…”

Jim: In the supporting information, we provided detailed information on the stoves, fuels (including moisture content and heat of combustion), cooking pots, operation (including fuel burning rates – see Figures S14-S16), test protocol, and equipment used.  I believe the information is sufficient for others to reproduce the experiments.  We can calculate fuel use with the assumption of discarded char, if there is a need for these data.  There would certainly be a significant increase in fuel consumption for char-making stoves if we assume the char is discarded.

Crispin: “What we have never seen in a Journal is a review of the procedures and mathematical methods conducted by an independent lab to put some confidence intervals on these WBT results.  Dr Penn Taylor's Masters and PhD theses have not been published. They address the issues and give a precision of about 50% for the WBT 3.1 (which was under review at the time).”

Jim: Robert Pendleton Taylor’s thesis is available at the web site:
http://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1534 <http://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1534&context=etd> &context=etd
>From Page 65 of the thesis: “Taking these errors together, the existing UCBWBT should be considered to have a minimum method error of between five and ten percent. Depending on the fuel used, the method error may be as high as twenty percent, as in the case of a fuel with a high ash content. If the test is altered to properly account for ash, the minimum method error drops to about five percent. The UCBWBT currently reports the uncertainty of fuel use and other results as the intra-test variation between three trials. It is not uncommon to see uncertainties of one percent or less reported for factors such as fuel use and energy efficiency. The sources of error identified in this thesis indicate that even if the repeatability of the test can be brought to such a fine level, the actual deviation of the test results from true values is much more likely to be on the order of ten percent. If results of the current UCBWBT are being used to compare two stove designs, the relative error in thermal efficiency, specific fuel use, firepower, turn-down ratio, and any emissions factors expressed on a per-energy or per-mass-of-fuel-consumed basis should be assumed to be ten percent, regardless of that cited as the intra-test error.”
Some of these sources of error have been addressed since the thesis was published in 2009, but many of us are working to continue to improve existing methods and to develop new methods (e.g., methods for plancha stoves and charcoal stoves).  USEPA QA personnel (independent from our project) are working on an uncertainty analysis including error propagation for the equipment and methods we are currently using.  The methodology may be useful for other labs using different equipment and methods.

Crispin: “Until the process of correcting the WBT is formalised the sorts of problems described above and in previous communications will continue to dog the stove community. In the meantime it is perfectly reasonable for stove projects to develop and use alternative test methods that are scientifically sound, properly documented and independently validated.”

Jim: With support and leadership from the Global Alliance for Clean Cookstoves, I expect we will soon be engaged in a more formalized process for developing and improving test protocols and standards.  We have challenging work to do, and I think this is an exciting time to be working together!  Thank you for constructive comments.

Respectfully,
Jim

------------------------------

Date: Sun, 25 Nov 2012 17:24:04 -0500
From: "Crispin Pemberton-Pigott" <crispinpigott at gmail.com>
To: "'Discussion of biomass cooking stoves'"
<stoves at lists.bioenergylists.org>
Subject: Re: [Stoves] ETHOS 2013: Where is the New Data on Stove
Performance in the Field?
Message-ID: <00ad01cdcb5b$96fbcfc0$c4f36f40$@gmail.com>
Content-Type: text/plain; charset="koi8-r"

Dear Dean

The tests were performed using the WBT 4:

"In this study, twenty-two cookstoves burning six fuel types (wood,
charcoal, pellets, corn cobs, rice hulls, and plant oil) at two fuel
moisture levels were examined under laboratory-controlled operating
conditions as outlined in the Water Boiling Test (WBT) protocol, Version 4."

You said: "You can see that fuel use is measured, not assumed, etc."

I beg to disagree. The fuel use is not measured, it is calculated from the
energy available from the mass of fuel burned, compensated for unburned
char. I have just had confirmation of this from Jim yesterday morning. In
other words, he followed the experimental procedure laid down in the WBT
protocol exactly. The spreadsheet does not have a place to enter the amount
of raw fuel needed to accomplish a cooking task. That is why the results of
the Quad 2 stove test (636 dry g) are so different from the actual fuel
needed to perform a 5-litre WBT (1550 dry g). 

Rocket stoves, which also produce quite a bit of char, are credited with a
fuel consumption number that is not reflective of their actual performance.
That will probably interest quite a number of organisations. It may explain
a lot of the difference between WBT, CCT and KPT results. Maybe not - there
are several other problems with the protocol and calculations.

You have many times referred testers to the CCT for field confirmation of
actual fuel use and warned people not to rely on the WBT results.  The CCT
makes the same type of calculation for 'dry fuel consumed' and does not
determine the amount of fuel the stove actually draws from the forest for
each cooking event. I checked the formulas.  Like the WBT, it too finds out
the approximate amount of heat that could be generated from the fuel and
divides it by the dry heat content of the fuel type used for that test. 

The calculation method assumes that charcoal produced is actually raw fuel
not consumed which leads to a significant misrepresentation of the
performance with certain stove types, generally crediting them with using
much less fuel than they do and over-rating the overall thermal efficiency
(but still giving a proxy for the heat transfer efficiency).

'Overall thermal efficiency' is not the same as the heat transfer
efficiency.  The WBT 4.1.2 tries to measure the HTE (see Fig 2 in the paper)
but the paper calls it the OTE. I understand the 'overall efficiency' as
being the amount of raw fuel needed to feed the stove to accomplish a
heating type task (like heating water). There are many industrial examples
of this.  In the paper it is labelled 'overall thermal efficiency', but is
it actually a proxy for the heat transfer efficiency with several losses
uncounted (as can be seen by the way the number is generated).

>And, as you can imagine, in this EPA peer reviewed study the statistical
significance is good.

Again, I  beg to disagree.  The statistical significance (or rather, the
confidence we can have in the result) is undermined by the use of an
averaging technique that is not accepted by statisticians as valid. How can
a test method (and then the derived test results obtained by using that
method) be 'statistically significant' if the comparison of individual test
results is only made after grouping them first? This demonstrates that the
peer review is only as good as the reviewers. If the reviewers do not
themselves review the test method, or do not ask that the author provide the
results of an independent review of the method, you 'get what you pay for'.
Even then, an independent review might be challenged if it contains its own
mistakes. These things happen all the time.

A peer reviewer might have noticed, for example, that the heat value of rice
hull char was credited with a heat content of 29.5 MJ/kg. This is far from
the actual heat content which is closer to 12-14 MJ (an error of ?100%). 

Because the fuel use number is calculated from the net heat value of fuel
burned (according to both me and Jim) and because the rice hull char
remaining has a far lower heat content than the number applied (according to
any reference you care to find) the stove was credited with a significantly
higher thermal efficiency than it actually has. This resulted (because of
the erroneous manner in which the fuel consumption is determined) in a much
better fuel consumption rating than it actually has for two reasons instead
of the usual one: 1) the error in the heat value for the char remaining, and
2) the error of not measuring the amount of raw fuel the stoves consumes to
perform each test.


If the Journal provides us with the spreadsheets from the actual tests as
supplementary material for the published article (normally required for
publication), we will be able to reproduce the work mathematically and see
what the effect of making the necessary corrections would be. I expect there
would be a significant change to the ratings given to all char-making stoves
and all those burning rice hull. How much? We can't tell until we see the
supporting documentation for the article. My guess? 50-100% increase in fuel
consumed per WBT and a halving of the number for the overall thermal
efficiency.  That supporting documentation will also include the version of
the WBT (might be v.4 or 4.1 or 4.1.1 or 4.1.2) so we can reproduce it
exactly.

The published article system is where the real progress comes in science and
there is no problem with challenging the content of peer-reviewed, published
material. Published articles are a form of discourse.  If something is
published without the supporting material required to replicate the result,
including calculations, it is not normally considered to have entered the
published domain and remains 'grey literature'. 

There are scores of grey literature articles, monographs and pamphlets
claiming that the WBT, CCT and KPT are good tools for developing and
determining the performance of stoves. What we have never seen in a Journal
is a review of the procedures and mathematical methods conducted by an
independent lab to put some confidence intervals on these WBT results.  Dr
Penn Taylor's Masters and PhD theses have not been published. They address
the issues and give a precision of about 50% for the WBT 3.1 (which was
under review at the time).  The accuracy was not determined.  

Berkeley's review of hundreds of old WBT 3.1's may be perfectly valid as a
method, but it's results rest on the platform of the validity of those tests
for which there is yet no independent confirmation and very real suspicion
that the results are not representative of true stove performance from an
efficiency or fuel consumption point of view. The frequent warning by
Aprovecho and others not to rely on WBT results (it is even written into the
IWA is it not?) is itself enough to view with suspicion anything based on a
review of hundreds of them made using an old, uncorrected version of the UCB
test, not so? That is what Tom Miles was asking.  What exactly were they
trying to show?

Regarding confirmation of more recent tests, there is a bit of a problem
with the WBT 4.1.2 because it appears to be getting some undocumented
changes made anonymously. I wrote to several people asking who is actually
controlling this (signing off on suggested changes). I have no name yet.
The reason I asked is because the current 19 Jun 2012 version has at least
two changes that were not present earlier in the month but the version
number is the same. 

One change I noticed is that the order of the test sheets reads Test 2, Test
1, Test3 instead of Test 1, Test 2, Test 3. The other is the introduction of
an error in the formula in cell i30 of the Results tab that materially
affects the CoV for the burn rate which may lead to a set of tests being
considered valid when they are not (as far as some labs are concerned - not
all labs use the CoV). Jim Jetter mentioned another one so that is three
just on that version.

Until the process of correcting the WBT is formalised the sorts of problems
described above and in previous communications will continue to dog the
stove community. In the meantime it is perfectly reasonable for stove
projects to develop and use alternative test methods that are scientifically
sound, properly documented and independently validated.

Regards
Crispin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bioenergylists.org/pipermail/stoves_lists.bioenergylists.org/attachments/20121129/3eac093d/attachment.html>