[Stoves] FW: ETHOS 2013: Where is the New Data on Stove Performance in the Field?

[rongretlarson at comcast.net]

Jim (cc list and Crispin) 

This is mainly to thank you for providing the link to Dr. Penn Taylor's thesis. I have now read it and I write to recommend it for others. 

The main thing I got out of the thesis was the emphasis he placed on the influence of char in the measurements. He has one chapter recommending a different type of measurement protocol for what he has termed "batch" stoves. "Batch" is equivalent to 
"charcoal-making" - the subject of another separate ("C/CO2 thread I started on using WBT measurements of the type you/EPA, Crispin, and Dr.Taylor have discussed. 

I am pretty sure that batch stove measurements of the type Dr. Taylor has recommended are inherently more accurate than the standards UCBWBT measurements. In part this is because there is no need for separating fuel from char. In part there is no need to separately measure ash. 

As this thread continues, I hope others active in these measurement protocols can comment on when/whether/where/how Dr Taylor's recommendations may have been implemented. (Any changes made since his thesis was written) Also whether there are inherently much lower errors when testing a "batch" stove. 

I will re-read this interesting thesis to better justify what I got out of a first reading. 

Ron 


----- Original Message -----
From: "Jetter Jim" <Jetter.Jim at epamail.epa.gov> 
To: stoves at bioenergylists.org 
Sent: Thursday, November 29, 2012 3:47:30 PM 
Subject: [Stoves] FW: ETHOS 2013: Where is the New Data on Stove Performance in the Field? 







Dear Crispin, 

As we’ve discussed (off the list), I think you and I agree that if remaining char is actually discarded in practice, then the test metric should reflect that. If char is sometimes discarded and sometimes used in practice, then two different test metrics could reflect the two cases. 

Following is my response to your message (copied below) regarding our recent article published in the journal, Environmental Science and Technology, available at the web site: 
http://pubs.acs.org/doi/abs/10.1021/es301693f 

Crispin: “The spreadsheet does not have a place to enter the amount of raw fuel needed to accomplish a cooking task.” 

Jim: The spreadsheet does have a place to enter the amount of raw fuel. The spreadsheet is available at: http://www.pciaonline.org/testing 
Equivalent dry fuel is calculated (in the spreadsheet) to provide a more fair comparison between stoves tested with fuels with different moisture content. It would be relatively easy to use the data already entered in the spreadsheet to calculate fuel use assuming char is discarded. 

Crispin: “In the paper it is labeled 'overall thermal efficiency', but is it actually a proxy for the heat transfer efficiency…” 

Jim: Overall thermal efficiency is not a proxy for heat transfer efficiency. Please see a discussion of this in the Supporting Information (available on the journal web site), Section 10, Page S19. 

Crispin: “The statistical significance (or rather, the confidence we can have in the result) is undermined by the use of an averaging technique that is not accepted by statisticians as valid.” 

Jim: We performed a minimum of three replications for each stove/fuel combination we tested, and we reported results as the average and standard deviation of replicates. The standard deviation indicates the variation between replicates, but does not indicate statistically significant differences between stove/fuel results. We used the Student’s t-test (assuming unequal variance) to evaluate statistically significant differences for some of the results discussed in the article, but we did not do this evaluation for the entire data set. 

Crispin: “A peer reviewer might have noticed, for example, that the heat value of rice hull char was credited with a heat content of 29.5 MJ/kg. This is far from the actual heat content which is closer to 12-14 MJ (an error of ?100%).” 

Jim: The heat value of rice hull char was not credited with a heat content of 29.5 MJ/kg. As we described in the article and supporting information, heat of combustion for fuels and remaining char was measured using ASTM Standard Method D4442-07 (bomb calorimeter). Measured heat of combustion values are shown in Figure S6 in the supporting information. The heat of combustion for rice hull char from the Mayon Turbo stove seemed high, but we had a second sample analyzed with nearly the same result. 

Crispin: “If the Journal provides us with the spreadsheets from the actual tests as supplementary material for the published article (normally required for publication), we will be able to reproduce the work mathematically and see what the effect of making the necessary corrections would be. I expect there would be a significant change to the ratings given to all char-making stoves…” 

Jim: In the supporting information, we provided detailed information on the stoves, fuels (including moisture content and heat of combustion), cooking pots, operation (including fuel burning rates – see Figures S14-S16), test protocol, and equipment used. I believe the information is sufficient for others to reproduce the experiments. We can calculate fuel use with the assumption of discarded char, if there is a need for these data. There would certainly be a significant increase in fuel consumption for char-making stoves if we assume the char is discarded. 

Crispin: “What we have never seen in a Journal is a review of the procedures and mathematical methods conducted by an independent lab to put some confidence intervals on these WBT results. Dr Penn Taylor's Masters and PhD theses have not been published. They address the issues and give a precision of about 50% for the WBT 3.1 (which was under review at the time).” 

Jim: Robert Pendleton Taylor’s thesis is available at the web site: 
http://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1534&context=etd 
>From Page 65 of the thesis: “Taking these errors together, the existing UCBWBT should be considered to have a minimum method error of between five and ten percent. Depending on the fuel used, the method error may be as high as twenty percent, as in the case of a fuel with a high ash content. If the test is altered to properly account for ash, the minimum method error drops to about five percent. The UCBWBT currently reports the uncertainty of fuel use and other results as the intra-test variation between three trials. It is not uncommon to see uncertainties of one percent or less reported for factors such as fuel use and energy efficiency. The sources of error identified in this thesis indicate that even if the repeatability of the test can be brought to such a fine level, the actual deviation of the test results from true values is much more likely to be on the order of ten percent. If results of the current UCBWBT are being used to compare two stove designs, the relative error in thermal efficiency, specific fuel use, firepower, turn-down ratio, and any emissions factors expressed on a per-energy or per-mass-of-fuel-consumed basis should be assumed to be ten percent, regardless of that cited as the intra-test error.” 
Some of these sources of error have been addressed since the thesis was published in 2009, but many of us are working to continue to improve existing methods and to develop new methods (e.g., methods for plancha stoves and charcoal stoves). USEPA QA personnel (independent from our project) are working on an uncertainty analysis including error propagation for the equipment and methods we are currently using. The methodology may be useful for other labs using different equipment and methods. 

Crispin: “Until the process of correcting the WBT is formalised the sorts of problems described above and in previous communications will continue to dog the stove community. In the meantime it is perfectly reasonable for stove projects to develop and use alternative test methods that are scientifically sound, properly documented and independently validated.” 

Jim: With support and leadership from the Global Alliance for Clean Cookstoves, I expect we will soon be engaged in a more formalized process for developing and improving test protocols and standards. We have challenging work to do, and I think this is an exciting time to be working together! Thank you for constructive comments. 

Respectfully, 
Jim 

------------------------------ 

Date: Sun, 25 Nov 2012 17:24:04 -0500 
From: "Crispin Pemberton-Pigott" < crispinpigott at gmail.com > 
To: "'Discussion of biomass cooking stoves'" 
< stoves at lists.bioenergylists.org > 
Subject: Re: [Stoves] ETHOS 2013: Where is the New Data on Stove 
Performance in the Field? 
Message-ID: < 00ad01cdcb5b$96fbcfc0$c4f36f40$@gmail.com > 
Content-Type: text/plain; charset="koi8-r" 

Dear Dean 

The tests were performed using the WBT 4: 

"In this study, twenty-two cookstoves burning six fuel types (wood, 
charcoal, pellets, corn cobs, rice hulls, and plant oil) at two fuel 
moisture levels were examined under laboratory-controlled operating 
conditions as outlined in the Water Boiling Test (WBT) protocol, Version 4." 

You said: "You can see that fuel use is measured, not assumed, etc." 

I beg to disagree. The fuel use is not measured, it is calculated from the 
energy available from the mass of fuel burned, compensated for unburned 
char. I have just had confirmation of this from Jim yesterday morning. In 
other words, he followed the experimental procedure laid down in the WBT 
protocol exactly. The spreadsheet does not have a place to enter the amount 
of raw fuel needed to accomplish a cooking task. That is why the results of 
the Quad 2 stove test (636 dry g) are so different from the actual fuel 
needed to perform a 5-litre WBT (1550 dry g). 

Rocket stoves, which also produce quite a bit of char, are credited with a 
fuel consumption number that is not reflective of their actual performance. 
That will probably interest quite a number of organisations. It may explain 
a lot of the difference between WBT, CCT and KPT results. Maybe not - there 
are several other problems with the protocol and calculations. 

You have many times referred testers to the CCT for field confirmation of 
actual fuel use and warned people not to rely on the WBT results. The CCT 
makes the same type of calculation for 'dry fuel consumed' and does not 
determine the amount of fuel the stove actually draws from the forest for 
each cooking event. I checked the formulas. Like the WBT, it too finds out 
the approximate amount of heat that could be generated from the fuel and 
divides it by the dry heat content of the fuel type used for that test. 

The calculation method assumes that charcoal produced is actually raw fuel 
not consumed which leads to a significant misrepresentation of the 
performance with certain stove types, generally crediting them with using 
much less fuel than they do and over-rating the overall thermal efficiency 
(but still giving a proxy for the heat transfer efficiency). 

'Overall thermal efficiency' is not the same as the heat transfer 
efficiency. The WBT 4.1.2 tries to measure the HTE (see Fig 2 in the paper) 
but the paper calls it the OTE. I understand the 'overall efficiency' as 
being the amount of raw fuel needed to feed the stove to accomplish a 
heating type task (like heating water). There are many industrial examples 
of this. In the paper it is labelled 'overall thermal efficiency', but is 
it actually a proxy for the heat transfer efficiency with several losses 
uncounted (as can be seen by the way the number is generated). 

>And, as you can imagine, in this EPA peer reviewed study the statistical 
significance is good. 

Again, I beg to disagree. The statistical significance (or rather, the 
confidence we can have in the result) is undermined by the use of an 
averaging technique that is not accepted by statisticians as valid. How can 
a test method (and then the derived test results obtained by using that 
method) be 'statistically significant' if the comparison of individual test 
results is only made after grouping them first? This demonstrates that the 
peer review is only as good as the reviewers. If the reviewers do not 
themselves review the test method, or do not ask that the author provide the 
results of an independent review of the method, you 'get what you pay for'. 
Even then, an independent review might be challenged if it contains its own 
mistakes. These things happen all the time. 

A peer reviewer might have noticed, for example, that the heat value of rice 
hull char was credited with a heat content of 29.5 MJ/kg. This is far from 
the actual heat content which is closer to 12-14 MJ (an error of ?100%). 

Because the fuel use number is calculated from the net heat value of fuel 
burned (according to both me and Jim) and because the rice hull char 
remaining has a far lower heat content than the number applied (according to 
any reference you care to find) the stove was credited with a significantly 
higher thermal efficiency than it actually has. This resulted (because of 
the erroneous manner in which the fuel consumption is determined) in a much 
better fuel consumption rating than it actually has for two reasons instead 
of the usual one: 1) the error in the heat value for the char remaining, and 
2) the error of not measuring the amount of raw fuel the stoves consumes to 
perform each test. 


If the Journal provides us with the spreadsheets from the actual tests as 
supplementary material for the published article (normally required for 
publication), we will be able to reproduce the work mathematically and see 
what the effect of making the necessary corrections would be. I expect there 
would be a significant change to the ratings given to all char-making stoves 
and all those burning rice hull. How much? We can't tell until we see the 
supporting documentation for the article. My guess? 50-100% increase in fuel 
consumed per WBT and a halving of the number for the overall thermal 
efficiency. That supporting documentation will also include the version of 
the WBT (might be v.4 or 4.1 or 4.1.1 or 4.1.2) so we can reproduce it 
exactly. 

The published article system is where the real progress comes in science and 
there is no problem with challenging the content of peer-reviewed, published 
material. Published articles are a form of discourse. If something is 
published without the supporting material required to replicate the result, 
including calculations, it is not normally considered to have entered the 
published domain and remains 'grey literature'. 

There are scores of grey literature articles, monographs and pamphlets 
claiming that the WBT, CCT and KPT are good tools for developing and 
determining the performance of stoves. What we have never seen in a Journal 
is a review of the procedures and mathematical methods conducted by an 
independent lab to put some confidence intervals on these WBT results. Dr 
Penn Taylor's Masters and PhD theses have not been published. They address 
the issues and give a precision of about 50% for the WBT 3.1 (which was 
under review at the time). The accuracy was not determined. 

Berkeley's review of hundreds of old WBT 3.1's may be perfectly valid as a 
method, but it's results rest on the platform of the validity of those tests 
for which there is yet no independent confirmation and very real suspicion 
that the results are not representative of true stove performance from an 
efficiency or fuel consumption point of view. The frequent warning by 
Aprovecho and others not to rely on WBT results (it is even written into the 
IWA is it not?) is itself enough to view with suspicion anything based on a 
review of hundreds of them made using an old, uncorrected version of the UCB 
test, not so? That is what Tom Miles was asking. What exactly were they 
trying to show? 

Regarding confirmation of more recent tests, there is a bit of a problem 
with the WBT 4.1.2 because it appears to be getting some undocumented 
changes made anonymously. I wrote to several people asking who is actually 
controlling this (signing off on suggested changes). I have no name yet. 
The reason I asked is because the current 19 Jun 2012 version has at least 
two changes that were not present earlier in the month but the version 
number is the same. 

One change I noticed is that the order of the test sheets reads Test 2, Test 
1, Test3 instead of Test 1, Test 2, Test 3. The other is the introduction of 
an error in the formula in cell i30 of the Results tab that materially 
affects the CoV for the burn rate which may lead to a set of tests being 
considered valid when they are not (as far as some labs are concerned - not 
all labs use the CoV). Jim Jetter mentioned another one so that is three 
just on that version. 

Until the process of correcting the WBT is formalised the sorts of problems 
described above and in previous communications will continue to dog the 
stove community. In the meantime it is perfectly reasonable for stove 
projects to develop and use alternative test methods that are scientifically 
sound, properly documented and independently validated. 

Regards 
Crispin 
_______________________________________________ 
Stoves mailing list 

to Send a Message to the list, use the email address 
stoves at lists.bioenergylists.org 

to UNSUBSCRIBE or Change your List Settings use the web page 
http://lists.bioenergylists.org/mailman/listinfo/stoves_lists.bioenergylists.org 

for more Biomass Cooking Stoves, News and Information see our web site: 
http://www.bioenergylists.org/ 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bioenergylists.org/pipermail/stoves_lists.bioenergylists.org/attachments/20121201/3057577f/attachment.html>