[Stoves] The WBT allows for comparison between tests and stoves, says Winrock

[Derby, Elisa]

Hi Xavier,

ETHOS is an all-volunteer nonprofit with a very small operating budget. Through some generous member donations we are able to support one international scholarship per year, and that is reserved for practitioners from developing countries, so that we can bring their field experience to the conversation. That said, we intentionally try to keep the registration, lodging and meal costs very low to enable broader access, so if you see a cheap flight from Paris to Seattle, keep that in mind!

Looking forward to connecting live-give me a call after Thanksgiving.

Logging off now-Happy Thanksgiving to all who celebrate!
Elisa

From: Xavier Brandao [mailto:xvr.brandao at gmail.com]
Sent: Tuesday, November 21, 2017 5:25 PM
To: Derby, Elisa <ederby at winrock.org>
Cc: 'Discussion of biomass cooking stoves' <stoves at lists.bioenergylists.org>
Subject: RE: The WBT allows for comparison between tests and stoves, says Winrock

Dear Elisa,

Thanks for the quick answer.

I carefully re-read the Winrock toolkit.
I will still make that same statement.

I saw what you mentioned, and agree that Winrock indeed insists in this toolkit on the importance of knowing local context and habits, making field testing and that lab testing is not always representative of what happens in the field.
But still, when it comes to testing, the toolkit gives a strong and misleading impression that the WBT should be used. That it is a great tool to test, compare and even select stoves. And that if you can add field testing, that's even better.

I am gonna quote below extensively the toolkit, and no, I am not cherry-picking.

By the way:

·         The WBT is mentioned 8 times

·         The CCT is mentioned 6 times

·         The KPT is mentioned 4 times

No other protocol is mentioned.

Page 19 « Best practices 3. Undertake in-country testing of the cookstoves you plan to promote, to ensure that they perform well with local foods and fuels. »
This is good. Here field tests are advised. As best practices, not prerequirements, but still.

Page 19 « Standardized laboratory testing protocols and metrics allow for replicability and comparability among tests and across stoves. As stoves often perform differently when consumers use them in their own homes, field testing in consumer homes often provides a more realistic picture of actual stove performance »
How people will read it: « lab protocols allow for a realistic picture. But field testing allows for a more realistic picture. »
They will interpret it as: « lab is good, field is better ». As a small manufacturer with no money, I could rely on the good results and the (« ISO » as the Aprovecho says) certification given by the lab. I would not necessarily proceed with a field testing because it is (deemed) long, painful, costly to execute.
The issue is that the lab testing the toolkit talks about is catastrophic, it is the WBT.

Page 19 « Laboratory testing often represents the best possible performance under ideal conditions »
No, the performance is not representative, because there are problems with the metrics and the calculations, it is not repeatable and the statistical approach is wrong.
« but cookstoves that perform well in the lab (where the standard test is boiling water) may not always perform well cooking a specific local food, or with nonstandard fuels. »
Note that here there is a warning. This is a good thing. I wished the toolkit would have said: « Never develop a stove without testing it in the field. Field testing is compulsory ».

Page 20 « Three common standardized tests are the Water Boiling Test (WBT), the Controlled Cooking Test (CCT), and the Kitchen Performance Test (KPT). The laboratory-based WBT is the most standardized of the three, and allows for comparison between stoves. »
I don't see how one would not want to use the WBT after reading that. Why choose another protocol when you can get « the most standardized of the three », that «allows for comparison between stoves »?
Again, the WBT does not allow for anything but wasting time and money.

Page 20 Then the toolkit gives a link to the GACC website, with the WBT at the top of the page:
http://cleancookstoves.org/technology-and-fuels/testing/protocols.html

Page 20 « Testing protocols are consistently updated and reviewed by the global community to ensure methods for lab and field testing provide the best possible depiction of how each stove performs »
Fantastic, so the « global community » made sure for me the last version of the WBT, the 4.2.3., was really top-notch and fully updated. I can now sleep on both ears.

Page 20 « For example, testing protocols are being evaluated and updated through an ongoing international standards process (see below). »
What I got from some people is that the WBT is being dropped by the TC 285 and that they are not working on the issues highlighted by the many studies about the WBT.
So why are we still talking about the WBT in handbooks and toolkits anyways?

Page 21 « That said, the tiers developed were based on the Water Boiling Test, as the sector has the most data available for that test, with the future goal being to develop tiers for other testing protocols. »
Decidedly, this WBT seems like a great protocol. Yes, I think I'll use it to test my stove, so I can compare it with the competition and place it on tiers!

At the bottom of the page 22, there is even a very nice graph, showing the tiers, under 3 of them, it is written « Water boiling test ».

Page 28 (it is in bold in the text) « All stove and fuel combinations should be tested by a reputable, nationally or internationally recognized testing lab, before being deployed in any local context »
Here it is. So, lab testing is a prerequisite. What are the reputable testing labs? Aprovecho?

Page 35 « Conducting all of the research techniques provided in the WASHplus toolkit can take a fair amount of time and resources. At a minimum, those interested in promoting cookstoves in a given context should undertake a simple controlled cooking test with a local cook cooking a typical local meal to at least rule out inappropriate cookstoves, and a few focus group discussions to get initial feedback on cookstove features desired by the consumers. »
Sounds really like a pain. I am not interested to sell my cookstoves in a given context only anyways, I want to export containers to big clients around the world! I think I'll just skip the field test or do only one simple CCT like recommended.

Page 57 « The laboratory-based WBT is the most easily controlled of the three, allowing for comparison between stoves. The CCT uses a controlled field environment to assess the performance of stoves relative to the traditional baseline option» « The KPT produces a more 'real- world' estimate as it measures the impact of the introduction of the stove on all household fuel use1, meaning that it takes into account not just the intervention stove performance, but also the degree to which consumers displace their traditional cooking option. This test is carried out in a larger number of households once the stove has shown positive results using the WBT and/or the CCT »
Again, reading that, if I could do only one test, I'd do the WBT. It seems we seek first for positive results with the WBT, then we go to the field.

Page 57 « Emissions monitoring is most often done in conjunction with the other testing protocols described above, and can be done in both lab and field settings.  The Water Boiling Test, for example, has a section on emissions testing as part of the formal protocol. »
On emissions: only the WBT is mentioned, no other protocols. Other lab protocols could have been mentioned.
I don't know anyone who would not want to know the stove emissions.
So what I would understand: WBT really is the go-to protocol.

So after all that, how I would not want to test my stove with the WBT? How would I not be mislead?

I am sorry, but this toolkit is unfortunately putting one more layer of cement on the status quo, on the WBT. But underneath that, the WBT is rotten. This toolkit is keeping us in the past.

I am waiting for a document that will, at last, make the choice not to talk about the WBT, and present the new alternative protocols. Do we want progress or not?

The fact that the toolkit advises to do field testing does not make the many mentions and promotion of the WBT any less problematic. Page 28 : lab testing (i.e. WBT) is strongly advised, before anything else. The GACC, D-Lab documents, even the Aprovecho documents admitted doing field tests was strongly advised.
But they never disavowed the WBT. The Winrock toolkit is not disavowing it neither, on the contrary!

You wrote: "If you read the whole toolkit and still have that impression, please give me a call, as that would be a truly problematic take-away"
Sure, that would be great to make a phone call, thanks for offering that. When there are critics, it is rare people are ready to discuss them in person, so thanks for that. I'll write to you to set up a call.
But first I would like the conversation to continue a bit here the List and would like to have your answer on the above, because I think everyone is concerned and needs to know.

You seem to want improvements and be ready for changes. That is great. I think the toolkit can easily be modified and reuploaded. It needs to state that there are serious concerns about the WBT, mention alternative protocols, and requires absolutely that field testing is performed.
That can be done easily.

You wrote: "Even better-come to ETHOS at the end of January and we can discuss further; you'd be welcome to also present to the group on your ideas and suggestions for the sector."
That is great, thanks for the invitation. I would love to go to ETHOS, if my schedule allows me, I've never been. I'd love to join and to discuss with everyone. The issue is that I live in France, and cannot pay for the trip from my personal budget.
Is ETHOS able to cover my travel expenses?

You wrote: " Have you considered joining TC 285 to help inform the testing protocols our sector is developing through the consensus-based ISO process? "
Yes, very seriously, and I actually tried: I asked the AFNOR if France could join, but they never replied to my queries.
I have been asked this question several times, notably by Sally Seiz and Ranyee Chiang, but haven't been offered the opportunity. I'd be grateful to hear from someone or find reports somewhere about the on-going talks.

And I have a final question: do you agree Elisa that the WBT is an unreliable protocol, which is no good to develop, compare, nor select stoves?

Best,

Xavier




De : Derby, Elisa [mailto:ederby at winrock.org]
Envoyé : mardi 21 novembre 2017 04:13
À : Xavier Brandao
Cc : 'Discussion of biomass cooking stoves'
Objet : RE: The WBT allows for comparison between tests and stoves, says Winrock

Dear Xavier,

Thanks for your message and for reading the toolkit, I'm glad you were able to find it. I can't imagine that anyone who has ever worked with Winrock or even read just the consumer preferences section of the toolkit could get the impression that Winrock would recommend making stove selection decisions based only on *any* laboratory test.

It seems you may have missed the "best practices" intro to that same section you quote that emphasizes the importance of local testing of stoves to ensure that they perform well with local foods and fuels.  And where we say that cookstoves that perform well in the lab won't necessarily perform well in homes cooking specific local foods, or with nonstandard fuels. And where we highlight the importance of updating and reviewing testing protocols to ensure methods for lab and field testing provide the best possible depiction of how each stove performs.

If you read the whole toolkit and still have that impression, please give me a call, as that would be a truly problematic take-away, and I'd like to better understand your perspective.  My direct line is 1-617-524-0466 or we can talk on skype.  Even better-come to ETHOS at the end of January and we can discuss further; you'd be welcome to also present to the group on your ideas and suggestions for the sector. It's a great place to have these kinds of discussions in-person.

Have you considered joining TC 285 to help inform the testing protocols our sector is developing through the consensus-based ISO process? Or the guidelines for how these voluntary standards get explained and rolled out? Having more perspectives at the table makes for a better outcome.

Kind regards,
Elisa

From: Xavier Brandao [mailto:xvr.brandao at gmail.com]
Sent: Sunday, November 19, 2017 12:38 PM
To: 'Discussion of biomass cooking stoves' <stoves at lists.bioenergylists.org<mailto:stoves at lists.bioenergylists.org>>
Cc: Derby, Elisa <ederby at winrock.org<mailto:ederby at winrock.org>>
Subject: The WBT allows for comparison between tests and stoves, says Winrock

Dear all,

Winrock has recently published, on September 2017, on its website, a toolkit:
https://www.winrock.org/wp-content/uploads/2017/09/Winrock_Cookstove_final_reduced.pdf
The toolkit « can be used by various stakeholders, but is primarily aimed at bringing USG staff and external project developers and implementers up-to-speed on significant developments in the cookstove sector in recent years. »

According to the toolkit, p 20, « Standardized laboratory testing protocols and metrics allow for replicability and comparability among tests and across stoves », and « The laboratory-based WBT is the most standardized of the three, and allows for comparison between stoves. »

It is unbelievable that Winrock still publishes this kind of claim, today. I am baffled. This has been time and time again proven wrong, and time and time again repeated, on this list and elsewhere that the WBT did not allow to compare stoves. That it should not be used to select stoves, for programmatic or project purposes.
Even the very rare supporters of the WBT (I am not sure who they are actually) agree about that.

I am not even talking about the proofs that the WBT is not even good to develop stoves.

In February 2017, a study by Lombardi and al. summarized the issues:
« Some of WBT critical issues remain unsolved. In particular, the main weakness of the WBT concerns its real-life relevance. [...] Criticism about WBT concerns also the repeatability of the protocol, with a number of researchers claiming that it would need to be reviewed in terms of accuracy. [...] As a matter of fact, uncertainties related to temperature reading and vaporisation in the boiling region lead to high variability between test replicates.
A lot of debate has been made around formulation of metrics, primarily on thermal efficiency, which is often interpreted as the most immediate and distinctive stove performance parameter. Studies from Bailis et al. highlighted how relying on WBT thermal efficiency outputs, regardless of the relative importance of high and low power cooking tasks among the target population, can lead to misleading interpretations. Furthermore, Zhang et al. and Jetter et al. questioned the scientific meaningfulness of thermal efficiency at simmering.
Finally, some unsolved issues concerning statistical significance of data are worth mentioning. WBT 4.2.3 includes "Statistic Lessons for Performance Testing". The appendix specifies that the minimum number of test replicates for each model of stove should be three, [...] Wang et al. investigated this topic using a simplified version of the WBT 3.0 and demonstrated that more than 5 replicates are likely to be required to avoid impractically large 95% confidence intervals and that even more replicates may be required to demonstrate a statistically significant difference in performance between two or more stoves. »

Yet, Winrock, like the GACC and D-Lab and their handbook, are again proving themselves to be the gravediggers of the already struggling project developers and implementers, rather than the much needed support they should be, on matters of testing.

This is totally irresponsible from Winrock, and very concerning.

I put Elisa Derby in copy of this email.

Elisa, I am very much looking forward to hear your views on that.

Best,

Xavier


[Image removed by sender.]<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>

Garanti sans virus. www.avast.com<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>



The information in this message, including any attachments, is the property of Winrock International (discloser), is intended solely for the use of the individual(s) or entity to which this message is addressed, and may contain confidential or privileged information. Therefore, the use or disclosure by a person who is not an intended recipient is prohibited. Use or distribution by the intended recipient is permitted if distribution is pursuant to an agreement with discloser authorizing distribution, or distribution is to those that have been previously approved by the discloser for sharing. If you are not the intended recipient, or the person responsible for delivering this message to the intended recipient, please notify the sender and immediately delete this message.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bioenergylists.org/pipermail/stoves_lists.bioenergylists.org/attachments/20171122/15e6b244/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 350 bytes
Desc: image001.jpg
URL: <http://lists.bioenergylists.org/pipermail/stoves_lists.bioenergylists.org/attachments/20171122/15e6b244/attachment.jpg>