[Stoves] The WBT allows for comparison between tests and stoves, says Winrock

[Xavier Brandao]

Dear Elisa,

 

Thanks for the quick answer.

I carefully re-read the Winrock toolkit.

I will still make that same statement.

 

I saw what you mentioned, and agree that Winrock indeed insists in this
toolkit on the importance of knowing local context and habits, making field
testing and that lab testing is not always representative of what happens in
the field.

But still, when it comes to testing, the toolkit gives a strong and
misleading impression that the WBT should be used. That it is a great tool
to test, compare and even select stoves. And that if you can add field
testing, that’s even better.

 

I am gonna quote below extensively the toolkit, and no, I am not
cherry-picking.

 

By the way:

·         The WBT is mentioned 8 times

·         The CCT is mentioned 6 times

·         The KPT is mentioned 4 times

 

No other protocol is mentioned.

 

Page 19 « Best practices 3. Undertake in-country testing of the cookstoves
you plan to promote, to ensure that they perform well with local foods and
fuels. »

This is good. Here field tests are advised. As best practices, not
prerequirements, but still.

 

Page 19 « Standardized laboratory testing protocols and metrics allow for
replicability and comparability among tests and across stoves. As stoves
often perform differently when consumers use them in their own homes, field
testing in consumer homes often provides a more realistic picture of actual
stove performance »

How people will read it: « lab protocols allow for a realistic picture. But
field testing allows for a more realistic picture. »

They will interpret it as: « lab is good, field is better ». As a small
manufacturer with no money, I could rely on the good results and the (« ISO
» as the Aprovecho says) certification given by the lab. I would not
necessarily proceed with a field testing because it is (deemed) long,
painful, costly to execute.

The issue is that the lab testing the toolkit talks about is catastrophic,
it is the WBT.

 

Page 19 « Laboratory testing often represents the best possible performance
under ideal conditions »

No, the performance is not representative, because there are problems with
the metrics and the calculations, it is not repeatable and the statistical
approach is wrong.

« but cookstoves that perform well in the lab (where the standard test is
boiling water) may not always perform well cooking a specific local food, or
with nonstandard fuels. »

Note that here there is a warning. This is a good thing. I wished the
toolkit would have said: « Never develop a stove without testing it in the
field. Field testing is compulsory ».

 

Page 20 « Three common standardized tests are the Water Boiling Test (WBT),
the Controlled Cooking Test (CCT), and the Kitchen Performance Test (KPT).
The laboratory-based WBT is the most standardized of the three, and allows
for comparison between stoves. »

I don’t see how one would not want to use the WBT after reading that. Why
choose another protocol when you can get « the most standardized of the
three », that «allows for comparison between stoves »?

Again, the WBT does not allow for anything but wasting time and money.

 

Page 20 Then the toolkit gives a link to the GACC website, with the WBT at
the top of the page:

http://cleancookstoves.org/technology-and-fuels/testing/protocols.html  

 

Page 20 « Testing protocols are consistently updated and reviewed by the
global community to ensure methods for lab and field testing provide the
best possible depiction of how each stove performs »

Fantastic, so the « global community » made sure for me the last version of
the WBT, the 4.2.3., was really top-notch and fully updated. I can now sleep
on both ears. 

 

Page 20 « For example, testing protocols are being evaluated and updated
through an ongoing international standards process (see below). »

What I got from some people is that the WBT is being dropped by the TC 285
and that they are not working on the issues highlighted by the many studies
about the WBT.

So why are we still talking about the WBT in handbooks and toolkits anyways?

 

Page 21 « That said, the tiers developed were based on the Water Boiling
Test, as the sector has the most data available for that test, with the
future goal being to develop tiers for other testing protocols. »

Decidedly, this WBT seems like a great protocol. Yes, I think I’ll use it to
test my stove, so I can compare it with the competition and place it on
tiers!

 

At the bottom of the page 22, there is even a very nice graph, showing the
tiers, under 3 of them, it is written « Water boiling test ».

 

Page 28 (it is in bold in the text) « All stove and fuel combinations should
be tested by a reputable, nationally or internationally recognized testing
lab, before being deployed in any local context »

Here it is. So, lab testing is a prerequisite. What are the reputable
testing labs? Aprovecho?

 

Page 35 « Conducting all of the research techniques provided in the WASHplus
toolkit can take a fair amount of time and resources. At a minimum, those
interested in promoting cookstoves in a given context should undertake a
simple controlled cooking test with a local cook cooking a typical local
meal to at least rule out inappropriate cookstoves, and a few focus group
discussions to get initial feedback on cookstove features desired by the
consumers. »

Sounds really like a pain. I am not interested to sell my cookstoves in a
given context only anyways, I want to export containers to big clients
around the world! I think I’ll just skip the field test or do only one
simple CCT like recommended.

 

Page 57 « The laboratory-based WBT is the most easily controlled of the
three, allowing for comparison between stoves. The CCT uses a controlled
field environment to assess the performance of stoves relative to the
traditional baseline option» « The KPT produces a more ‘real- world’
estimate as it measures the impact of the introduction of the stove on all
household fuel use1, meaning that it takes into account not just the
intervention stove performance, but also the degree to which consumers
displace their traditional cooking option. This test is carried out in a
larger number of households once the stove has shown positive results using
the WBT and/or the CCT »

Again, reading that, if I could do only one test, I’d do the WBT. It seems
we seek first for positive results with the WBT, then we go to the field.

 

Page 57 « Emissions monitoring is most often done in conjunction with the
other testing protocols described above, and can be done in both lab and
field settings.  The Water Boiling Test, for example, has a section on
emissions testing as part of the formal protocol. »

On emissions: only the WBT is mentioned, no other protocols. Other lab
protocols could have been mentioned.

I don’t know anyone who would not want to know the stove emissions.

So what I would understand: WBT really is the go-to protocol.

 

So after all that, how I would not want to test my stove with the WBT? How
would I not be mislead?

 

I am sorry, but this toolkit is unfortunately putting one more layer of
cement on the status quo, on the WBT. But underneath that, the WBT is
rotten. This toolkit is keeping us in the past.

 

I am waiting for a document that will, at last, make the choice not to talk
about the WBT, and present the new alternative protocols. Do we want
progress or not?

 

The fact that the toolkit advises to do field testing does not make the many
mentions and promotion of the WBT any less problematic. Page 28 : lab
testing (i.e. WBT) is strongly advised, before anything else. The GACC,
D-Lab documents, even the Aprovecho documents admitted doing field tests was
strongly advised.

But they never disavowed the WBT. The Winrock toolkit is not disavowing it
neither, on the contrary!

 

You wrote: ”If you read the whole toolkit and still have that impression,
please give me a call, as that would be a truly problematic take-away”

Sure, that would be great to make a phone call, thanks for offering that.
When there are critics, it is rare people are ready to discuss them in
person, so thanks for that. I’ll write to you to set up a call.

But first I would like the conversation to continue a bit here the List and
would like to have your answer on the above, because I think everyone is
concerned and needs to know.

 

You seem to want improvements and be ready for changes. That is great. I
think the toolkit can easily be modified and reuploaded. It needs to state
that there are serious concerns about the WBT, mention alternative
protocols, and requires absolutely that field testing is performed.

That can be done easily.

 

You wrote: “Even better—come to ETHOS at the end of January and we can
discuss further; you’d be welcome to also present to the group on your ideas
and suggestions for the sector.”

That is great, thanks for the invitation. I would love to go to ETHOS, if my
schedule allows me, I’ve never been. I’d love to join and to discuss with
everyone. The issue is that I live in France, and cannot pay for the trip
from my personal budget.

Is ETHOS able to cover my travel expenses?

 

You wrote: “ Have you considered joining TC 285 to help inform the testing
protocols our sector is developing through the consensus-based ISO process?
“

Yes, very seriously, and I actually tried: I asked the AFNOR if France could
join, but they never replied to my queries. 

I have been asked this question several times, notably by Sally Seiz and
Ranyee Chiang, but haven’t been offered the opportunity. I’d be grateful to
hear from someone or find reports somewhere about the on-going talks.

 

And I have a final question: do you agree Elisa that the WBT is an
unreliable protocol, which is no good to develop, compare, nor select
stoves?

 

Best,


Xavier

 

 

 

 

De : Derby, Elisa [mailto:ederby at winrock.org] 
Envoyé : mardi 21 novembre 2017 04:13
À : Xavier Brandao
Cc : 'Discussion of biomass cooking stoves'
Objet : RE: The WBT allows for comparison between tests and stoves, says
Winrock

 

Dear Xavier,

 

Thanks for your message and for reading the toolkit, I’m glad you were able
to find it. I can’t imagine that anyone who has ever worked with Winrock or
even read just the consumer preferences section of the toolkit could get the
impression that Winrock would recommend making stove selection decisions
based only on *any* laboratory test. 

 

It seems you may have missed the “best practices” intro to that same section
you quote that emphasizes the importance of local testing of stoves to
ensure that they perform well with local foods and fuels.  And where we say
that cookstoves that perform well in the lab won’t necessarily perform well
in homes cooking specific local foods, or with nonstandard fuels. And where
we highlight the importance of updating and reviewing testing protocols to
ensure methods for lab and field testing provide the best possible depiction
of how each stove performs. 

 

If you read the whole toolkit and still have that impression, please give me
a call, as that would be a truly problematic take-away, and I’d like to
better understand your perspective.  My direct line is 1-617-524-0466 or we
can talk on skype.  Even better—come to ETHOS at the end of January and we
can discuss further; you’d be welcome to also present to the group on your
ideas and suggestions for the sector. It’s a great place to have these kinds
of discussions in-person.

 

Have you considered joining TC 285 to help inform the testing protocols our
sector is developing through the consensus-based ISO process? Or the
guidelines for how these voluntary standards get explained and rolled out?
Having more perspectives at the table makes for a better outcome.

 

Kind regards,

Elisa

 

From: Xavier Brandao [mailto:xvr.brandao at gmail.com] 
Sent: Sunday, November 19, 2017 12:38 PM
To: 'Discussion of biomass cooking stoves' <stoves at lists.bioenergylists.org>
Cc: Derby, Elisa <ederby at winrock.org>
Subject: The WBT allows for comparison between tests and stoves, says
Winrock

 

Dear all,

 

Winrock has recently published, on September 2017, on its website, a
toolkit:

 
<https://www.winrock.org/wp-content/uploads/2017/09/Winrock_Cookstove_final_
reduced.pdf>
https://www.winrock.org/wp-content/uploads/2017/09/Winrock_Cookstove_final_r
educed.pdf 

The toolkit « can be used by various stakeholders, but is primarily aimed at
bringing USG staff and external project developers and implementers
up-to-speed on significant developments in the cookstove sector in recent
years. »

 

According to the toolkit, p 20, « Standardized laboratory testing protocols
and metrics allow for replicability and comparability among tests and across
stoves », and « The laboratory-based WBT is the most standardized of the
three, and allows for comparison between stoves. »

 

It is unbelievable that Winrock still publishes this kind of claim, today. I
am baffled. This has been time and time again proven wrong, and time and
time again repeated, on this list and elsewhere that the WBT did not allow
to compare stoves. That it should not be used to select stoves, for
programmatic or project purposes. 

Even the very rare supporters of the WBT (I am not sure who they are
actually) agree about that.

 

I am not even talking about the proofs that the WBT is not even good to
develop stoves.

 

In February 2017, a study by Lombardi and al. summarized the issues: 

« Some of WBT critical issues remain unsolved. In particular, the main
weakness of the WBT concerns its real-life relevance. [
] Criticism about
WBT concerns also the repeatability of the protocol, with a number of
researchers claiming that it would need to be reviewed in terms of accuracy.
[
] As a matter of fact, uncertainties related to temperature reading and
vaporisation in the boiling region lead to high variability between test
replicates.

A lot of debate has been made around formulation of metrics, primarily on
thermal efficiency, which is often interpreted as the most immediate and
distinctive stove performance parameter. Studies from Bailis et al.
highlighted how relying on WBT thermal efficiency outputs, regardless of the
relative importance of high and low power cooking tasks among the target
population, can lead to misleading interpretations. Furthermore, Zhang et
al. and Jetter et al. questioned the scientific meaningfulness of thermal
efficiency at simmering. 

Finally, some unsolved issues concerning statistical significance of data
are worth mentioning. WBT 4.2.3 includes “Statistic Lessons for Performance
Testing”. The appendix specifies that the minimum number of test replicates
for each model of stove should be three, [
] Wang et al. investigated this
topic using a simplified version of the WBT 3.0 and demonstrated that more
than 5 replicates are likely to be required to avoid impractically large 95%
confidence intervals and that even more replicates may be required to
demonstrate a statistically significant difference in performance between
two or more stoves. »

 

Yet, Winrock, like the GACC and D-Lab and their handbook, are again proving
themselves to be the gravediggers of the already struggling project
developers and implementers, rather than the much needed support they should
be, on matters of testing.

 

This is totally irresponsible from Winrock, and very concerning.

 

I put Elisa Derby in copy of this email.

 

Elisa, I am very much looking forward to hear your views on that.


Best,


Xavier

 

 


 
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai
gn=sig-email&utm_content=emailclient> Image removed by sender.

Garanti sans virus.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai
gn=sig-email&utm_content=emailclient> www.avast.com 

 

 

The information in this message, including any attachments, is the property
of Winrock International (discloser), is intended solely for the use of the
individual(s) or entity to which this message is addressed, and may contain
confidential or privileged information. Therefore, the use or disclosure by
a person who is not an intended recipient is prohibited. Use or distribution
by the intended recipient is permitted if distribution is pursuant to an
agreement with discloser authorizing distribution, or distribution is to
those that have been previously approved by the discloser for sharing. If
you are not the intended recipient, or the person responsible for delivering
this message to the intended recipient, please notify the sender and
immediately delete this message.



---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bioenergylists.org/pipermail/stoves_lists.bioenergylists.org/attachments/20171121/ed5b0a00/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 350 bytes
Desc: not available
URL: <http://lists.bioenergylists.org/pipermail/stoves_lists.bioenergylists.org/attachments/20171121/ed5b0a00/attachment.jpg>