[Stoves] Advocacy action: ask the GACC to stop promoting the WBT

Wed Jan 25 15:46:29 CST 2017

Hi Crispin, 

What attributes does the Geres WBT have that produce "better results" than the GACC WBT? Like wise for CSI Indonesia test method. Please be specific and detailed if possible. 

Many thanks, 

Rebecca 

From: "Crispin Pemberton-Pigott" <crispinpigott at outlook.com> 
To: "DISCUSSION OF BIOMASS COOKING STOVES" <stoves at lists.bioenergylists.org> 
Sent: Wednesday, January 25, 2017 1:06:06 AM 
Subject: Re: [Stoves] Advocacy action: ask the GACC to stop promoting the WBT 

Dear Ron again 

No one is saying to start over again. ‎India and China never accepted the WBT at all. There are many tests. Colorado State didn't accept it and Morgan produced his own WBT test. Penn Talyor also thought it was pretty useless. 

GERES also didn't think it was technically up to scratch and produced their own version that gave 'better results'. 

No one cares about the bad math of the char deduction formula in the WBT. That is over - it lies about fuel consumption and that is the end of it. 

Conceptual issues arise right from the start. The WBT is constructed on the basis that you can learn something fundamental, inherent, about a stove using a,'standard test' with a 'standard fuel' that once demonstrated, will carry over into all other situations. This is a logical error, demonstrating fundamental misunderstandings about how performance is generated in the first place. 

The performance, particularly the emissions performance of a stove is strongly affected by the totality of the context. That is what sets the WBT which is 'fixed variables' apart from the CSI test, which is strongly contextual: 'context of use variables'. 

The EPA provides good examples ‎of contextual rating methods. Their space heating stove tests, though fixed, are carefully reflective of the expected scenario of use. Also their vehicle testing duty cycles which number more than 100 according to Morgan Defoorte. Why then would anyone expect a single fuel, cycle, pot and water load to reveal anything 'generic' about a set of stoves? It won't. 

The standard defense of the WBT, from Dean and Michael Johnson (I pick them because they both said in my presence) is that, "Stoves which perform well on the WBT generally perform well in the field."‎ also they add that a stove should be tested in a lab before testing in the field because, "A stove that doesn't perform well in the WBT lab test will not perform well in the field." 

So there is a clear divergence: using a defective test as a filter that runs the stove in a decontextualised manner before going to the fuels to find out 'how it really performs.' 

This is conceptual bunk. ‎It fails to fly before the engine is started. The initial claim for choosing 'water boiling' was to 'simulate cooking'. The WBT doesn't simulate anyone cooking except by chance. 

Can you imagine assessing the load capacity of a bridge using a method that matched reality 'by chance'? Would we use a method of testing the efficacy of a new drug that matched the dose people would take only 'by chance' instead of how it really would be applied? 

So from the start the WBT fails the first conceptual test. It makes no attempt at all to replicate conditions where the stove will be used, on the basis that this is not necessary because we can learn something 'inherent' about its performance with a 'standard test'. 

The second conceptual failure is that a test method designed for homogeneous combustion of liquid and gas fuels can be applied to a stove burning solid fuels inhomogeneously. There are multiple failures under that column. 

With regard to reproducibility I will cite ‎one example. The CSI test has a text sequence protocol that requires the developer of a new sequence to reproduce the emissions and energy metrics within 10% of the sum of the individual cooking sequences that make up the combined test sequence. This is a validation that the combined test sequences is a true reflection of the average performance of a set of meals expected to be prepared with the stove in that demographic. Once the test sequence is validated it is given a number and anyone can reproduce it at another lab. 

The '10%' is the current ‎requirement of the CSI Pilot. Dean's comment on this target was that he, 'didn't think it was possible to achieve that.' Not even possible. Yet it is the minimum standard for the CSI Indonesia pilot. Why the skepticism? Because Dean has always been using the WBT and it is not possible, apparently, to get numbers like that. Why then do you think that Camilla has lent some credence to the CSI test method? 

The 'why' for the failure of the WBT to achieve that involves a long list of conceptual, mathematical and operational missteps. Consider: the three low power metrics of the WBT do not even have a physical basis ! ‎ What is the point of 'getting a good number' on a metric that has no physical meaning? How can you get good agreement between tests if the meaningful number has been divided by a random number of litres? Before defending a test method you should review how it works and 'test the test'. 

As for Fabio's analysis: Fabio had no expectation of reproducibility. He just analysed the method. He correctly calculated what one can claim for three replications. He reported the results. Good scientist, he gets a cookie. It has nothing to do with his expectations. The conclusion is that the the statistical approach taken in the 3 replication and 10 replication ‎approaches are defective conceptually, and then the test method taken as a whole (without looking into its mechanisms) can only properly claim that 15% of stoves claimed by the WBT to be 'improved' (at all) actually are at a 90% confidence interval. That says nothing about how improved they are, just answering the question they they were either better or not. No mention of‎ putting on tiers only 10 or 20 per cent apart. Just 'better or not'. 

That is a catastrophic failure to uphold a claim. Worse, that is without considering that the task set for the stove was removed from reality. How good is the assessment if one considers that the context of use is also different? 

No wonder Cecil says it stinks. 
Crispin 

Hi again to list and all ccs 

This is #6 (forgot to put a #5 on last to Cecil). See below (reminding I am trying to avoid repeat comments) 

On Jan 24, 2017, at 9:26 AM, Crispin Pemberton-Pigott < crispinpigott at outlook.com > wrote: 

Dear All 

If Camilla can select from a set of WBT results the one that best suits the program qualification‎(s) sought, and the product is in fact the same, what is the point of setting performance criteria? 

[RWL: The point to me is obvious - to assure improvement - both for individual stove producers and overall. I am sure that starting over again will greatly slow down a process (with GACC finding funding) that is progressing quite a bit more rapidly than when this list started (I was the first coordinator) in 1995. 

BQ_BEGIN

There is literally no point. Anyone can send their stoves to 5 or 10 testing stations, get 5 or 10 fundamentally different results and then send in the one that suits the occasion. 

BQ_END
[RWL: I have now heard that one stove company did this (not for 10). I don’t know costs of testing, but there is a limit on what one can achieve this way (especially when customers find something different from what is being advertised). I doubt that Ms. Fulland was saying that she was doing this to sell more stoves. 

BQ_BEGIN

Two points are supported by these results: 

As Philip says, the WBT results are irreproducible, with very sophisticated confirmation by Fabio that we should expect no more. 

BQ_END
[RWL: I don’t hear that all " WBT results are irreproducible ”. In fact the two cites I put in yesterday from Jim Jetter looked very good on this. I am adding as a cc my main resource on testing, Dean Still (Aprovecho, the only place I have participated in this testing) to comment on his experience with reproducibility. 

I have skimmed Fabio’s material (I have used Chebycheff functions in the past, but have not yet understood his material), and think he is expecting way too much repeatability for an inherently very complicated process (biomass materials, but especially the way and timing that fuel if placed in the stove). As I said earlier, there is much to be gained to be gained from understanding differences. 

If anyone has data they can send, I would be glad to try to understand any unexplainable differences. Dean Still seems to get a great deal out of these tests, in his continually improving stoves. 

BQ_BEGIN

The fundamental conceptualization of the test and what it purports to report is somehow defective. It has been analyzed technically more than conceptually, but I can add that the conceptual problems are large and require a ground-up-reanalysis of what we are trying to ‎assess and how. 

BQ_END

[RWL: I get nothing out of this last, except likely a repeat that the “denominator equation” is faulty (which I and all of the group I talk to disagree on). Again, we should hear more at ETHOS on this topic; Crispin is far from being the only authority on this topic. 

BQ_BEGIN

Xavier wrote: 

“I am appalled that we are in 2017, and again another study shows how much the Water Boiling Test is flawed, and is leading us in the wrong direction(s). ‎" 

BQ_END

[RWL: Xavier seems unconcerned about the main issue (the “denominator equation”) separating Crispin and myself - and his reason for unhappiness there is still a mystery. I still do not understand any detail of Xavier’s concerns - and have earlier responded on each of about 7 cites he sent me. 

BQ_BEGIN

Camilla's point is that you can take it in any direction you want. That is from the horse's mouth, so to speak. The meaning is the race house actually running says that is now to win. 

BQ_END
[RWL: I have responded earlier today to Ms. Fulland - and expressed hope for details (the devil being in the details). I have no sense of how good or bad the results actually were - much less why. She may have been shopping for best results, which is Ok (or were these part of [badly needed] GACC-funded round-robin testing?) But shopping can happen with any test conducted in the free-world. I hope we are not hoping for a limitation on the number of tests? How can killing 4.2.3 and starting over again possibly help? Especially when all the other tests are basically very (very) similar to WBT 4.2.3? 

Ron 
BQ_BEGIN

Regards 
Crispin 

PS I have pasted Cecil's relevant comment below this for convenience. 

++++++ 
‎Dear fellow cookstove enthusiasts and frustrati, 

As an over the hill back slide applied social anthropologist‎ I whole heartedly agree it is time for the stovers of the world to unite and give the WBT a belated burial. 

Why? Because the WBT is beginning to stink. The accumulating human, economic, environmental and societal costs to prolong the life of the WBT can not be justified. I vote we pull the plug on the life support sophistry and permit the various versions of a fundamentally defective stove performance test for efficiency to exit the public arena once and for all. 

Why? The costs and complexities of prolonging it's existence now far outweigh the benefits of WBTesting to identify candidate stoves that qualify for inclusion in or exclusion from the category of stoves good enough for gov't subsidies, big orders from UNHCR, and investor financing, It is time for stovers of the world to unite and stop our fiddling while the stoves of Rome continue to burn and emit dangerous smoke! 

‎Fortunately we now have alternative ways to assess the performance of traditional and innovative candidate stoves that are culturally, environmentally, and economically contextualized. Surely it is time for the "united stover's of the world" to return to a more eclectic and open minded phase of small stove R&D where we allow ourselves the space we need to continue experimenting with stoves as combustion technologies, as heating devices, and as tools for cooking culturally variable foods, and carrying out many different tasks such as small scale agri-processing and commercial food preparation. 

As an anthropologist I tend to focus on the stove operators role and skills, the fuels available and used, the economics of production, etc and how these components combine holistically into a dynamic system that also includes the fabricators and marketing agents. My preference is to step back and to allow traditional and innovative stove technologies, elements, behaviors, and cultures to creatively interact and evolve toward new optima with the smallest possible interventions and costs. 

If we "stovers of the world" actually unite, listen to each other and get better at learning from our cantankerously different approaches it should be possible for us to gradually grow an inclusive eclectic approach to stove assessment that will allow us to select those tests and observations which document and compare the different stove/fuel/culture realities. Some time back Crispin referred to a tool kit of different metrics. 

So my vote is to retire the WBT as an adequate indicator of a stove's efficency. If we had a plebiscite on this list, what are our choices? What do we replace the WBT with after it is dead and buried? RIP! 

Lastly I believe we are collectively learning about the negative consequences ‎of allowing wanna be global authorities to PREMATURELY impose universal metrics to rank the performance of stoves that are ripped out of their various meaning giving contexts. 

So let's do the Roman thing and hold a plebiscite on "to WBT or not to WBT"....that is the question. 

Cecil 

From: Xavier Brandao 
Sent: Wednesday, January 18, 2017 5:08 PM‎ 

I am appalled that we are in 2017, and again another study shows how much the Water Boiling Test is flawed, and is leading us in the wrong direction(s). 

Dear all, 
It is not very often I do participate in the discussion on this list. I do, however, follow it closely. I am writing now to give my support to Xavier and his call to stop the use of the WBT. Being a user of test centres and highly dependent on having valid results for our stoves, I must say I’m deeply concerned about the use of the WBT. Prime has so far tested the same stove in five various test centres recommended by GACC and the results are not just different each time, in some cases they are even the opposite of each other. So how can we as a producer say anything about the efficiency or emissions of our stoves to our donors or potential customers? Basically I can ask, what are you after? A clean stove or an efficient stove? And then I select the report that is suitable for the purpose. To me, the WBT and ranking of stoves using this is a joke and a danger to the credibility of the entire stove industry. Moreover, we are sick of having spent thousands of dollars on test results that mean nothing! 

The only protocol we do support is the CSI Indonesia protocol as this seems to produce relatively “true” results backed by field results. 

I raised our concerns about the WBT with the GACC and the Gold Standard in 2015 but no progress for the users of such tests seems to have happened since then. This is highly worrying as millions of dollars are paid out based on manipulated test results “verified and approved” by the GACC as they are the main promoters of this testing method giving it credibility towards other “non-stove” donors. 

Camilla 

Camilla Fulland 
CEO | Prime Cookstoves 

camilla.fulland at primestoves.com 
Norway: +47 48 12 05 37 
Indonesia: +628 2147 600 141 
Skype: camilla.fulland 
www.primestoves.com 

From: Stoves [ mailto:stoves-bounces at lists.bioenergylists.org ] On Behalf Of Xavier Brandao 
Sent: onsdag 18. januar 2017 23.08 
To: Discussion of biomass cooking stoves < stoves at lists.bioenergylists.org > 
Subject: [Stoves] Advocacy action: ask the GACC to stop promoting the WBT 

Dear friends, 

There have been a lot of fierce discussions on the stove discussion list lately. I think most of us are unhappy with a certain number of sector-wide issues. 

There is one thing in particular that appalled me. I read the study by Riva et al.: 
"Fuzzy interval propagation of uncertainties in experimental analysis for improved and traditional three–stone fire cookstoves" 
https://www.researchgate.net/publication/308900170_2016_Riva_Fuzzy_interval_propagation_of_uncertainties_in_experimental_analysis_for_improved_and_traditional_three_-_Stone_fire_cookstoves 

I am appalled that we are in 2017, and again another study shows how much the Water Boiling Test is flawed, and is leading us in the wrong direction(s). 
I am appalled when I think that Crispin has been telling us about it all these years (10 years now?), and his warnings have too often fallen on deaf ears. 

Radha Muthiah, the CEO of the GACC, said in her introduction of the GACC 2016 annual report: "The Alliance worked with UNHCR to develop procurement guidelines that include evaluation of stove and fuel performance, affordability, usability, safety, and durability. This is an example of how organizations can adapt international standards to support their priorities and decision-making." 
Crispin mentioned that the procurement guidelines were based upon the WBT. 
We are in 2017, and important decisions for thousand of stoves to refugees are still based on the WBT, a test which does not allow us to really know how a stove will perform. This is scary. 

At the end of last year, Prakti was commissionned by the World Bank to review the testing ecosystem in India. I had the chance to talk with many stakeholders, especially from the U.S., and to have enlightening discussions. Things have barely started to move it seems. A few people have started to see the shortcomings of the WBT. But this is all too slow. 

What we fail to measure are the damages. 
How many tests results are probably meaningless, and so useless? How ignorant were we about the real performance of so many stoves? Many of us could see how different performance in the field was, compared to performance in the lab. 
How many not-at-all-improved stoves were built, promoted, and disseminated? 
How many stove projects or companies failed because of a test that was problematic in the first place? 
How many efforts, how much money was lost because of poor testing? 

Crispin is right, I could not agree more with him on that: before being able to do the R&D that we need so much, we need valid metrics. There is no way around that. This should be our first priority. 

We need to act now; but, to act and get results, we must focus on what is important, on what is central, on what is foundational. Now, we need good testing; it is as simple as that. 

The goal is not to point fingers, nor put the blame on anyone. What is done is done, the only thing that matters now is: what do we do from here? We need to work together constructively. 
We cannot say we didn't know. 

I discussed with Crispin, Adam, and a few others. I am starting an advocacy action with this email actually. 
It is very simple. 
I think we need to address the GACC. Despite all the criticism, the GACC did a lot of work for the stove sector, and, like it or not, is still regarded as our main representative to policy makers and those with the capital to fund meaningful change at scale. We need to be working with them. 

We would ask the GACC: 
· to publicly acknowledge that the WBT has major shortcomings 
· to remove the protocol from its website, so cookstoves sector stakeholders are not using it 
· to actively promote development and use of other, valid, protocols 

There were a lot of heated discussions around the WBT in the past. Because of that, other protocols were included in the TC 285 process. This is not good enough. 
We don't want invalid protocols to be promoted alongside valid ones. I believe the WBT is still the most used protocol, and anyone can download it on the web, and (mis)use it. That does a lot of harm. 
Here it is, on top of the page: 
http://cleancookstoves.org/technology-and-fuels/testing/protocols.html 

Good and valid protocols do exist. They work very well, give more and better data than the WBT. We do not have, at all, to use the WBT. Why would we keep using it at all? 

Recent articles have shown that we are losing credibility as a sector. We cannot regain it if we keep using invalid testing. I think this sector is comprised of good people who want to do good science and build healthy foundations. 

If you think you are one of these people, please contact me at my email address to show your support for this initiative: 
xvr.brandao at gmail.com 
If you have an opinion on the matter, let's please discuss it here on the stovelist. If you dont agree, if you think there are good scientific grounds for the WBT as a valid protocol, or think for this or another reason that it should still be used, please let's discuss it, in scientific terms, on this list. 

The outcomes of this discussion will be brought to the GACC, and we will continue discussing it with them. 
We have all dedicated a significant portion of our lives to this work; let us not let bad science, or past wrong decisions undermine all that we have accomplished, or all that we might. Can we stand together, whatever our differences of opinion on the best stoves, or fuels, or development practices and agree that in testing our assumptions, pseudo-science and demonstrably flawed methods have no place in this sector? 

Please join, please tell people around you about this initiative. 

2017 is just starting, let's start it on the right foot. 

Thanks, 

Xavier 

Dear all, 
It is not very often I do participate in the discussion on this list. I do, however, follow it closely. I am writing now to give my support to Xavier and his call to stop the use of the WBT. Being a user of test centres and highly dependent on having valid results for our stoves, I must say I’m deeply concerned about the use of the WBT. Prime has so far tested the same stove in five various test centres recommended by GACC and the results are not just different each time, in some cases they are even the opposite of each other. So how can we as a producer say anything about the efficiency or emissions of our stoves to our donors or potential customers? Basically I can ask, what are you after? A clean stove or an efficient stove? And then I select the report that is suitable for the purpose. To me, the WBT and ranking of stoves using this is a joke and a danger to the credibility of the entire stove industry. Moreover, we are sick of having spent thousands of dollars on test results that mean nothing! 

The only protocol we do support is the CSI Indonesia protocol as this seems to produce relatively “true” results backed by field results. 

I raised our concerns about the WBT with the GACC and the Gold Standard in 2015 but no progress for the users of such tests seems to have happened since then. This is highly worrying as millions of dollars are paid out based on manipulated test results “verified and approved” by the GACC as they are the main promoters of this testing method giving it credibility towards other “non-stove” donors. 

Camilla 

Camilla Fulland 
CEO | Prime Cookstoves 

camilla.fulland at primestoves.com 
Norway: +47 48 12 05 37 
Indonesia: +628 2147 600 141 
Skype: camilla.fulland 
www.primestoves.com 

From: Stoves [ mailto:stoves-bounces at lists.bioenergylists.org ] On Behalf Of Xavier Brandao 
Sent: onsdag 18. januar 2017 23.08 
To: Discussion of biomass cooking stoves < stoves at lists.bioenergylists.org > 
Subject: [Stoves] Advocacy action: ask the GACC to stop promoting the WBT 

Dear friends, 