<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0mm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0mm;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:Consolas;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0mm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoNoSpacing, li.MsoNoSpacing, div.MsoNoSpacing
{mso-style-priority:1;
margin-top:5.0pt;
margin-right:0mm;
margin-bottom:5.0pt;
margin-left:0mm;
mso-add-space:auto;
font-size:11.0pt;
font-family:"Arial","sans-serif";
color:#1F497D;}
p.MsoNoSpacingCxSpFirst, li.MsoNoSpacingCxSpFirst, div.MsoNoSpacingCxSpFirst
{mso-style-priority:1;
mso-style-type:export-only;
margin-top:5.0pt;
margin-right:0mm;
margin-bottom:0mm;
margin-left:0mm;
margin-bottom:.0001pt;
mso-add-space:auto;
font-size:11.0pt;
font-family:"Arial","sans-serif";
color:#1F497D;}
p.MsoNoSpacingCxSpMiddle, li.MsoNoSpacingCxSpMiddle, div.MsoNoSpacingCxSpMiddle
{mso-style-priority:1;
mso-style-type:export-only;
margin:0mm;
margin-bottom:.0001pt;
mso-add-space:auto;
font-size:11.0pt;
font-family:"Arial","sans-serif";
color:#1F497D;}
p.MsoNoSpacingCxSpLast, li.MsoNoSpacingCxSpLast, div.MsoNoSpacingCxSpLast
{mso-style-priority:1;
mso-style-type:export-only;
margin-top:0mm;
margin-right:0mm;
margin-bottom:5.0pt;
margin-left:0mm;
mso-add-space:auto;
font-size:11.0pt;
font-family:"Arial","sans-serif";
color:#1F497D;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-CA link=blue vlink=purple><div class=WordSection1><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Dear Tom<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>“</span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>If the variation from testing is greater than the variation required to elevate a stove between tiers then why bother with tiers to categorize stoves? By using averages have we just removed the variability? If we don’t keep the outliers in the data set how can we analyze them to determine sources of variation or error? I haven’t seen evidence that anyone has really questioned the data.”<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Well Tom, you have nailed that one as well. The WBT 4.1.2 asks the tester to repeat the test three times and to fill in the form. The individual results have a coefficient of variation (CoV) on the Results page. Some are using this or similar tests who say that if the CoV is outside a certain range (for example 25%) the tests should be discarded and it should be done over.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Now suppose the CoV 10%. What does CoV mean? It means the Standard Deviation divided by the Mean (average) of the results.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Using the most recent version of the WBT 4.1.2 file dated June 2012 from<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><a href="http://www.pciaonline.org/files/WBT_data-calculation-sheet_4.1.2_updated_19-Jun-2012.xls">http://www.pciaonline.org/files/WBT_data-calculation-sheet_4.1.2_updated_19-Jun-2012.xls</a><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I entered the values 10, 11 and 12 g per minute into cell AP31 for tests 1, 2, and 3 respectively. That is where the calculated burn rate of fuel for the low power portion of the test is found, normally called the simmering phase. Just enter it manually.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The Result page pulls these results into line 30 and it will bring show three numbers in cells G30, H30 and i30. The numbers will be 10, 11 and 11 respectively. It will not show 10, 11 and 12 because Cells H30 and i30 both look at Test 2. So correct the formula in i30 to read <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>='Test-3'!$AP31 <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The CoV is 9%. This is nothing <i>close</i> to the precision so don’t get excited just yet.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>A difference of 1 g per minute in the burn rate in 10 gives a Standard Deviation of 1. For a TLUD gasifier stove with a very constant burn rate like the Quad 2 (for which the test is publicly available) the result is St Dev 0.1 and a CoV of 1.2%. Highly replicable.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>It has been said that a Cov of 25% is acceptable. In order to get that result change the test results for AP31 cells to 7.5, 10 and 12.5. The CoV is 25% because the St Dev is 2.5 and the Mean is 10. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Consider that the <i>test method</i> is the same and the fuel and everything is the same for all three tests, but the result varies by 25% on this one section. This indicates a high variability, yes? Is this result acceptable? Can you confidently place a stove on tiers separate by 20% if one metric, a portion of one final metric, the fuel consumption, varies by 25%? The mathematical answer is ‘no’. The total variation has to be less than 1/3 of the gap. And this is just for one variation.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The graphic error bar for a single test is not a dot. There should be an error bar and it relates to the precision of the test metric, with all the errors properly rated and summed using a standard method which I will not explain here. Suffice to say that if there is an error bar of 10% above and below the values for each test, that is the result that should be plotted for a single test, and all subsequent tests. They should not be averaged first into groups of three. Averaging does not reduce the error for each or any test. It does not increase the accuracy nor the precision.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>What is happening is there are no error bars being shown on stove test results. The average result is usually plotted and the upper and lower test numbers are shown with a vertical line as if they are error bars but they are actually results, not indications of error or precision or imprecision. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The average of some results is not the result. The results are the results. The fact that the <i>protocol</i> calls for three results to be averaged does not make the test method a valid way to provide a rating. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>What is happening now is that when the stove is re-tested three new times, a second average of results is calculated. In the Berkeley paper the averages were them compared with each other. For example the average of two sets of tests were found to be 6% different. So what? That is not a useful number. It does not indicate anything about the accuracy or precision of the test method.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The tests stand individually because there is a test method and following it produced the results with all the procedural and systematic errors that occurred that day. If repeating the test using the same method does not produce results that are within 1/3 of the Tier gap, then the test method (or the person testing) does not show the necessary skill to place stoves on a tier with confidence. Averaging groups of tests and then comparing the group results is not a valid way if determining if the 1/3 tier gap has been met.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>When you are performing a health survey in a community you normally have to submit your method to a review board who will check for errors like that. Then you correct it, then you are allowed to proceed. It should not be different for the methods used for testing stoves. In fact the IWA calls for it to be done properly.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Regards<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Crispin<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p></div></body></html>