Organic Farming

1.- Introduction

2.- What is the Purpose of Sampling in Organic Group Certification?

As explained above, the performance assessment of an ICS takes place at four different levels: at the ICS office, in the buying centres, at the farms and during witness audits with the internal inspectors (Table 1 and Figure 1, also [23]). The results of the audits at the first two levels are mostly qualitative, but a meaningful assessment of the findings from the farm level requires some kind of quantification (Figure 1). Quantification of the results of the witness audits with internal inspectors may not be necessary in small groups with one or few inspectors, but becomes important in large groups with many internal inspectors (Section ). A key underlying question is: What exactly is the goal of sampling a certain number of member farmers?

a. Is the goal to determine the exact percentage (incidence) of each kind of NC? Not really. Let us assume we are dealing with a group, where many farmers use herbicides, which are prohibited in organic farming. Does it matter for the CB, if, say, 14%, 32% or 45% of the farmers use herbicides? The answer is “no”, because in any of these cases, the conclusion would be the same: the ICS is not functional, and certification would have to be suspended, temporarily or terminally. Or let’s imagine a group, where some farmers do not keep records of their daily field activities. Would it make a difference for the CB, if this problem were found among, say, 2%, 4% or 10% of farmers? No, because in any of these cases, the ICS would be requested to propose corrective actions, to ensure that farmers in the future keep their records. And in none of these cases would the group’s certification be at risk.

b. Is the goal to find each and every NC that may exist in the group and has slipped through the ICS? Any type of sampling always involves the risk of a certain number of cases slipping through. This may not be acceptable when it comes, e.g., to high food safety risks, but it would not be appropriate for organic group inspections, because (i) compliance with organic production rules is not a food safety issue, (ii) the idea of “group” certification would become meaningless, since ultimately the sample size would have to be equal to the total number of farmers, and (iii) even with 100% external inspections, not all NCs existing at the time of the inspection will be detected, let alone those NCs, which may not be detectable on the day of the inspection.

c. Is the goal to ensure that non-compliant farmers identified during external inspections are excluded from the group? This is a common misunderstanding (see also Textbox 1), which completely misses the point of group certification. If the CB inspects, e.g., 10 out of 100 farmers, and finds in this sample two farmers using synthetic fertilizers, then we assume that in the entire group there are many more farmers with this problem, and excluding the two members would not solve the problem.

d. Is the goal to decertify groups, when the incidence of severe NCs exceeds a certain threshold? This is how, e.g., the Rainforest Alliance group certification works: “if an irreversible non-compliant practice occurred on more than 5% (of the whole group, after extrapolation (…) and/or on at least 5 of the audited small farms this is considered to be a systemic issue (…) and therefore shall result in non-certification and/or cancellation” [19]. There may be different opinions among CBs and regulatory authorities in this regard, but the authors believe that this approach does not sufficiently consider the efforts made by the ICS. Let’s look again at the example above of a group with widespread herbicide use: When in a group of 100 farmers, the ICS has never detected any case of herbicide use, but then the CB in a sample of 10 farmers detects one case—this situation should be treated differently from the case where the ICS has already excluded 20 out of 100 farmers, but then the CB finds one more case.

e. The real goal of external inspections should be to determine (i) which existing NCs have been properly handled by the ICS and which not; (ii) among the latter, which are “systemic” and which are “isolated” cases; and (iii) which of the systemic cases put at risk the integrity of the products sold on the organic market, and the credibility of the certification system.

3.- Judgement Sampling vs. Statistical Sampling

The U.S. Office of the Comptroller of the Currency [24] distinguishes between “judgement sampling” and “statistical sampling”. The definition of judgement sampling is quoted in Textbox 2.

The current organic group certification procedures are mostly based on judgement sampling. The problem is, however, that the involved CBs do not always have the necessary level of “professional judgement” that would lead to satisfactory results (see [17]). A solution to the problem presented in Textbox 1 can only be found using “statistical sampling”, which allows extrapolation of sample results to the entire group. Statistical sampling methods must select the sample randomly, not risk-based [24], otherwise the results would be biased. If a CB knows, e.g., that a specific problem is more frequent in one village belonging to a producer group, and therefore targets farmers from that village more than the rest of the group, the results from this inspection cannot be extrapolated to the entire group, because the problem would be over-estimated (Figure 3).

4.- What Does “Systemic” Mean? What Does “Integrity” Mean?

For finding a solution to the problem described in Textbox 1, we must first define systemic NCs vs. isolated NCs and in which cases systemic NCs should lead to decertification. In this section, we propose a new procedure for quantifying these terms and for answering these questions, with the help of the variables defined in Table 2. Readers who are not so much interested in the statistical details, can jump directly to Table 3, from there to Figure 5, and then continue with the real life examples in section .

a. We count $M$ and compute ${\pi }_a$ (see Table 2 and Figure 4).

b. As explained in Section (d), one approach for assessing the performance of the ICS would be to simply define a threshold, above which a group should be de-certified. This would mean using an estimate of ${\pi }_e$ (Table 2) for this purpose, i.e. ${\widehat{\pi }}_e=\frac{m}{n}$ . For the reasons explained in Section (d) (we want to value the efforts made by the ICS, which have already detected certain cases), we suggest to use the difference between the incidence of a specific NC identified by the CB in the sample (extrapolated to the entire group), and the incidence identified and corrected by the ICS in the entire group. This better values the efforts made by the ICS (an approach, which may not be shared by all CBs and regulatory authorities):

c. Next, to reflect that an estimate is used, we compute the lower and upper limits of a 95% confidence interval for ${\pi }_e$ (Table 2) using standard procedures as described by Agresti ([25], pp. 15,18-21) and also described in detail below (see Equations to ). The lower and upper limits for ${\pi }_e$ are denoted as ${\widehat{\pi }}_{e(L)}$ and ${\widehat{\pi }}_{e(U)}$ , respectively. Those on $\delta$ are denoted as ${\widehat{\delta }}_L$ and ${\widehat{\delta }}_U$ , respectively.

d. We define a threshold above which the incidence of an NC is considered systemic. Since this threshold should be different, depending on the type of NC, we group the existing NCs in five categories $s$ , from 1 (least severe) to 5 (most severe). Refer to Table 3 for examples. These severity categories are associated with an acceptable threshold ${\delta }_0$ , above which $\delta$ is considered “systemic” (third column in Table 3).

The more severe the category, the lower the acceptance threshold. If neither of the two conditions hold, the sample size was too small to reach a definitive assessment. This is likely to happen only when $\widehat{\delta }$ is close to the threshold ${\delta }_0$ . Note that this step amounts to a significance test at the 5% level to decide if $\delta$ is significantly smaller or larger than the threshold ${\delta }_0$ .

e. As a second condition for considering an NC as “non-systemic”, we introduce the requirement that ${\pi }_e$ must be below 0.3 - regardless of $\widehat{\delta }$ . The rationale is as follows: if the ICS makes serious efforts for handling NCs, but in spite of these efforts the CB still finds many undetected or uncorrected cases, there is a systemic problem.

The assessment is inconclusive otherwise. If this happens for NCs with $s\le 4$ , we suggest the CB decides from case to case, if the NC is considered systemic or not. For NCs with $s$ = 5, the sample should be increased until getting a clear picture.

Finally, we suggest how often a systemic NC can be repeated ( $r$ , see Table 3), before it seriously affects the integrity of the system and should therefore lead to (temporary or final) decertification. We call this threshold “repetition tolerance” $r_0$ . $r_0$ is also related to $s$ (Table 3, column 4). For NCs with $s$ = 5, we have defined $r_0$ = 1, meaning there is no tolerance for systemic NCs of this category.

5.- Two Real Life Examples

For exemplifying the proposed method, we have selected two cases of group certification from the CERES database: a positive case of a group with a functioning ICS and only minor deficiencies, and a negative case, which lost its certification. If the method suggested had been applied, these results would have been confirmed—but based on a more transparent and reliable procedure.

The first case study refers to a cocoa farmers group with 1,079 members. Since this was the first inspection to this group, the risk factor had been calculated as 1.2, based on theoretical assumptions, leading to the sample size: $n=\sqrt{1,079}\times 1.2\approx 40$ .

Three NCs were found, two of which were systemic, but none of these with serious implications for integrity (Table 4).

The rather small NCs could be easily corrected, and the group was certified.

The second case study is for a group of 1,413 coffee farmers, spread over a large area, with highly heterogeneous geographical conditions. A risk factor of 1.4 had been determined, leading to the following sample size: $n=\sqrt{1,413}\times 1.4\approx 54$ .

During the four previous years, only minor NCs had been detected. During inspection planning in 2016, the CB found that the samples in previous years had not been random, because they had only covered a relatively small part of the region. This was corrected by randomly including farmers from all parishes in the new sample. Furthermore, the CB had learned that agrochemical use among coffee smallholders in the entire region had increased substantially. Therefore, coffee leaf samples were taken from 16 out of the externally inspected 54 farmers and tested for pesticide residues.

As a result of this change in inspection procedures, in addition to several other (systemic and non-systemic) NCs, on 10 farms the inspectors found synthetic pesticides and/or fertilizers. In 6 out of 16 leaf samples, residues of synthetic fungicides were found at levels, which could only be explained by application by the organic farmers (Table 5).

None of these NCs had been detected by the ICS, therefore the group’s organic certificate had been withdrawn immediately. If the method proposed here had been used, the result would have been the same. These severe problems in the group, however, were detected not because the sample size was increased as compared to previous years, but because (a) the sample was chosen randomly, and (b) because the inspection procedure was improved by testing leaf samples, which had not been done in previous years.

6.- Sample Size Determination in Scientific Surveys

In a scientific survey with the goal explained above, neither a fixed percentage nor a square root of the total population size would be used as sample size. Instead, a specification would be made regarding the precision with which a population parameter is to be estimated, based on a random sample, and then the necessary sample size would be determined accordingly [26]. Again, readers who are not so much concerned about the mathematical details at this point, can go directly to Figure 9, from there to Textbox 3, then to Figure 11 and then continue with section on stratification.

Assuming we deal with a very large population (as, e.g., in consumer studies or pre-election polls), an asymptotic interval with 95% coverage probability could be employed, based on the estimate ${\widehat{\pi }}_e=m/n$ and these equations for the lower (L) and upper (U) 95% confidence limit for ${\pi }_e$ [25]:

is the half width of the confidence interval. Further, we may compute lower and upper 95% confidence limits for $\delta$ as ${\widehat{\delta }}_L={\widehat{\pi }}_{e(L)}-{\pi }_a$ and ${\widehat{\delta }}_U={\widehat{\pi }}_{e(U)}-{\pi }_a$ , respectively.

It is important to point out that the half-width ( $HW$ ) of the interval is inversely proportional to the square root of the sample size $n$ (see Equations and ). Thus, the larger the sample size, the smaller the $HW$ . This relation can be used to determine sample size, if we can make a specification of the desired $HW$ .

Thus, the sample size to achieve a desired $HW$ can be computed as

Note that population size $N$ does not enter this equation. The sample size remains the same, regardless if the population is, e.g. ${10}^4$ or ${10}^8$ , so long as the population size is large relative to sample size. What matters, are the variables ${\pi }_e$ and $HW$ . If, e.g., our rough guess in such a large group was that there is a proportion of up to ${\pi }_e=0.10$ or ${\pi }_e=0.20$ of undetected non-compliant members remaining for a specific problem (NC), then the sample size plotted against $HW$ would take the form of Figure 6.

To shed further light on the equation for determining the sample size, we may give a second interpretation. If the sample size is chosen to equal $n$ , then the probability is 5% that the estimate of ${\pi }_e$ deviates from the true value by more than $HW$ [26].

So far, we have assumed that the population size is very large. In smaller populations, as in the case of group certification, the exact Clopper-Pearson interval should be used [25], which takes the population size into account. There are no exact equations to determine sample size for this procedure, which yields asymmetrical intervals. As an approximation, we may employ the fact that in finite populations the standard error ( $s.e.$ ) takes the form described by Thompson [26]:

As opposed to Equation 8, population size N does enter here. The factor $\frac{N-n}{N-1}$ relates to the finite population correction. From this, assuming approximate normality of the estimator of ${\pi }_e$ , the sample size may be computed according to [26]:

Note that for large $N$ , this equation approaches the simpler one in equation . Also note that, even though there is a dependence on $N$ , the required sample size is not proportional to $N$ . And only in very small populations is the finite population correction at all noticeable. In Figure 7 we have plotted $n$ against ${\pi }_e$ , for four different $HWs$ , showing that $n$ is inversely related to $HW$ (the higher our expectations on precision, the larger the sample must be), while in relation to ${\pi }_e$ , $n$ is biggest for 0.5, and decreases both towards 0 and towards 1.

In Figure 8 we see the impact of five different $HW$ s and five different ${\pi }_e$ on the required sample size for populations up to 1,000.

Two decisions remain to be made: (a) which is the highest ${\pi }_e$ in the range from 0 to 0.5 that we must consider in an unknown group of farmers, and (b) which $HW$ are we ready to accept? Statistics cannot answer these questions, which require normative or political answers.

Nevertheless, we can try an approximation:

a. In most cases, we do not know the incidence ${\pi }_e$ , therefore it is reasonable to assume a value that is close to the worst-case scenario. The worst case is ${\pi }_e=0.5$ (50% of the farmers have the NC we are dealing with)—for this scenario we need the largest sample for arriving at a correct decision (Figure 7). If we move too far away from this worst case, there is a risk of arriving at wrong conclusions.

b. Furthermore, we consider that the $s.e.$ should not be too far above 0.05, corresponding to a $HW$ of 0.10.

c. Based on these two considerations, let us use ${\pi }_e=0.50$ and $HW=0.10$ as a starting point. The corresponding sample size is represented by the green line for $HW$ 0.10 in Figure 8b. The sample sizes are substantially higher than the square root (e.g. $N=100$ : 48 vs. 10; $N=500$ : 78 vs. 23; $N=2,000$ : 84 vs. 45).

d. Then we looked for real life examples, where the CB CERES had used sample sizes, which were equal, higher, or at least close to these figures. Since CERES has also been using the square root multiplied by a risk factor, there are not many examples meeting these criteria. The examples we have found, are all from very large groups, because, as shown in Figure 8, above a certain population size, the sample sizes resulting from Equation remain in the same range. We used the procedure explained in Section for assessing the systemic condition of the NCs found during inspection of these example groups.

e. As a result, we selected nine groups, all of them from Africa, because this is where the largest producer groups exist [17], with between 3,554 and 78,496 members each. At this point, we do not want to enter into the debate, if such large groups are certifiable or not—the groups were solely selected for the reasons explained in (d). Adding up all the NCs resulting from the nine inspections to these groups, in total CERES had identified 57 NCs, out of which (using the procedure explained in section ), 29 were systemic, 19 were non-systemic, while 9 remained unclear (Method (I) in Figure 9).

f. Then we calculated for each of the nine groups different sample sizes, using Equation , with ${\pi }_e$ ranging from 0.50 to 0.10, and $HW$ from 0.10 to 0.20. The frequency of each NC was calculated proportionally to the sample size: When a specific NC had occurred 22 times in the original sample of 75 farmers, we assumed that in the same group, it would be detected 14 times in a sample of 58 farmers. From these proportional frequencies, we assessed the systemic condition of each NC, using the same procedure explained in Section . The results are shown in Figure 9 (Methods II to XIX).

g. To summarize what is represented in Figure 9:

We therefore suggest to use ${\pi }_e=0.35$ and $HW=0.1$ . This is depicted as Method XI in Figure 9 and yields the following equation:

Another option would be to use a slightly larger $HW$ , e.g. 0.125, being aware that many cases may remain in the “unclear” category, and especially when it comes to NCs of severity class 5, the sample size may have to be increased and the inspection extended, for getting a clearer picture. Figure 11a shows the sample size for Equation ( $HW=0.1$ dotted black line) and $HW=0.125$ , dashed black line), as compared to square root and percentage approaches. In Figure 11b we have plotted $HW$ against sample size, showing that for groups up to approximately 1,000 members, the method established by the European Commission accepts very large and questionable $HW$ s, i.e. standard errors.

7.- Stratification

Even though most group certification rules include provisions for risk based sample selection (see Section ), in real life these rules are mostly not followed, because the risks are generally unknown (with the exception of obvious risks, such as e.g. larger farms posing a higher risk than small ones, and farms on steep slopes being more prone to soil erosion than farms on flat land). Therefore, and because most group certification rules prescribe that members should be located in geographic proximity and have similar farming systems, the situation presented in Figure 3 is a rather exceptional one. If a CB faces such a situation, where a specific risk in one specific sub-group exists, which might slip through when applying random sampling, then the sampling method to the rest of the group is applied as described above, while for the “risky sub-group” one of the two following procedures is used:

a. If the risk situation is very clear, judgement sampling may lead to clear results, without the need for quantification. If, e.g., in a risk-based sample of 10 farmers there are three cases of insecticide use, while in the random sample from the rest of the group there are no similar problems, then the sub-group can be excluded, while the rest of the group remains certified. b. The group can be stratified into two subgroups [26], and the sampling procedures described above are applied independently to each of the two subgroups. We should be aware, however, that a stratification, with a certification decision being taken separately for each sub-group, means that the overall sample size is increased substantially (often doubled) compared to simple random sampling.

8.- Witness Audits: Sample Size and Quantification of Results

Witness audits with internal inspectors are an essential tool for assessing competence and compliance of an ICS [17, 23]. Typically, such audits are combined with farm visits (see also Table 1 and Figure 1). For streamlining the assessment of the internal inspectors’ performance, we suggest to use a scoring tool based on a weighted Likert scale [28]. To oblige users to make a clear decision between positive and negative scores, we recommend the use of a scale with four possible answers [29], as explained in Table 6.

The results are then summarized for all witnessed internal inspectors. If the total score for all witness audits is below a certain threshold (we suggest 70% of the maximum possible score), the ICS is considered to be not functional. If it is between 70 and 100%, corrective actions should be implemented (Table 7).

Small producer groups often have only one or two internal inspectors. In these cases the question of sampling does not come up. For groups with more internal inspectors, based on [27] we propose the following method for determining the sample of internal inspectors to be witnessed, out of a total of N internal inspectors (again: readers not interested in the statistical details, can jump to Figure 12):

While in Equation we deal with a binominal distribution (farmers comply or don’t comply with a specific requirement), here we are assuming an approximate normal distribution with unknown variance. Therefore, as opposed to Equation , the variance ${\sigma }^2$ of scores enters Equation (in place of the variance ${\pi }_e(1-{\pi }_e)$ in Equation ). Figure 12 shows the results of this equation, for $HW=0.1$ and five variances.

From the CERES database, we evaluated the witness audit results from 18 producer groups from eight different countries, with a total of 72 internal inspectors. CERES has been working with a Likert scale with only three possible answers (Yes / Partly / No), but this should not substantially bias the variability of results, as compared to a scale with four answers. The within-group variance ${\sigma }^2$ for the performance of internal inspectors ranged from 0 to 0.34. For estimating the pooled variance ${\sigma }_p^2$ across $k$ groups, we used [27]:

which yielded $\widehat{{\sigma }_p^2}=0.079$ for our case (orange line in Figure 12). To be on the safe side, we suggest to use a variance of 0.15 (black dotted line in Figure 12). Here, it is assumed that the underlying true variance is constant. If the performance of internal inspectors is more variable, larger samples must be used accordingly. According to our data, the variance tends to increase with lower score means. By way of analogy with the binomial distribution, and taking into account the fact that scores are integer values with a fixed lower and upper bound, it may be assumed that the variance drops to zero when the score mean $\mu$ attains the minimum or maximum value and follows a quadratic function of the mean in between. This model may be used to estimate a variance function for ${\sigma }^2$ which could then be used in Equation with a prior estimate of the mean. Our estimate of the variance function based on the evaluation of the scores from 18 producer groups, is

In lack of such an estimate, the worst case scenario may be considered by plugging in the midpoint between the minimum and maximum score. Details are described in the Appendix. For the sake of simplicity we will assume a constant variance here.

The suggested threshold of a minimum score of 70% is a political, normative proposal, and other choices are possible, of course. If the result is close to this threshold (see Table 7), the results should be assessed in combination with the results of the other inspection levels (Table 1, Figure 1). This can be done e.g. using the traffic-light system described in Table 8.

9.- Conclusions

a. Experts agree that many CBs lack the ability of addressing NCs in producer groups at a systemic level. Our procedure for defining the systemic condition of NCs at farm level, depending on the incidence and severity of each NC, offers a tool for solving this problem. The method should be tested in practice, and the variables adjusted, as necessary.

b. Sample selection should be random, not risk oriented. If a combination of random and risk oriented sampling is used, then the group must be stratified, which leads to a larger sample size.

c. Neither a square root nor a 5% sampling rule are in line with the basic principles of sample size determination in scientific surveys. Especially for smaller groups, there is a high risk of cases slipping through with these methods. We suggest to use Equation for sample size determination. If a larger $HW$ (and thus smaller sample) is used, instead of 0.1 as in Equation , the CB must be ready to increase the sample if NCs of severity class 5 come up, for which it is not clear if they are systemic or not.

d. Similar to the quantification of farm inspection results, also results from witness audits with internal inspectors can be quantified and summarised in a meaningful way.

e. The combination of the results from farm inspections, witness audits, ICS office and buying system assessment, allows for differentiated certification decisions.

f. As a general rule, most important for assessing the functioning of an ICS are not large sample sizes, but personal integrity of inspectors, organisational integrity of CBs, inspector competence, inspection procedures (e.g. witness audits with internal inspectors, testing for residues where appropriate), asking the right questions to the right persons, cross-checking the right documents, and conducting inspections at the right time of the year.