Edited by:
Reviewed by:
*Correspondence:
This article was submitted to Quantitative Psychology and Measurement, a section of the journal Frontiers in Psychology
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Elicitation is a commonly used tool to extract viable information from experts. The information that is held by the expert is extracted and a probabilistic representation of this knowledge is constructed. A promising avenue in psychological research is to incorporated experts’ prior knowledge in the statistical analysis. Systematic reviews on elicitation literature however suggest that it might be inappropriate to directly obtain distributional representations from experts. The literature qualifies experts’ performance on estimating elements of a distribution as unsatisfactory, thus reliably specifying the essential elements of the parameters of interest in one elicitation step seems implausible. Providing feedback within the elicitation process can enhance the quality of the elicitation and interactive software can be used to facilitate the feedback. Therefore, we propose to decompose the elicitation procedure into smaller steps with adjustable outcomes. We represent the tacit knowledge of experts as a location parameter and their uncertainty concerning this knowledge by a scale and shape parameter. Using a feedback procedure, experts can accept the representation of their beliefs or adjust their input. We propose a Five-Step Method which consists of (1) Eliciting the location parameter using the trial roulette method. (2) Provide feedback on the location parameter and ask for confirmation or adjustment. (3) Elicit the scale and shape parameter. (4) Provide feedback on the scale and shape parameter and ask for confirmation or adjustment. (5) Use the elicited and calibrated probability distribution in a statistical analysis and update it with data or to compute a prior-data conflict within a Bayesian framework. User feasibility and internal validity for the Five-Step Method are investigated using three elicitation studies.
“
According to
There are many elicitation procedures available, overviews can be found in for instance
Software that can be used in the elicitation with the trial roulette method is available in the MATCH Uncertainty Elicitation Tool (
One of the reasons the trial roulette method is popular is that the procedure provides immediate visual feedback to experts. Feedback is important in elicitation procedures to reduce bias and improve the quality of the elicitation (
Feedback is believed to improve the quality of the elicitation procedure by making experts; reflect and maintain self-consistency (
Therefore, to assist experts in the process of creating a representation of their beliefs in a statistical distribution we propose to decompose the elicitation task in smaller steps to encourage and assist in structured reasoning. Decomposing a problem into more tractable and familiar components is suggested by for instance
We propose the Five-Step Method which consists out of the following steps:
Elicit the location parameter of the SN using the trial roulette method.
Use software to provide instant feedback on the interpretation of the expert’s beliefs by the researcher so the expert can accept this representation or adjust their input.
Elicit the (un)certainty of the expert by determining the scale and shape parameters of the SN using expert’s statements on the lower and upper bounds for a plausible range of the parameter values.
Use software to provide instant feedback on the interpretation of the expert’s (un)certainty about the location parameter by the researcher so expert can accept this representation or adjust their input.
Use the elicited and calibrated probability distribution in a Bayesian analysis to update it with data or to compute a prior-data conflict.
The remainder of the paper is ordered as follows. We first provide details on the Five-Step Method. Thereafter we present a user feasibility study in which we elicited beliefs regarding a trivial sports related question from respondents to investigate visual and procedural preferences of users for the digitized version of the trial roulette method. A second study was carried out by asking experts working at a staffing company about certain key performance indicators which we used to validate the internal validity of steps 1 and 2 of the elicitation procedure. A final study was done with regional directors working at a large financial institution. They provided actual forecasts concerning average turnover per professional in the first quarter of the year 2016 with the Five-Step Method. The participating companies already make predictions concerning the parameters we elicit, yet they do this in the form of point estimates. The experts are thus already used to thinking about these data and predicting these data which makes them highly suitable to include as experts in an elicitation exercise. Yet, it is an extension for them to actively specify and separate knowledge and uncertainty. Because the companies also provided us with data on the predicted parameters we were able to compare the forecasts of the experts with data and thereby get an indication of the internal validity of the elicitation procedure. The proposition to split the elicitation process results in a procedure differing from the existing elicitation procedures as, for example, proposed by
In this section we describe the technical details of the Five-Step Method which has been programmed in R (
The first step of the Five-Step Method consists of a digitized version of the trial roulette, which can be seen in
Shiny application for steps 1 and 2. On the left the input fields can be found for the reasonable lower and upper bound as minimum and maximum values. The input grid in which “chips” can be placed is found on the lower right with the leftmost dot being the minimum value and the right most dot being the maximum value. Further “chips” are placed by clicking the mouse drawing a maximum of 11 pixels left and right. On the top right feedback is provided, presenting the fitted distribution based on the input.
The vector of values that is elicited in step 1 are used to fit a SN distribution. The SN distribution is defined in this paper as a normal distribution with the additional shape parameter γ. The shape parameter is based upon a general method for the transformation of symmetric distributions into skewed distributions as described in
The effect of the shape parameter on the allocation of mass can be seen in
Example of the influence of shape parameter γ on the allocation of mass for a normal distribution with a variance of 1.
To fit the SN distribution we make use of the snormFit function from the fGarch package (
The SN distribution that is fitted based upon the expert’s input is provided as visual feedback to the expert, see
Step 3 of the Five-Step Method is used to derive the distributional representation of the expert’s prior beliefs concerning the parameter of interest and can be seen in
Shiny application for steps 3 and 4. On the left the input fields require entering the estimate for μ_{0} and the reasonable lower and upper bound for the estimate. On the right the distribution that is fitted based on the input can be found.
Based on the input provided in step 3 we will obtain estimates for the scale parameter
Use the elicited distribution that represents the expert’s beliefs.
In this section we describe the three studies we conducted. During the user feasibility study R version 3.1.2 was used and R version 3.2.3 was used during the elicitations done with the staffing company and the large financial institution. We conducted the elicitations in a semi-structured face-to-face setting so that the researcher could provide interpretations accompanying the visual feedback. An advantage of a face-to-face setting is that it allows clarification of procedural and elicitation related questions thereby improving the validity of the responses (
With the user feasibility study we evaluated the usability of the first two steps of the Five-Step Method. Procedural and visual preferences were investigated. Four variations of the shiny application were tested. The respondents (
In the first procedural option, we used the procedure of the trial roulette where 20 digital “chips,” starting with the expected minimal and maximum value, each representing five percent of a distribution, were to be placed in a grid following the procedure described by
The respondents evaluated the two variations they were appointed to with a questionnaire asking if the fitted distribution was a good reflection of their beliefs and what visual and procedural preferences were. Additional questions were based on the taxonomy of
All respondents indicated that their beliefs where accurately represented. Five of the seven respondents allocated to both procedural variants preferred the second variation. Four of the six respondents allocated to both visual variants preferred the large grid, one abstained from answering. Three out of the nine respondents indicated for at least one of the variations that they did not understand the meaning of the “chips.” In the first procedural option the “chips” each represented 5% of the data whilst in the second procedural option the meaning depended on the amount of chips that were placed. They allocated mass for the distribution that was fitted. The meaning of the chips was not completely understood by one person who used the first procedure and by two persons who used the second procedure. All three of them used a small grid variation. The three respondents all indicated that they knew what the distribution representing their opinion meant in the end and agreed that this accurately described their view. Based on the results we decided to continue working with the second procedural variation, requiring the minimal placement of seven “chips” without further restriction on the number of “chips,” and a large grid.
The goal of the second study was to test the internal validity of elicitations obtained with the first two steps of the Five-Step Method. We found a staffing company willing to participate with experts (
The experts were asked to predict the distribution of the data. In some sectors staffing companies staff a lot of individuals at low margins and thus generate a large turnover. In different sectors they staff few individuals at high margins thereby obtaining the same profit at lower turnover rates. These are all relevant considerations and the experts should know which is the case for their company. The company provided us with actual budgets they made which were indications of carefully constructed predictions. By comparing the predictions of the experts to the budget we could gain an indication of the internal validity of predictions made with the first two steps of the Five-Step Method. If the elicitation results match the budget this indicates that the procedure is able to represent the underlying construct of carefully constructed predictions.
The results can be found in
Results for elicitation with the staffing company. Experts’ predictions plotted with actual budget for contract hours
Results for elicitation with the staffing company.
Contract hours |
Hourly cost buying |
Hourly cost selling |
Turnover |
Hourly sales margin |
|||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
μ | σ | γ | μ | σ | γ | μ | σ | γ | μ | σ | γ | μ | σ | γ | |
Expert 1 | 3.88 | 0.83 | 4.01^{∗}10^{-6} | 3.21 | 0.99 | 1.10 | 3.99 | 1.14 | 1.73 | 3.39 | 0.98 | 1.48 | 2.18 | 0.94 | 1.68 |
Expert 2 | 3.56 | 0.61 | 4.48^{∗}10^{-8} | 2.74 | 0.69 | 1.57 | 3.86 | 1.04 | 2.14 | 3.21 | 0.81 | 1.53 | 2.46 | 0.97 | 1.31 |
Expert 3 | 3.85 | 0.70 | 0.51 | 2.91 | 0.80 | 1.78 | 3.72 | 0.81 | 1.41 | 2.71 | 0.72 | 0.93 | 2.25 | 0.76 | 1.69 |
Expert 4 | 3.34 | 0.89 | 0.74 | 3.45 | 0.97 | 7.20 | 4.59 | 1.43 | 12.80 | 3.16 | 1.17 | 0.98 | 2.18 | 0.84 | 1.97 |
Budget | 3.37 | 0.91 | 5.52^{∗}10^{-4} | 3.09 | 1.05 | 566.00 | 3.87 | 0.99 | 1.29 | 2.71 | 0.99 | 0.76 | 2.06 | 0.96 | 1.51 |
In the third elicitation study the experts (
The team that participated consisted of 11 experts, 10 regional directors and one director. All were eligible to be included in the study. To comply with conditions set by the Ethics Committee, we ensured that experts whom did not wish to participate could do so without it being known that they refused. Therefore we randomly selected seven out of the 11 experts and invited them to participate. Out of the seven selected experts that we approached, three indicated that they did not want to participate in the study and four indicated that they were willing to participate. All four experts that agreed to participate, did participate and completed the elicitation. The participating experts first performed a practice elicitation for their own sales team before moving on to their estimate for the whole country, enabling them to acquaint themselves with the elicitation applications. Offering this practice elicitation could improve the quality of the elicitations (
The Five-Step Method was used in this elicitation study and it consists of the following two parts: the first step is designed to support the expert in the use of reasoned and structured thoughts to obtain an estimate for the location parameter μ_{0}. In the second step the estimate for μ_{0} is used and the expert is asked to provide a reasonable lower and upper bound for their estimate so the prior distribution for the mean turnover per professional can be constructed.
The “chips” placed in the first step were intended to represent individual professionals in the trial run and clusters of similar professionals in the elicitation concerning the whole country. Visual feedback was provided on the elicited distribution, accompanied by a description of the value for μ_{0} by the researcher. The expert could accept the representation of their beliefs or adjust input until the representation matched their beliefs. Results concerning country wide performance where discussed in terms of total turnover for all professionals within the team, therefore the estimate for μ_{0} was transformed using the following function
where 𝜃 represent the parameter of interest and 𝜃 ∼ N(μ,σ^{2}) so that 𝜃^{∗} ∼ N[aμ+b,(aσ)^{2}].
The use of the mean as location parameter offered additional options to accommodate differences in reasoning of experts, e.g., a sales expert might feel comfortable to provide estimates for the total turnover of a store, represented by 𝜃^{∗}in Eq. 2, but not be comfortable providing estimates for the mean turnover per product sold in the store, represented by 𝜃 in Eq. 2. By knowing the total amount of products that are sold in the store, entering the amount as value for a and 0 for b in Eq. 2, the prior beliefs regarding the total turnover can be transformed to prior beliefs regarding mean turnover per product and compared to predictions by other experts. The transformation procedure ensures no expert is forced to adhere to a certain scale. To illustrate this flexibility let us imagine that a store sells nine different types of products and in total sells 104 products. In steps 1 and 2 we wish to elicit and verify the location parameter for the mean turnover. Two experts feel comfortable supplying estimates for turnover per product whilst two other experts only feel comfortable supplying estimates for turnover per product type. They can both adhere to the scale they feel comfortable with as we can use a linear transformation to get them onto the same scale for steps 3 and 4. In
Illustration of linear transformations using Eq. 2.
Steps 1 and 2 product scale mean result ( |
Steps 1 and 2 product type scale mean result ( |
Mean turnover per product used in steps 3 and 4 | Total turnover used in steps 3 and 4 | |
---|---|---|---|---|
Expert 1 | 1.8 | – | 1.80 | 187.2 |
Expert 2 | 2.1 | – | 2.10 | 218.4 |
Expert 3 | – | 23 | 1.99 | 207 |
Expert 4 | – | 24.5 | 2.12 | 220.5 |
In step 3 of the Five-Step Method, we asked the experts to provide a reasonable lower and upper bound for the total turnover of all professionals_{.} Based on the input a distribution was fitted and visual feedback was provided. The researcher supported the visual feedback with a description explaining that more density on places of the axis indicate more perceived likeliness for that value. The expert could accept the representation of their beliefs or adjust the input for the reasonable lower and upper bound until the representation matched their beliefs. The elicited distribution was transformed back to represent the average turnover per professional using Eq. 2.
During the elicitation procedures we noticed that not all experts reasoned in the same way. One expert reasoned for his own region in the expected elements, such that each “chip” represented a professional, but concerning the elicitation for the whole country the “chips” represented regional performances not clusters of professionals that are alike. This deviation did not require an adjustment of procedure just a different value for
All data were analyzed anonymously and were transformed to avoid revealing business-sensitive information. The elicited priors π_{d}(𝜃) can be found in
Results Five-Step Method elicitation study with large financial institution. Elicited expert distributions π_{d}(𝜃) plotted with results π(𝜃|y).
The values of the hyper parameters of π(𝜃|y) and π_{d}(𝜃) for the study with the large financial institution.
μ_{0} | σ_{0} | γ_{0} | μ_{1} | σ_{1} | γ_{1} | |
---|---|---|---|---|---|---|
Preferred distribution | – | – | – | 2.29 | 0.10 | 0.99 |
Expert 1 | 2.15 | 0.09 | 0.78 | – | – | – |
Expert 2 | 2.16 | 0.07 | 0.82 | – | – | – |
Expert 3 | 1.97 | 0.11 | 0.82 | – | – | – |
Expert 4 | 2.35 | 0.11 | 0.94 | – | – | – |
The Five-Step Method provides a first step for eliciting experts in a flexible manner such that no expert is forced to reason on a scale they are uncomfortable with, yet ending up with comparable priors for all experts.
In essence the Five-Step Method resembles the structure for eliciting a distribution as is proposed by
We acknowledge that asking experts for the reasonable lower and upper bound for their estimate in step 3 of the Five-Step Method could perhaps be an oversimplified procedure and other researchers might prefer to replace this step with eliciting quantiles.
Besides providing graphical feedback it is desirable to stay as close as possible to the reasoning experts use on a daily basis. The method should be adjusted to fit the expert’s reasoning and not the other way around if we do not want to introduce unnecessary bias. As shown in the study with the large financial institution, the Five-Step Method allows for just that. We can help experts order their thoughts, whether they reason in terms of individuals, regions or totals. All these ways of reasoning can be used by simply altering the value for a in Eq. 2 and thereafter transforming the values back to be compared on the same scale.
Using graphical feedback and flexible procedures remains a challenging task in an elicitation process. In the seminal work by
This study was carried out in accordance with the recommendations of the internal Ethics Committee of the Faculty of Social and Behavioural Sciences of Utrecht University, with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the internal Ethics Committee of the Faculty of Social and Behavioural Sciences of Utrecht University.
DV and RvdS mainly contributed to the study design. All authors have been involved in the design of (part) of the elicitation procedure. DV programmed the elicitation software. All elicitations have been facilitated by DV and DS. DV wrote and revised the paper with feedback and input of DS, MZ-Z, and RvdS. RvdS supervised the project.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The authors are grateful to all participants of the elicitation studies for their time, energy, and predictions. Also they would like to thank the participating companies for allowing us access to their resources and information thereby enabling us to provide empirical support for the theoretical work.
Using the SN distribution we represent the tacit knowledge of experts by eliciting the location parameter of the distribution, in this case the mean. The uncertainty of the expert about his/her belief on the location parameter is represented by the scale and shape parameter (i.e., variance and skewness of the normal distribution). Eliciting the mean of a normal distribution offers the advantage of easily transformable scale for elicitation procedures. An adjustable scale means that even if one expert reasons in averages and the other expert in sums they can be transformed to be comparable, i.e., let 𝜃 represent the parameter of interest and 𝜃∼N(μ,σ^{2}) and if we transform 𝜃 via the following function 𝜃^{∗} = a𝜃 + b, then 𝜃^{∗}∼