An Examination of Best Practices for Survey Research with Agricultural Producers

Abstract To improve the economic and environmental sustainability of agriculture, information is needed on how to target research, teaching, and outreach programs. However, conducting survey research in general, and with agricultural producers specifically, is increasingly challenging given issues such as declining response rates and limited resources. While studies examining the best practices for promoting higher response rates exist, few focus explicitly on agricultural producers. In three separate surveys conducted with agricultural producers in South Dakota in 2018 and 2019, we included experiments testing how token pre-incentives, a research partnership, and response mode options impacted response rates. We also examined how sample source and email augmentations influence survey responses. The study findings indicate that providing pre-incentives and multiple simultaneous response options can increase response rates with agricultural producers. On the other hand, email augmentation to mail surveys, sample source, and identification of select institutional research partnerships appear to have minimal effects.


Introduction
While survey research has become more challenging in recent years with declining response rates, it remains a commonly used method for gathering information from agricultural producers about their attitudes, values, and behaviors . Benefits of survey methods can include versatility, efficiency, and, if using probability sampling methods, the generalization of the findings from the sample to the study population (Chambliss and Schutt 2019). Farmers receive frequent requests to take surveys on a variety of topics from the government, private sector, and academic institutions, yet our understanding of best practices for conducting survey research with agricultural producers is not often tailored to this specific, and in some ways unique, population 1 . We included experiments and tests in three separate surveys conducted with agricultural producers in South Dakota to test how pre-incentives, partnership with a government agency, response mode options, sample source, and email augmentation impacted survey response quality and quantity. The information presented in this research note can be used by those designing and conducting survey research with agricultural producers in the Midwestern United States and elsewhere to gather generalizable and valid data for targeting research, teaching, and outreach programs aimed at improving the economic and environmental sustainability of agriculture.

Response Rates and Survey Research
There are increasing concerns about the generalizability of surveys given the steady decline in response rates during the last few decades (Rogelberg and Stanton 2007;Dillman, Smyth, and Christian 2014;Stedman et al. 2019). Response rates are considered an important, though not the sole, determinant of data quality in probability sample surveys (Biemer and Lyberg 2003;Kreuter 2013). A similar decline over time has been shown in surveys with agricultural producers (Johansson, Effland, and Coble 2017). Some studies attribute declining response rates to "survey fatigue" given the growing number of requests that people receive to participate in survey research (Baruch 1999;Cycyota and Harrison 2006;Rogelberg and Stanton 2007).
A variety of practices including sending advance notifications and reminders and contacting participants multiple times and in multiple modes are effective at increasing response rates (Anseel et al. 2010;Dillman, Smyth, and Christian 2014). Studies have also found that respondent research topic saliency generates higher response rates (Edwards et al. 2002;Heberlein and Baumgartner 1978;Roth and Craig 1998;Stedman et al. 2019). Conversely, the presence of complex or hypothetical questions have been found to lower them (Stedman et al. 2019). In research with farmers, Pennings, Irwin, and Good (1999) found that survey timing (e.g., not during planting or harvest), survey length, and questions that do not require record searching improved response rates.
With regard to the first practice of interest in this study, incentives, particularly preincentives, in which potential respondents are given an incentive before they complete a questionnaire, can increase response rates among different populations (Dillman, Smyth, and Christian 2014;Glas et al. 2019). Research with U.S. farmers has also shown that monetary and non-monetary incentives can increase response rates, but that monetary pre-incentives work best (Arora et al. 2020;Beckler and Ott 2007).
For the second practice of interest, survey sponsorship, studies have shown increased response rates to surveys sponsored by universities or other public organizations as opposed to private sponsors (Anseel et al. 2010;Brunner and Carroll 1969;Dillman 2000;Fox, Crask, and Kim 1988;Heberlein and Baumgartner 1978). Other studies have found that when participants are unaware of the sponsoring organization, why the information is being collected, or have a negative view of the collecting organization, they are less likely to participate (McCarthy, Johnson, and Ott 1999).
For the third practice of interest, response options, providing potential respondents with multiple options (e.g., mail or online) for responding to a survey request is considered a best practice (Dillman, Smyth, and Christian 2014). When a variety of modes are available sequentially for participants over the course of a study, it can improve the response rate and reduce nonresponse error. However, providing multiple response mode options simultaneously can lead to lower response rates (Medway and Fulton 2012). At the same time, approaches using simultaneous response options that remove barriers to responding to both options (web versus mail) can turn choice into a positive rather than a negative feature (Dillman, Smyth, and Christian 2014).
Our fourth practice of interest is sample source. A common challenge in survey research using probability sampling methods is gathering a complete sampling frame of the population of interest. Those surveying agricultural producers often use a Freedom of Information Act request of the Farm Service Agency (FSA) to get lists of farmers who participate in government farming programs (see, for example, Floress et al. 2017). However, not all farmers participate in government programs (e.g., Amish producers [Brock, Ulrich-Schad, and Prokopy 2018;Ulrich-Schad, Brock, and Prokopy 2017]) and the FSA does not provide email addresses or any additional background information about program participants that would allow for nonresponse bias testing. Private organizations such as Farm Market ID, which is used as a sample source in this study, puts together their lists using data sourced from the United States Department of Agriculture, geospatial imagery, and other public and private sources. They can provide emails and/or mailing addresses for producers, as well as additional background information useful for nonresponse bias testing (e.g., acres operated). However, limited background information is available on individuals/operations included in the sample, emails are not available or incorrect for some, and how they obtain some information is unclear. We are unaware of any studies that examine how sample source plays a role in response rates and/or other forms of data quality.
Lastly, email augmentation, or combining postal mail contacts with timed follow-up email messages, is a technique that is increasingly being used to attempt to increase response rates. While limited research exists on this method, it has been shown to increase response rates (Millar and Dillman 2011). Anseel et al. (2010) point out that general guidelines on how to conduct effective survey research may not provide the most useful information for researchers. Survey researchers should be aware that response rates depend on the study population and type of respondent being surveyed because not all response-enhancing approaches are effective among all populations. Thus, this study seeks to determine whether some approaches that are seen as generally effective in increasing response rates work with agricultural producers in the U.S. Midwest today.

Surveys
In 2018 and 2019 interdisciplinary groups of researchers at South Dakota State University (SDSU) worked with a variety of partners throughout the state (e.g., South Dakota Corn Utilization Council, Natural Resource Conservation Service [NRCS], FSA) to conduct three separate surveys with agricultural producers in South Dakota. The questionnaire for each survey directed the person in the operation who made most of the land management decisions to answer the questions, providing both mail and online response options. Those operations not currently farming (e.g., renting out the land or retired) were asked to notify us via mail or online. Below we outline some key characteristics distinguishing each survey (see Table 1). Response rates for each survey were calculated as the percentage of completed surveys per total number of eligible respondents (Beaman and Vaske 2008). Significant differences in response rates were assessed with chi-square tests.

Commodity Crop Producer Sustainability Survey (Sustainability Survey)
In the winter of 2018, crop producers were surveyed regarding the sustainability of their land management practices and attitudes. Farming operations in 34 South Dakota counties east of the Missouri River, where most of the corn and soybean farming activities in the state are located, were targeted to take the questionnaire. We obtained a list of 10,000 farming operations that had participated in FSA programs in 2016 from the FSA and randomly selected 3,000 operations to participate using proportionate stratifiedrandom sampling according to number of farming operations in the study counties.
The sample was contacted up to four times via mail (advance letter, paper questionnaire and stamped return envelope, reminder postcard, 2nd paper questionnaire and envelope). One-half of the 3,000 operations selected for participation were also randomly assigned to receive a $2 bill with the advance letter to test if the pre-incentive increased response rates. The process used achieved a response rate of 30% after receiving 708 questionnaires back.
To assess the representativeness of the respondents (see Coon et al. 2019), we compared our respondents' age and cropland acres data with the USDA's 2017 Census of Agriculture. We found the key demographics of our respondents to be comparable to the state-level demographics. The average age of producers in South Dakota was 56.1 years, and the average age of respondents in our sample was 56.7 years. Farm size averaged 1,443 acres in South Dakota, while our respondents had an average farm size (total acreage of farmland operated in 2017 planting season) of 1,170 acres.

Grassland Livestock Producer Parasiticide Survey (Livestock Producer Survey)
From April to July 2019, South Dakota livestock producers were surveyed on their views and usage of parasiticides. The sampling frame came from two sources: (1) participants in the 2017 USDA FSA Livestock Forage Disaster Program (LFP) and (2) a sample purchased from Farm Market ID. Two sources were used in order to examine the effect of source on response rates and data quality. The list of operations purchased from Farm Market ID were randomly selected among their list of operations who use grazing land or pastureland for grazing and who manage 50 to 400 cattle in South Dakota (n ¼ 1,206). We purchased both mail and email contacts from Farm Market ID for half of the sample in order to enable test email augmentation (n ¼ 603). After removing duplicates from the LFP complete list of recipients, we randomly selected a sample of 1,210 operations to contact to participate in the survey. Combining both sources, we had 2,416 operations in our sample. Operations in the sample were contacted a maximum of four times at approximately two-week intervals (same design as Sustainability Survey). For operations with both mail and email addresses (n ¼ 603), we also sent an email reminder timed to arrive a day or two after the advance letter and reminder postcard were sent. The survey achieved an overall 27% response rate with 596 respondents.
We also included one experiment in this survey. To test whether including information about the NRCS research partnership along with the university sponsorship would increase the response rate, we designed two versions of the advance letter and cover page of the questionnaire. One version included the NRCS logo and wording about our partnership with them and the other did not. We randomly selected half of the final sample (n ¼ 1,208) to include the NRCS information as the experimental group, and the other half (n ¼ 1,208) without the NRCS information as the control group.
We conducted non-respondent tests on a few key variables in the data we collected from the Farm Market ID sample. Planted acres for respondents ranged from 1 to 8,062, with an average of 749 acres and a median of 519 acres. Planted acres for nonrespondents ranged from 1 to 10,904 with an average of 966 acres and a median of 602 acres. In this case, we find, using a two-sample t-test, that non-respondents had a significantly higher average acreage than respondents. The gross farm income (GFI) for respondents ranged from $28,011 to $2,424,012 with an average of $416,442 and a median of $304,193, while it ranged for non-respondents from $32,244 to $6,456,478 with an average of $508,285 and median of $343,250. Again, we find that the difference is statistically significant and that non-respondents tended to have higher GFI than respondents. Overall these tests suggest that the results are somewhat biased toward smaller operations both in terms of acres operated and annual income generated. However, the median values were much closer for both indicators than the means indicating the impact of larger operation outliers on the averages.

South Dakota Farmer Nutrient Management Survey (NM Survey)
For the NM Survey, producers in South Dakota whose primary crops are corn, wheat, and soybeans were targeted to answer questions about their attitudes and behaviors related to soil health management. The sample purchased from Farm Market ID (n ¼ 3,000) included farmers who were selected randomly from six crop reporting districts in the eastern part of the state where most of the commodity crops are produced. Stratified proportionate-random sampling was used to select farmers from each cropping district based on the number of operations in the district. Those in the sample were contacted three times (similar to the Sustainability Survey, but with no reminder postcard). Emails were sent to respondents who had email addresses in the sample approximately two days after receiving the mail correspondence to remind respondents to complete the survey if they had not (n ¼ 1,361). This allowed us to test whether using email augmentations increases response rates. Despite the length, relatively complex questions, and unfavorable timing (unexpected flooding), the survey achieved an overall 18% response rate with 465 respondents after excluding bad addresses (n ¼ 56) and not qualified (n ¼ 326) producers from the sample.
In this survey, we included an experiment to test whether the response rate differed by the number of response options provided to two randomly assigned groups (n ¼ 1,500 for each group). Group 1 was given (or notified about) multiple options to take the survey at the same time, either online or on paper. Group 2 was given only one option to take the survey at any given point in time. As such, in the advance letter, participants in Group 2 were given a link to follow to complete the survey online without telling them a paper survey would follow, whereas Group 1 was told a paper survey would follow. For Group 2, the paper survey, which was sent in subsequent waves, did not include a link to take the survey online, whereas for Group 1 it did. The same methods were applied in the email augmentations sent to subsamples.
Planted acres for respondents ranged from 5 to 17,681, with an average of 1,151 acres and a median of 681 acres. Planted acres for non-respondents ranged from 12 to 65,478 with an average of 1,365 acres and a median of 792 acres. Again, using a two-sample ttest, we find that nonrespondents had a significantly higher average acreage than respondents. The GFI for respondents ranged from $150,076 to $6,956,941 with an average of $593,270 and median of $381,060. The GFI for non-respondents ranged from $150,249 to $20,496,506 with an average of $702,600 and a median of $436,552. Non-respondents had a significantly higher GFI than respondents. These nonrespondent bias tests again suggest that the results are somewhat biased toward smaller operations, both in terms of acres operated and annual income generated. A few large operations serve as outliers that are skewing the mean values as the median values were much closer for both indicators than the means.

Results
In the Sustainability Survey, the overall response rate was 30%. Of those who received the $2 bill incentive, 32% responded, while 25% of those without incentive completed the survey, a statistically significant difference at p < 0.01. Including the NRCS partnership information and logo in the Livestock Producer Survey did not help with improving the response rate (Table 2). We also tested for interaction effects between the logo and sample source, finding no difference. In the Livestock Producer Survey, the USDA sample had a somewhat higher response rate than Farm Market ID (Table 2), though the difference was not statistically significant. In terms of data quality, 7.3% of those in the USDA sample and 15.2% of those in the Farm Market ID sample indicated that they were not currently farming, a statistically significant difference. In the USDA sample 2.4%, and in the Farm Market ID sample 1.8%, were returned from the post office as bad addresses, a non-significant difference. The mean total acreage of USDA sample is 3,084, while the mean total acreage of Farm Market ID sample is 2,683, a statistically significant difference. In the NM Survey, those with a single response option had a 15.9% response rate while those with multiple response options had a 19.8% response rate (Table 3), a statistically significant difference at p < 0.03.
In the Livestock Producer Survey, the response rate for those with the email augmentation was slightly higher than for the non-email subsample (Table 4), yet the difference was not statistically significant (Table 4). For the NM Survey, the response rate for those with email augmentation was slightly lower than for the non-email sample, and the difference was also not statistically significant.

Discussion and Conclusion
Our findings lead us to offer a few recommendations for future survey research with agricultural populations in the Midwest. Similar to Glas et al. (2019), we find that including pre-incentives in surveys to agricultural producers can increase response rates, and we recommend including a small token pre-incentive to boost response rates. Qualitative feedback from a small number of survey takers (in the comment box at the end of the surveys) suggests that some people appreciate a token incentive and/or the novelty of receiving a $2 bill. However, others also volunteered that the $2 did not adequately compensate them for the time they spent completing the survey (yet they still filled out the survey). While these extreme positions provide some insight into how pre-incentives work differently for different farmers, overall, evidence suggests that using token pre-incentives can improve response rates in surveys with producers. Future studies might continue to explore how different amounts and types of incentives work with different farming populations. Care should be taken to communicate to potential respondents that the $2 bill is not intended as compensation, but as a token of appreciation for their time. We did not find that including information in correspondence with our sample about partnering with another organization, the NRCS, made a statistically significant difference in response rates. In this study, we focused on whether adding sponsorship of a government organization that frequently works with farmers along with university sponsorship increased response rates and found it did not. Perhaps in this era of increasing skepticism of institutions, including the government (Pew Research Center 2017), respondents are less likely to use some affiliations as a determining factor in completing a survey. Our results also reflect that participants may have already trusted the primary source of the survey (e.g., SDSU) and that adding an additional organization did not provide any boost to responses. Our findings suggest that the efforts put into obtaining approval from partnership organizations might not be worthwhile in research endeavors as long as the primary sponsor is well-respected by the study population. Future research might consider how different types of partnerships or interactions between them impact response rates.
Contrary to some literature (Medway and Fulton 2012), we find that providing survey respondents with multiple response mode options simultaneously (online and by mail) increases the tendency to respond. With perhaps increasingly divergent preferences and skills among various segments of the farming population, offering multiple options to respond in each wave of the study provides participants the opportunity to respond with the option that works best for them. Given their greater exposure to the internet and technology (Ackoff, Bahrenburg, and Shute 2017), it is likely that younger farmers are more comfortable taking surveys online and prefer to do so. Providing multiple options can also save costs by pushing some respondents from mail to the web. While creating multiple response mode options can create some additional work for research teams, it is worthwhile in terms of boosting response rates and is thus recommended.
Using a sample provided by a government organization like the FSA provided a slightly higher response rate than the sample provided by a private company and significantly fewer non-eligible respondents. However, when purchasing contact information from private organizations, you can often get multiple types of contact information for some of the study population. At the same time, you might increase costs by having to contact more respondents given the higher level of non-eligibility. Private organizations can also provide some additional, though limited, data on characteristics of those in the sample that can be used for nonresponse bias tests. In sum, choosing a sample source should depend upon research design, the resources available to the research team, and the type of farmers being studied. Given the limitations inherent in nearly any sampling frame, this finding points to multiple options for researchers seeking to obtain a sample of Midwestern producers to survey. Given it is increasingly important to conduct nonresponse bias testing as response rates decline, those conducting research with agricultural populations might want to consider budgeting the purchase of all or some of their sample from private organizations to allow for this important assessment of the data collected.
Finally, we had mixed, but not statistically significant, results in regard to whether email augmentation increases response rates. This finding suggests that purchasing email lists from private organizations may be of limited utility in boosting response rates with Midwest agricultural producers and thus should only be done if resources allow. Notably, most farmers still prefer to take mail surveys (62.2% of the NM Survey, 65.6% of the Livestock Producer Survey, and 60.1% of the Sustainability Survey respondents took the paper version), thus the email augmentation appears to do little to push more respondents online.
To further understand how to improve survey research with agricultural producers, future studies could also focus on a number of areas including the usage of reminder postcards in mail surveys, the complexity of survey questions and the type of questions which produce item nonresponse, data quality collected from panel studies, and survey fatigue. Conducting additional experiments with agricultural producers throughout the Midwest will lead to a clearer understanding of how the findings from South Dakota apply across the region and contribute to the collection of more valid and reliable survey data.