There are numerous ways to collect data. Some methods include:
- Surveys
- Polls
- Focus groups
- In-depth interviews
- Case studies
The strengths and limitations for these various methods are described in-depth in Unit 3 as well as a comparison of quantitative and qualitative measures. However, what is important to note here are the basic decision points. As noted earlier, the nature and extent of the research question and potential application of any findings will determine what data to collect and thus, what measure(s) to use the collect the data. Decisions on how to obtain data should consider the following factors:
- Type of data: Is data on service utilization desired for reporting purposes to renew funding? Is data on perceived effectiveness of distance education courses desired to justify expansion of online course offerings? If the data is based on a quantitative instrument or even a pre-existing, standardized measure, what will the data capture (e.g., significant differences, trends, group comparisons, etc.)? What will it not capture (e.g., detailed rationales or beliefs as to why something is, individual nuances in perception, personal histories, etc.)? There are always trade-offs to any type of data collected, so it just depends on what will the yield the best data for the stated purpose and parameters of the research.
- Size and recruitment of sample: Is the goal to collect data from as many people as possible within a population? Is the goal to obtain a smaller random sample from a larger population? Is the goal only to collect data from a few individuals? Can the sample be obtained from an existing database, or is active recruitment necessary? How and when should sample recruitment occur? For example, if the research question is to explore the experiences of first-year students, should a sample of students be recruited in their second semester on campus? Would fuller data be obtained if students were sampled and surveyed after one full year on campus, as in May before the academic year ends? If so, will survey response rate be affected by final exams and projects at that time? Will responses be skewed towards the negative at that time because of final exam stress? These points to ponder are essential because they influence the quantity and quality of the data, the validity of the results, and the accuracy of any conclusions drawn from the findings.
- Available resources: If the goal is to survey all student athletes at an institution but only one person can be allocated to the project at 25% FTE and that person is responsible for all assessment activities (survey development, sample recruitment, data collection and analysis), is surveying all athletes logistically possible? Perhaps it is possible at a small liberal arts college, but what about a large public university? And even if it is possible, does it make sense to survey all in the target population? Would the data be any more or less valid if it were collected from a smaller, randomly selected sample of athletes? In short, access to resources such as staff time, number of staff, institutional monies, supplies and equipment, etc. need to be weighed against the need for desired data and sample size. Many times the mistake is made to initiate too ambitious of an assessment or study that is not feasible to conduct given initially limited resources or resources that are reallocated for another purpose down the road. That is why it is often best to start with a smaller study and build upon it rather than reaching for the stars right away.
Similarly, as part of evaluating available resources and what is logistically feasible, it is always wise to determine if any of the needed data exists by contacting offices on campus that regularly collect data (e.g., admissions, financial aid, graduate schools, human resources, etc.) and by researching prior studies in the literature or that may have been conducted on campus. It is also wise to determine the cost of developing an assessment instrument (surveys, polls, interviews, focus group protocols) versus purchasing a published package or pre-existing survey. For small, more informal studies, developing and customizing one’s own survey may be all that is needed. However, for larger studies requiring a high degree of empirical rigor and generalizability to other populations, pre-existing measures that have been statistically validated are usually the preferred method and have to be purchased from the developer or publisher. For instances in which pre-existing instruments do not exist on a topic of inquiry, it may be necessary to pay a consultant to develop and statistically validate an instrument, particularly if that expertise is not available in-house.
In sum, these are all factors to consider when planning an assessment process and deciding upon research methods:
- What resources (monies, office staff or Advisory Committee time and labor, supplies and equipment, etc.) are available in my program or unit?
- What resources can be obtained through in-kind contributions?
- What resources can be obtained elsewhere at minimal or discounted cost (as through campus partners, Advisory Committee members, community businesses, etc.)?
- What aspects of the assessment will cost the most but are integral to the assessment’s success?
- What aspects of the assessment can be scaled down to conserve costs without undermining the integrity of the research and intended goals?
