Climate assessments, as with most research studies, are driven by a formulated research question. The questions range from being exploratory in nature (e.g., “What are faculty attitudes towards students with disabilities?”) to correlational (e.g., “There is a positive relationship between learning supports and retention of students with disabilities) to experimental with hypothesis testing (e.g., “Students’ academic outcomes will increase when instructors use multi-modal teaching techniques in their classrooms”). A good research question will specify variables and how those variables are to be measured. For instance, “attitudes” and “academic outcomes” are far too broad and complex to measure. Thus, to keep the assessment measurable, meaningful, and manageable, variables or indicators must be chosen that represent the domain(s) of inquiry. Using the hypothesis statement example, a better way to phrase that statement to specify greater measurement would be as follows: “Student academic outcomes, as measured by grade point averages (GPAs), will increase when instructors use multi-modal teaching techniques in their classrooms, as measured by frequency of use on the Office of Teaching Excellence’s annual faculty survey.”
Though it is not the purpose of this text to provide a review of basic research, it is important to note that selection and phrasing of the research question is fundamental to the assessment process. Domains and variables specified ultimately determine the type of data desired, the appropriate stakeholders and resources to involve, who will find the results relevant, and how the data is to be collected. Therefore, individuals or Advisory Committees should carefully deliberate and consider what it is they really want to measure, what it is they really want to know about in order to focus their efforts. Some important factors to consider in creating a research question include:
- Who is the population of study? Based on the research question, the sample of subjects asked to respond will have an impact on the outcome. The audience targeted to respond to data collection must be determined early on and may influence the research question being asked. For example, different responses are likely to come from students who have been on campus for four years than incoming freshmen in their first semester who are not as familiar with their campus. Asking seniors “Have you ever had a class where someone used a sign language interpreter?” will yield different answers than asking freshmen in their first semester that same question because seniors have more than three more years of experience behind their responses. Each group may have responses relevant to improving campus climate, but different information will be received from them. The kind of information that is desired from the beginning of the climate assessment as well as the anticipated target audience(s) are critical factors to be considered in the form and content of the research questions.
- What kind(s) of data is/are desired? It is critical to ask and determine this question so that appropriate data collection methods are selected. So if a person or committee wants in-depth information on the on-campus experiences of graduate students who are Latino and African-American, then the data desired is more suited for qualitative measures such as focus groups and interviews that are meant to capture greater depth and that do not require large samples to be effective. In contrast, if a unit on campus wants to know about alumni satisfaction with college outreach and event announcements or how frequently alumni, including graduates from various ethnic or underrepresented groups, take advantage of reduced discounts for events, then a brief survey would probably be the best choice because it can be distributed to a large number of people fairly inexpensively via mailings, listservs, etc. It should be noted that before initiating the assessment process, the development team will want to conduct some campus research to determine if the research question or something similar has already been investigated and if there is existing data available on the subject. Avoiding reinventing the wheel is a good maxim to follow in climate assessment and in research in general. Because of the growing emphasis on outcome research, many units on campus are collecting vast quantities of data stored in databases on a central or local server. It is always a smart idea to investigate what data has been collected and what data is available for access before committing staff, money, time, or energy to a process that may, in fact, be yielding needlessly duplicated data.
- Who will have access to the results and how will they be used? When framing the research question, stakeholders should also consider who will be sifting through the data received and how it can possibly be interpreted in light of the research question(s) the stakeholders hope to answer. Implications of the results for program evaluation or for future research must be weighed against assessment limitations, shortcomings, and possible consequence. For example, if an administrator learns of declining enrollment of students with disabilities at his/her institution, the administrator will want to explore possible contributing factors to the decline, whether the data is reflecting a longitudinal trend or a brief snapshot in time, whether the trend is local or national, and possible impact on funding from federal, state, and local sources, including resulting increases in tuition for students. Anticipated (and non-anticipated) outcomes and their applications should be considered as much as possible so the data is utilized to its fullest extent once it is obtained.
