STEM Central Short Course Part 1: Foundations of Culturally Responsive Evaluation (1 of 3)
This is the first of a three-part STEM Central Lecture Series: Program Evaluation Principles and Strategies for NSF-Funded STEM Reform Practitioners. . In this series, NSF PIs will learn about the purposes of evaluation and the multiple methods of evaluation that can be used to demonstrate the effectiveness of advanced undergraduate STEM teaching strategies. The lecture series will address such topics as study design, data analysis, and data interpretation. This lecture series will also blend the foundations and issues of evaluation with methods of data collection and the importance of cultural responsiveness – all with a particular emphasis on NSF STEP and S-STEM projects. Each lecture builds upon the previous, and thus we encourage our attendees to register for and attend all three webinars.
Melvin Hall
Sharon Jones
Caitlin Cairncross
The following is the third question asked by a member of the audience, followed by Sharon and Caitlin's Response:
Q3: How important is it to have an evaluator who is local and can visit the school frequently?
A3: I think that it depends on the purpose and the role of the evaluation, but generally speaking, it is not a deal breaker for the evaluator to be at some distance from the project. What I always recommend is that you have an internal evaluation component and an external, and if the external person is involved in developing the internal, there is data being fed to the evaluator continuously throughout the project, and so you can be efficient in the amount of time the person spends actually on site. So I think what’s more important is having an internally valid data collection that is ongoing on a regular basis, a daily basis, and in communication with an external evaluator, moreso than having to have somebody on site a great deal. Our evaluator, while separate from our program and separate from our school, is in our university, and so the local knowledge that she brings regarding the type of students we have here and the financial issues that they face, the type of sort of unique culture that’s here at a small private Catholic institution in Oregon has helped us, especially in terms of how she’s been able to conduct the focus groups. So we feel that we benefited from the fact that she is local.
The following is the second question asked by a member of the audience, followed by Sharon and Caitlin's Response:
Q2: How do you utilize your initial evaluation time in this case study?
A2: To go back to the initial evaluation plan, I’ll be honest, we certainly could have benefited from having an evaluator help design the project and the proposal upfront and we did not. We brought on an evaluator after we were funded. So the initial evaluation plan was developed by the PI and the Co-PI, and the program manager. And then we looked to find an external evaluator. And so the initial plan, I’d say, was changed after we brought in an external evaluator,a dn that plan is what has been the basis for our qualitative evaluation. We have a quantitative evaluation plan we developed up front, and we have been following that very faithfully, but the qualitative evaluation is where we/ve learned so much about the changes needed in the program, learning about student populations, really learn more about how to design this type of study and type of intervention. I think most evaluators today would say that a mixed method approach is really called for by most programs. Any program that involves humans, which is all the programs we’re concerned about, probably needs some sort of qualitative component in order to understand the quantitative. We knew that upfront, we always planned on a mixed method, we had some preliminary surveys and focus group questions and things like that, but bringing someone who was outside the program but still works at the University of Portland was very beneficial in terms of improving.
The following was the first question asked by a member of the audience, followed by Sharon and Caitlin's response:
Q1: Can you say more about why you prefer a sophomore focused program over a freshman bridge program?
A1: Originally, we were looking at the track record of bridge programs that had worked in other institutional settings. I think as Melvin pointed out, culture, and the context of the institution and the students is important to consider. What we did originally was to adopt the best practice, which would be the pre-freshman year summer bridge, without necessarily realizing the differences in our student population compared to the student populations in some of the other institutions that had worked well. And for us in particular, distance and logistics became a real problem in terms of attracting the numbers of students we needed to succeed. What we also realized as far as qualitative evaluation, is that many of these students in pre-freshman bridge were coming to it, in a large part, because of an outside influence telling them to come there, not necessarily their own motivation. I guess that gets back to all of the influences on them before they start college. What we found was with the sophomore bridge, the students in that bridge are self motivated to participate, moreso than a external influence from a parent or something like that. The other reason we decided that the sophomore bridge was a better way to use our resources was because of something that Caitlin referred to in her presentation. She talked about having to find a way to hook students into participating in the year long program, and having the sophomore bridge, at the end of that year long program, provided that hook. We see students very anxious about applying to the program: will I get accepted? Have I participated in academic programs? And we believe that the year long program has just as much value as the summer bridge, and so we are trying to encourage participation in both. We tried doing professional development with both groups and I will say, the sophomore group responded much better than the pre-freshman group did, just because they had a year of courses under their belt, I think they were more engaged with learning opportunities and things like that. It's not a make or break for us, but it is something that has been a positive coming out of the summer bridge program. We learned a lot throughout the project, and one thing we are trying to do is help the students develop their academic identity, their social identity, and their professional identity. The sophomore bridge seems to be more effective in that regard. If you can, get your evaluator involved in any way you can with writing the proposal. One of the things that having an evaluator at the table with you during the design phase will and can do is be the counterpoint. You’ll be focused on how we can help these students, and the evaluator will be focused on how can we see this come together? And will raise questions during the design process that may identify things like maybe that worked at some other place but that won’t work here. So I encourage you to get maximum value from your evaluator by having them involved in the design phase.
Here is a list of resources that Melvin shared during the webinar:
American Evaluation Association Evaluator Search http://www.eval.org/p/cm/ld/fid=108
Claremont Evaluation Center http://www.cgu.edu/pages/9963.asp
Washington University in St. Louis Evaluation Center http://www.cgu.edu/pages/9963.asp
The Evaluation Center at Western Michigan University https://www.wmich.edu/evaluation/services https://www.wmich.edu/evaluation/checklists http://www.evalu-ate.org/
University of California Educational Evaluation Center http://ucec.gseis.ucla.edu/