Introduction
There are plenty of higher education programs that proved beneficial for students and are relatively easy to implement for the faculty. MIT’s Advanced Study Program (ASP) is one of those choices the working students and young professionals make to empower themselves with some additional valuable knowledge related to their current field of expertise. Correspondence students of ASP have an opportunity to spend only from 2 to 5 hours a week being physically present at the courses, which allows them to not quit their jobs and get the education they need (“Advanced study programs FAQs,” 2021). The program also has many other essential features and opportunities, like the extensive curriculum, the option to save the credit for getting degrees in the future, travel for hands-on research, and found their companies (“Education,” 2021). The program-oriented approach will be the most effective to evaluate the presented program. This paper aims to discuss the ASP, explain the rationale for selecting a program-oriented evaluation approach, including the strengths and weaknesses of the evaluation approach, and determine the future needs for evaluating the ASP.
The rationale for Selecting the Program-Oriented Evaluation Approach
Evaluation is a necessary element of good functioning for any academic program or other people-oriented activity. For ASP, evaluation can be particularly effective since it will allow the stakeholders to get more insights into the program’s opportunities and potential (Whitaker, 2017). Practical evaluation will also create a basis for further improvements; moreover, the program coordinator can use the evaluation results for cooperation and marketing purposes (Whitaker, 2017). Therefore, performing the program evaluation makes it possible to enhance the decision-making processes related to the program initiative and develop new branches of the existing programs. The evaluation usually helps the stakeholders focus on how the initial idea can be improved and how good it is working at the moment.
The program evaluation approach is highly applicable for educational programs, especially for the mentioned ASP program. According to the Program Theory, the program-oriented evaluation approach was initially designed in 1983 by Chen and Rossi to measure the variables that impact the program outcomes (Mertens & Wilson, 2018). Mertens and Wilson (2018) claim that later the Program Theory was then used by other scholars to develop the logic models or log frames that facilitate the evaluation. Logic models are a branch of the program-oriented approach, just like the objective-oriented evaluation. According to Mertens and Wilson, logic models feature the “graphic depictions of inputs, resources, assumptions, activities, outcomes, outputs, and impacts of a program being evaluated” (p. 43). Therefore, the program-oriented evaluation presents a clear and straightforward frame for evaluation that considers all critical elements in the subject under evaluation.
Program-oriented evaluation theory was also later extended by scholars to create various practical methodologies in evaluation. This approach can also be mixed with other evaluation models, approaches, and theories, due to its versatility and universalism. For example, the program-oriented approach can be mixed with the participant-oriented evaluation and other program-theory-driven strategies with inclusive and transformative elements, including the utilization approach and empowerment evaluation (Mertens & Wilson, 2018). Therefore, the program-oriented evaluation approach has many facets and can be applied to different types of evaluation, including the assessment of training and educational programs. It should be applied to the ASP since this program has many directions and opportunities covered, which creates a good base for measuring the outcomes and the student’s activity.
Strengths and Weaknesses of the Program-Oriented Evaluation
The program-oriented approach was developed more than three decades ago, and it has gained popularity due to some outstanding strengths. For instance, scholars and program stakeholders usually choose the program-oriented approach thanks to its clarity and logical utility. Koleros et al. (2018) note that the program-oriented approach is one of the predominant ones currently in use. Scientists also note that the logic model or log frames are most often used. Remarkably, these models “represent change as a linear process that can be modeled using a direct causal pathway of inputs, outputs, outcomes and impacts,” which is an undeniable advantage of the approach (Koleros et al., 2018, p. 34). Program-oriented logical models are mainly used to provide accountability measures, planning objectives, and development programs.
However, the scientists also point out some particular weaknesses, such as excessive linearity. Approach-based methodology and design are most often focused on “the immediate causal logic of the program, with less emphasis on feeds and feedbacks, or identifying feedback loops (both positive and negative), full recognition of the program context, and multiple factors that affect the process of change, regardless of the specific program” (Koleros et al., 2018, p. 35). It can be considered the weakness of the given approach since simplified causal models are likely to limit the effectiveness in planning, monitoring, and evaluation. Therefore, scientists concluded that widely used methods should be more multidirectional and multifunctional. This is particularly important, given the recent tendencies for more thoughtful analysis and evaluation of the programs in business, training, and education.
Future Needs for Evaluation of ASP
Therefore, ASP can have future needs for additional enhanced evaluation, which will consider the weaknesses of the program-oriented logical model evaluation approach. Interestingly, Saalman (2019) discusses an objectives-driven participatory evaluation model that has the elements both of program-oriented and participant-oriented models. Initially, it is planned to evaluate how the ASP program meets its objectives to provide valuable educational services for students who have permanent jobs. For instance, in the first stages, the program-oriented approach will be applied to gather the data on learning outcomes, learning activities, inputs of faculty, and resources spent by both university and students. The evaluation will measure whether the outcomes are good enough to cover the number of resources spent, including money and non-monetary resources.
However, at some point, ASP may need a more extended evaluation, which could be performed by applying the mixed approach. Saalman (2019) suggests using a mixed objective-participants-oriented model to cover the less linear outcomes, like getting feedback from students participating in the ASP. Harman and Azzam (2018) are also supporters of mixed approaches and suggest using crowdsourcing to perform the qualitative analysis of the participants’ experiences. In particular, scholars believe that the study of the experience of many participants who themselves help conduct qualitative research of their experience is a promising direction in the theory of program assessment.
The use of mixed objectives-participants-oriented and participants’ crowdfunding approaches can be effectively used in the future to achieve the original goals of the ASP evaluation. Engaging participants and actively learning about their experience with the program will broaden the evaluation perspective through direct feedback. Details of participant experience scores can be of significant qualitative value compared to other performance indicators in linear scores. However, the logical model will be used in the future to support the participant-oriented approach by providing the rational basis for the multi-faceted qualitative evaluation. The objectives-oriented evaluation should also be applied for the same reason. Moreover, the program-oriented theory can and should be applied in the future in its initial, non-mixed state to juxtapose the newest mixed evaluation methods and ensure the two types of approaches are complementary.
Conclusion
Thus, the ASP was discussed, and the rationale for selecting a program-oriented evaluation approach was explained. ASP is a program for students who work at permanent jobs and want to empower themselves by taking additional courses in their expertise. The strengths and weaknesses of the program-oriented evaluation approach were named, and the future needs for evaluating the ASP were determined. The ASP program has plenty of benefits, like an opportunity to live in the campus dormitory together with other students, take credits applicable to other courses and higher education organizations, and participate in travels for hands-on research. The program-oriented evaluation, which can include the logical model and objectives-oriented evaluation, can be beneficial for assessing the ASP, as these models can cover all multi-faceted opportunities offered by the ASP. Since the program-oriented approach and the logical model are sometimes blamed by scientists for excessive linearity, it is considered the presented approach’s weakness.
Therefore, the program-oriented approach can be mixed with the participation-oriented evaluation to provide more detailed feedback that can be further used as valuable qualitative information. Applying an original crowdfunding methodology is another practical option since involving participants in a qualitative analysis of their learning experience within the program will reduce the time spent on evaluation. It is noteworthy that mixed approaches are an innovation, as scientists initially preferred to use pure models.
Nevertheless, given the multi-component nature of the educational and training programs, a mixed approach proved beneficial both for scholars and the program stakeholders. Therefore, this innovation will be used in the future for ASP evaluation to consolidate the trend and contribute to developing a new theoretical framework for performing the evaluation. The program-oriented evaluation approaches, and their branches like objective-oriented evaluation and logical model evaluation, are highly effective for assessing educational programs, including MIT’s ASP.
References
Advanced study program. (2021). MIT. Web.
Advanced study program FAQs. (2021). Web.
Education. (2021). MIT. Web.
Harman, E., & Azzam, T. (2018). Towards program theory validation: Crowdsourcing the qualitative analysis of participant experiences. Evaluation and Program Planning, 66, 183-194. Web.
Koleros, A., Mulkerne, S., Oldenbeuving, M., & Stein, D. (2020). The actor-based change framework: A pragmatic approach to developing program theory for interventions in complex systems. American Journal of Evaluation, 41(1), 34-53. Web.
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice. Guilford Publications.
MIT subject listing and schedule fall 2021. MIT. Web.
Saalman, D. R. (2020). Objectives driven participatory evaluation model. Journal of Modern Applied Statistical Methods, 18(2), 14. Web.
Whitaker, S. (2017). Developing a program evaluation proposal.