Summary
Using a SWOT framework, we evaluated a multi-day workshop on integrating generative AI into course design, gathering insights on strengths, weaknesses, opportunities, and threats through open-ended feedback. This structured approach provided valuable information on participant reflections, enabling facilitators to refine workshop effectiveness and support future educational initiatives.
Recently, we employed a comprehensive evaluation framework to appraise a multi-day workshop on integrating generative AI into course design. In this article, we will walk you through the evaluation and data collection procedures and reflect on how such evaluation studies can aid practitioners in refining their workshops. We started by selecting the framework where, in this case, we focused on the Strengths, Weaknesses, Opportunities, and Threats (SWOT) framework to guide our analysis.
Using SWOT Analysis to Understand Workshop Impact
Similarly, you can take advantage of the multiple virtual boards available during the workshop to facilitate real-time SWOT analysis:
i. Miro : https://tinyurl.com/3bh6uv5z
ii. Mural: https://www.mural.co/templates/swot-analysis
If you’re interested in conducting small group feedback, we suggest taking advantage of the variety of free templates available. For instance, Adobe Express offers 15 Free SWOT analysis templates that are fully editable. https://www.adobe.com/express/learn/blog/swot-analysis .
However, SWOT analysis focuses on a more qualitative approach, and we used participant feedback to uncover recurring themes and valuable insights. The thematic analysis enabled us to identify patterns across the data, while IPA provided deeper insights into individual perspectives, effectively capturing the workshop’s impact. Here are some open-ended questions that can help gather in-depth insights from a workshop:
- Strengths:
- Which elements of this workshop have had the greatest positive impact on your learning or teaching practices?
- Please share any specific activities or strategies that proved to be effective.
- How did the workshop enhance your skills and understanding of the subject matter?
- Weaknesses:
- What challenges or difficulties did you encounter during the workshop?
- Were there any parts of the workshop that could have been more effective? How?
- How did the structure or content of the workshop fall short of your expectations?
- Opportunities:
- What opportunities for professional growth did you identify through this workshop?
- How can the insights gained from this workshop be applied to improve future sessions?
- Are there any new ideas or methods you discovered that you would like to explore further?
- Threats:
- What potential obstacles could hinder the effective implementation of what you learned?
- Did you experience any concerns about balancing new methods with existing practices?
- How could the workload or stress associated with the workshop impact your future engagement?
To enhance understanding of the workshop experience, it is recommended to incorporate pre-workshop questions to understand participants’ expectations and post-workshop questions to gather other feedback, recommendations, and future application strategies. Researchers/ practitioners may also consider including questions to assess training objectives, content, delivery, teaching strategies, facilitator performance, and overall reaction post-workshop. These questions can be adapted from various questionnaires/ instruments, and one good example is the Workshop Evaluation Questionnaire at https://www.utpjournals.press/doi/pdf/10.3138/cjpe.26.006
Data Collection: The Workshop Experience
Our data collection process involved two key sources: daily reflections or virtual discussions and end-of-workshop surveys. These sources offer valuable insights and diverse perspectives from workshop participants. To enhance the credibility of our findings, we aim to triangulate the data due to our limited sample size. Daily reflections provide real-time group feedback, while open-ended surveys allow participants to articulate their individual experiences in greater detail. Researchers should always consider the richness of data and the time required for data interpretation, analysis, and reporting. Furthermore, it’s essential to consider the number of participants. We would recommend employing a qualitative approach for smaller sample sizes (<10-20) and a pre-post quantitative analysis for larger sample sizes. Likewise, combining a QUANTITATIVE + QUALITATIVE methods can enhance the credibility and depth of our findings, leading to a comprehensive understanding of the participants’ experiences.
Equally, educators and workshop facilitators can benefit greatly from systematic evaluation processes like using SWOT. These processes offer more than just insights into workshop outcomes; they also provide a structured approach to understanding the implications of adopting new practices and improving future workshop effectiveness. Below are some potential benefits:
Conclusion:
Successfully evaluating workshops can be challenging, but by conducting thoughtful evaluations, fostering critical engagement, and providing continuous training, educators can enhance their teaching practices while minimizing potential risks. The experiences of these participants serve as a powerful example of the impact of reflective and evidence-driven evaluation, inspiring practitioners to elevate their educational initiatives.
If you have questions, need further consultation, or are interested in discussing a potential collaboration, please email us at edli.team@msu.edu.