Quality Criteria

On this page you’ll find the documented measurable quality criteria for DID-ACT project deliverables.


Quality Criteria for D1.1 (a) Report on specific needs, preoccupations of stakeholders, and barriers: web survey results

  • ≥ 35 interview participants (approx. 5/partner)
  • ≥ 70 survey responses (approx. 20 per partner)
  • Interview guideline for implementation and analysis provided
  • Diverse range of participants (faculty, students, educators,…) for the interviews and survey
  • Survey, interview questions, and report reviewed and agreed upon by all partners
  • Pre-testing of survey (think aloud, piloting with partners)
  • Conduction of as many interviews as necessary for clear understanding of current practices and barriers.
  • Scientific publication as a long-term quality criteria beyond this deliverable

Quality Criteria for D1.1 (b) Report on specific needs, preoccupations of stakeholders, and barriers: interviews

  • Interview guideline for implementation and analysis provided
  • Survey and interview questions reviewed and agreed upon by all partners
  • Diverse range of participants (faculty, students, educators,…) for the interviews and survey
  • Conduction of as many interviews as necessary for clear understanding of current
  • >= 35 interview participants

Quality Criteria for D1.2 Report on solutions for the needs described in D1.1 and consequences for the curriculum development process

  • Implement a Design thinking workshop informed by the interview and survey results (deliverables D1.1a and D1.1b) according to best practices (had to be adapted to an online format).
  • Participation of all partners in the design thinking workshop and providing an open atmosphere for discussion.
  • Prioritization of solutions by the partners regarding their importance and feasibility.
  • Provide and discuss solutions on an institutional and project level to ensure applicability also for non-partner institutions.
  • Discuss consequences for all identified barriers, even if no solutions can be provided by DID-ACT for all barriers.

Quality Criteria for D2.1 Learning objectives for the clinical reasoning curriculum in a database.

  • Consensus workshop to discuss CR frameworks, one for students and one for faculty.
  • Review CR frameworks in relation to the needs analysis and the literature
  • A graphical representation describing aspects and dimensions of the CR curriculum frameworks will be reviewed by two associate partners and an external expert in addition to project partners.

Quality Criteria for D2.2 Curricular framework with teaching/assessment methods for the student curriculum and the train-the-trainer course

  • Consensus workshop to discuss the clinical reasoning framework, for the student curriculum and the train-the-trainer course.
  • Review clinical reasoning frameworks in relation to the need’s analysis and the literature.
  • A tabular representation describing aspects and dimensions of the clinical reasoning curriculum frameworks will be reviewed by associate partners.
  • Constructive alignment serves as a guiding principle for the learning objectives and curricular framework development.

Quality Criteria for D2.3 Collection of available open educational resources (OER) suitable to be used in WP3 and 4

  • Review of each resource of at least one DID-ACT partner, other than the one entering the resource, in order to secure access and relevance.

Quality Criteria for D2.4 Publication of recommendations for learning objectives of a clinical reasoning curriculum

  • As a base for the recommendations we will analyse at least five different learning objectives catalogues from different healthcare professions and partner countries.
  • Consensus workshop in which different competences and perspectives in the DID-ACT team are used to provide input to the recommendations.

Quality criteria D3.1 Course outline and online course material

  • Tailored and adaptable to the educators needs
  • Easily accessible, applicable, and understandable
  • Based on the DID-ACT curricular framework (D2.2)
  • Reviewed by all partners and additionally by associate partners

Quality criteria D3.2 Pilot implementations Train the Train Course(s)

  • > 50 participants (grant proposal)
  • Thoroughly evaluated incl. direct feedback & learning analytics
  • Covering a wide range of topics of the train-the-trainer courses that fit to the partner faculty development program
  • Piloting of at least two same learning units by 2-3 partners
  • Piloting at least to courses in an inter-/mutliprofessional setting

Quality criteria D3.3 Refinement of course(s)

  • Workshop on sharing experiences of pilots at the beginning of this deliverable
  • Considering all issues identified in the pilot implementation
  • In close cooperation with target group(s), partners, and associate partners and with repeated feedback rounds
  • Discussed and agreed upon by all partners

Quality criteria D3.4 Development of course certificate (In Review)

  • Meets the needs of the target group (refine after WP1)
  • Fullfils local and national requirements
  • Automatically created for end-user
  • DID-ACT layout

Quality criteria D4.1 Course outline and online course material

  • Tailored and adaptable to the learners and educators needs
  • Easily accessible, applicable, and understandable
  • Usage of OERs where applicable
  • Based on the DID-ACT curricular framework (D2.2)
  • Reviewed by all partners and additionally by associate partners

Quality criteria D4.2 Pilot implementations Student Course(s)

  • > 500 participants
  • Thoroughly evaluated incl. direct feedback & learning analytics
  • Running at partner institutions (except INS) + 3 other institutions
  • Covering a wide range of topics that are relevant for the partner curricula
  • Piloting of at least two same learning units by 2-3 partners
  • Piloting at least two courses in an inter-/mutliprofessional setting

Quality criteria D4.3 Refinement of course(s)

  • Workshop on sharing experiences of pilots at the beginning of this deliverable
  • Considering all issues identified in the pilot implementation
  • In close cooperation with target group(s), partners, and associate partners and with repeated feedback rounds
  • Discussed and agreed upon by all partners

Quality criteria D4.4 Plans for long-term curricular integration (In Review)

  • Each partner to provide a plan
  • Involvement of at least 3 associate partners or external stakeholders
  • Aligned with D7.3
  • Plans include description, milestones, stakeholders, potential costs

Quality Criteria for D5.1 Set of evaluation and analysis tools

  • Recommendations will be made for two separate course evaluation target groups: students and educators
  • We will select evaluation tools that measure outcomes at the reaction level (level 1) in the Kirkpatrick model
  • In addition, we will select evaluation tools that include self-assessment items to cover DID-ACT learning goals set in D2.1
  • The selection of evaluation items and tools will be informed by a literature review to promote re-use of validated items and tools
  • Data covered by the learning analytics models will be mapped against the learning goals set in D2.1 and learning methods from D2.2
  • The selected learning analytics tools should not require additional payment
  • Selected usability evaluation tool should meet standards in web ergonomics evaluation

Quality Criteria for D5.2 Evaluation of train-the-trainer course

  • Selection of evaluation tools
  • Each partner delivers an evaluation schedule
  • Collected data provided by each partner institution
  • Analysis of collected data based on state-of-the-art analytical methods
  • Report and interpretation of results reviewed and agreed by all partners

Quality Criteria for D5.3 Evaluation of pilot student curriculum

  • Selection of evaluation tools
  • Each partner delivers an evaluation schedule
  • Collected data uploaded from each partner institution
  • Analysis of collected data
  • Report and interpretation of results reviewed and agreed by all partners

Quality Criteria for D6.1 Planning and introduction into the agile project management

  • F2F part on kick-off meeting
  • Limited Support questions
  • Feedback from team members

Quality Criteria for D6.2 Quality criteria for each deliverable (In Review)

  • For declaration, please see section for each deliverable in this table
  • Defined QC fulfilled for all deliverables?

Quality Criteria for D6.3 Monitoring of the working progress

  • Half year reports on project outcomes, documenting whether deliverables were on track and deviation from project proposal
  • Report reviewed by all partners

Quality Criteria for D7.1 Social network analysis (SNA) & SM strategy

  • SNA: Includes all partners present at the kick-off meeting and the channels they are most active in
  • SNA: provides practical conclusions for the social media strategy
  • SNA: consider data privacy aspects and does not include or publish any personal not-publicly-available information
  • SM:> 200 new followers until M36
  • SM: avg. of 4 posts/month over all active channels
  • SM: followers from >200 institutions worldwide

Quality Criteria for D7.2 Website and learning management platform

  • Website: > 50K visit website hits
  • Website: > 5.000 unique visitors
  • Website: Apply rules for accessibility, usability, access for disabled persons
  • LMS: Apply rules for usability, accessibility (also for disabled persons)
  • LMS: Evaluation if ≥ 5 different LMS
  • LMS: Solution fullfils defined requirements

Quality Criteria for D7.3 Guideline and support for integrating the train-the-trainer course and the student curriculum into non-partner healthcare institutions

  • > 50 downloads in M33 – 36 (grant proposal p.52)
  • ≥ 2 external reviews from associate partners or other stakeholders
  • Reviewed and agreed by all partners

Quality Criteria for D7.4 Dissemination events

  • > 200 participants in workshops, presentations during the project
  • Evaluation of dissemination events when possible: Feedback from participant(s)?
  • ≥ 10 dissemination events / congress abstracts (national, international, non-partner countries)
  • At least 1Dissemination event per profession in an interprof. and/or nursing conferences (national / international)

Quality Criteria for D7.5 Developing Sustainability model (In Review)

  • Reviewed and agreed upon by all partners
  • Basic costs calculation for all activities
  • Defined future roles for all partners and associate partners

Quality Criteria for D8.1 Signed consortium agreements

  • Signed agreements from all partners
  • Signed amendments from all partners, when necessary

Quality Criteria for D8.2 Kick-Off Meeting

  • Kick-Off meeting in Augsburg

Quality Criteria for D8.3 Yearly Project Reports

  • Reviewed and agreed upon by all partners

Participants: Medical educators

Learning units: Person-centered approach and the role of patients
Level: Teacher

Description: Offered as part of the faculty development to faculty educators.
Mode: Part to the faculty development program with emphasize on the educational potential of clinical reasoning teaching techniques

Technical Integration: Access to the DID-ACT Moodle, synchronous sessions were held via the virtual platform NewRow.

Tips & Tricks:

Participants: Multi-professional educators (nurses, physicians, paramedics)

Learning units: What is Clinical Reasoning and Models
Level: Teacher

Description: Offered as additional faculty development opportunity to staff members of the Medical Education Department.
Mode: Option / additional part to the faculty development program with emphasize on the educational potential of clinical reasoning teaching techniques

Technical Integration: Access to the DID-ACT Moodle via EduGain, synchronous sessions were held face-to-face.

Tips & Tricks: Give participants time to familiarize themselves with the learning material between the sessions. Do not put too much in one day - better to meet twice for shorter sessions. Face-to-face sessions lead to more productive discussions than Zoom meetings. Focus on discussion with the audience and on examples from practical teaching to illustrate the learning objectives.

Participants: Multi-professional educators

Learning units: Differences and similarities in clinical reasoning among health professions
Level: Teacher

Description: Offered as part of the faculty development program, it was held as a blended learning course with the synchronous phase online.
Mode: Part of the faculty development program with certificate

Technical Integration: Access to the DID-ACT Moodle via EduGain, synchronous sessions were held online via Zoom.

Tips & Tricks: Give good practical information how Moodle works and how the participants can find and work with the different assigments of the learning unit. Emphasize the importance to the participants to work with the different individual tasks, as the discussions will be more interesting and fruitful. Also important that all the participants from different occupations feel comfortable to meet and that they all are a part of the discussion. This is an important and maybe the primary task for the facilitator!

Participants: Multi-professional and international educators

Learning units: Clinical Reasoning teaching and assessment & What is Clinical Reasoning and Models
Level: Teacher

Description: Offered as optional learning unit for participants of the Master of Medical Education (MME) program in Bern/Switzerland.
Mode: Part of the faculty development program with certificate

Technical Integration: Access to the DID-ACT Moodle via EduGain, synchronous sessions were held face-to-face.

Tips & Tricks: To foster a valuable learning experience it is indispensible to provide good case examples that are tailored to the needs and experiences of the participants, so that they can relate to their prior knowledge. Thus, the provided examples in this learning units might need some adaptations for your target group of educators.

Participants: Multi-professional educators across German-speaking coutries

Learning units: Differences and similarities in clinical reasoning among health professions
Level: Teacher

Description: Offered as part of the faculty development program at the University of Augsburg, but open to participants from Germany, Austria, and Switzerland. It was held as a blended learning course with the synchronous phase online. Participants were eager to exchange their experience and disucss their views across institutions and professions.
Mode: Part of the faculty development program with certificate

Technical Integration: Access to the DID-ACT Moodle via EduGain, synchronous sessions were held online via Zoom.

Tips & Tricks: To foster a valuable interprofessional experience the participants should represent a balanced mix of professions and also ideally the facilitators should at least represent two different professions. This allows a good discussion and also guarantees that the small groups can work interprofessionally. Our experience with a less-balanced group composition was that the over-represented profession dominates the discussions and it was quite difficult to counter-balance this.

Target group: Medical students in year 2

Learning units: Person-centered approach to clinical reasoning
Level: Novice

Description:The Learning Unit was run as an extra-curricular session as a virtual class. The facilitators were trained on this topic by attending the Train-the-trainer learning unit on "Person-centered approach and the role of patients".
Mode: Extracurricular activity

Technical Integration: Access to the DID-ACT Moodle via EduGain, synchronous sessions were held online via NewRow.

Tips & Tricks: Learning objective cross-referencing with the existing curriculum helps identify the level of integration. Encouraging and helping students to get familiar with Moodle before the session facilitates a smooth integration of the asynchronous phases.

Target group: Medical students in year 4 and 5

Learning units: Generating differential diagnoses and deciding about final diagnoses
Level: Novice

Description: The learning unit was integrated into a pediatric emergency department clerkship with a relation to virtual patients already used in this clerkship. Facilitators were trained by attending the train-the-trainer unit on "Information gathering, Generating differential diagnoses, Decision making, and Treatment planning".
Mode: Part of a regular curricular activity

Technical Integration: Access to the DID-ACT Moodle via EduGain, synchronous sessions were held face-to-face.

Tips & Tricks: Integration into clerkships works well, especially with a relation to already used content.

Target group: Medical students in year 1-6

Learning units: All 25 learning units of the DID-ACT curriculum
Level: Novice - Advanced

Description:
Longitudinal integration of the DID-ACT learning units into a clinical skills & communication course with the following suggested distribution across years:
Year 1: What is Clinical Reasoning, Person-centered approach to clinical reasoning, Health profession roles in clinical reasoning, and Biomedical Knowledge & Clinical Reasoning. (Required time in curriculum ca. 5 hours / semester)
Year 2: Dual Process Theory, Illness scripts, Collect and prioritize key clinical findings/problems, and What is clinical reasoning and How can theories be put into practice (Intermediate). (Required time in curriculum: ca. 4 1/2 hours/semester)
Year 3: Generating differential diagnoses and deciding about final diagnosis, Biases and cognitive errors - an Introduction, Analyzing and avoiding errors. (Required time in curriculum: ca. 4 1/2 hours/semester)
Year 4: Using the Outcome Present State Test Model, Developing a treatment plan, Metacognition, reflection and models for reflection, Collaboration of health professions in Clinical Reasoning (Intermediate). (Required time in curriculum: ca. 6 hours/semester)
Year 5: All remaining intermediate learning units: Shared Decision Making in Clinical Reasoning, Decision Support Systems, Ethical aspects - patient management and treatment, Uncertainty. (Required time in curriculum: ca. 5 hours/semester)
Year 6: All 6 advanced learning units: Collaborate with others in clinical reasoning, Decision Support Systems, Biases and cognitive errors, Uncertainty, Metacognition, reflection and models for reflection, Analyzing and avoiding errors.
(Required time: ca. 6 hours / semester. Final year students often have a day/week off from clinical work so these days could be used for DID-ACT learning units and optional participation in the train-the-trainer units)
Years 1-5: Virtual Patients (VPs) as additional deliberate practice activities in increasing number and complexity (e.g. starting with 5 VPs (=ca. 1.5 hours)/semester) in Year 1 and increasing to 10 VPs/semester) in Year 3-5.
Mode: Integrated into a clinical longitudinal course that runs from year 1 to 6 resulting in a total of 2.4 ECTS.

Technical Integration: Access to the DID-ACT Moodle via EduGain, synchronous sessions face-to-face or online.

Tips & Tricks:
Alining a longitudinal curriculum with curricula of other health professions remains a challenge and requires careful and early planning, but the asynchronous phases might be a good starting point for interprofessional teaching sessions. The VPs can be quite easily aligned with other curricular courses e.g., based on key symptoms.

Target group: Medical students in year 3

Learning units: Person-centered approach to clinical reasoning
Level: Novice

Description: The learning unit was integrated into the Laboratory Training of Clinical Skills. This course consists of six face-to-face meetings for groups of around 10 students. Within the meetings the students discuss various topics related to communication skills and person-centered approach. The fifth of the six meetings of the course was replaced by the DID-ACT learning unit. Students could then refer back to this learning unit during final session of the course.
Mode: Part of a regular curricular activity

Technical Integration: Access to the DID-ACT Moodle via EduGain, synchronous sessions were held face-to-face. If possible, a uniform technology to provide access to the online resources in the class (using university tablets in our case) and support of the technical staff on-site was helpful to lower the technical barrier.

Tips & Tricks: Changing of the standard format of classes is interesting and motivating for the students. Virtual patients are a tool to present authentic clinical scenarios which are appreciated by the students. A blend of role-play and virtual patients allows students to take advantages of the two methods to reach the learning objectives.

Target group: Medical students in year 2 and nursing students in year 3 across Europe

Learning units: Collaboration of Health Professions in Clinical Reasoning
Level: Intermediate

Description:
This learning unit was implemented with , aside from clinical reasoning, tow additional objectives: 1) providing internationalization experience for students without traveling. 2) Deepening the knowledge of one's own professional roles and responsibilities and knowledge of another profession .
Mode: Extra-curricular interprofessional learning session with international participants

Technical Integration: Self-registration on DID-ACT Moodle, synchronous sessions via zoom.

Tips & Tricks: Facilitators running this course and supporting the discussions should be careful about supporting and encouraging all the participating professions' perspectives.

Target group: Medical students in year 1 and 2 (preclinical) across Europe

Learning units: Introduction into Clinical Reasoning & Health profession roles in clinical reasoning
Level: Novice

Description: The course was offered as a blended learning module to students from different medical schools in Europe as an international elective. Synchronous phases were held online via Zoom. The international aspect was very motivating for students and they learnt a lot from each other by exchanging their perspectives and how they are taught clinical reasoning vaspects. This teaching mode could also be implemented as an activity to welcome or prepare new Erasmus students. Facilitairs were trained by attending the train-the-trainer learning unit on "Differences and similarities in clinical reasoning among health professions".
Mode: Elective course

Technical Integration: Self-registration on DID-ACT Moodle, synchronous sessions via zoom and use of Padlet for interactivities.

Tips & Tricks: The organization with the registration of students was a bit complex, so, we suggest just setting dates and let students book. Allow enough (more time) for discussion and introduction rounds as participants do not know each other and are eager to hear and learn from peers at other schools and countries. Ideally, this session could also be held interprofessionally, however, it makes the finding of suitable dates even more complex.

How to teach synchronously in a virtual setting

  • You need a reliable camera, microphone, and virtual platform and be familiar with its features, such as whiteboard, chat, polling, breakout rooms, etc.
  • At the beginning establish communication rules, e.g. whether participants should raise their (virtual) hand, use the chat, and/or just speak. Also, we recommend asking participants to turn on their camera
  • For small group work break out rooms work very well, just be clear about the tasks the groups should work on prior to dividing them into the groups.
  • For collaboration the use of integrated virtual whiteboards or other platforms such as Padlet are very useful. Just make sure prior to the session that you have everything setup and the links at hand, e.g. to post them in the chat.
  • Allow a bit more time for starting the session and the group works as there might be participants who are not familiar with the platform or technical problems might occur.

How to motivate unprepared participants

  • Make clear that the asynchronous assignments are a core part of the course and that its content will not be repeated. Even if it is difficult, stick to that when starting the synchronous teaching session.
  • If you expect unprepared participants, you can start the session with a student-centered group exercise mixing prepared and unprepared students to increase peer-pressure and make them realize that being unprepared does not feel good.  
  • Use the introductory or closing quizzes / tests so that participants can self- assess whether they have the required knowledge and you as a facilitator can see the level of knowledge and preparation of your participants.

Further recommended reading:

How to involve participants with different levels of experience

  • To account for such different levels, we recommend making use of the asynchronous preparatory phases which also include introductory quizzes in which participants can self-assess their prior knowledge and you as a facilitator can assess the differences within your group. Participants with less prior experience can also be guided to additional preparatory resources.
  • Encourage participants to work in pairs or small groups when preparing so that they can help and learn from each other. You could even facilitate this by dividing them into groups with different levels of experience.
  • Similarly, during the synchronous phases, we recommend forming groups with participants different levels of experience and emphasize the peer support aspects of such group activities.
  • We also recommend starting with rather smaller groups and allow more time than stated in the course outlines, if you expect a heterogenous level of experience. This way you can better manage this challenge.
  • Encourage your participants to ask questions, emphasizing that nobody knows everything and that it is important for learning to ask questions.  
  • Especially in the train-the-trainer course you might have to deal with over-confident participants, who especially in an interprofessional setting can dominate the group. This is a complex cultural challenge, but you could try to establish (and follow) communication rules at the beginning of a session.  

How to address potential overlaps or redundancies

  • Identify what is already included and what is missing in your curriculum related to clinical reasoning outcomes and compare it to the DID-ACT blueprint. Prioritize learning outcomes that are not yet covered but regarded as important.
  • Identify activities, resources, or teaching sessions with similar learning outcomes that might be in need for change anyway because of low evaluation results, teachers or students struggle with it. These could be suitable for adding or replacing parts with DID-ACT activities.
  • Ask teachers and students about overlaps and gaps they see in their teaching / learning of clinical reasoning and where they struggle. This could also be done by a reflection round after related teaching activities in the curriculum
  • Although ideally a longitudinal integration is aimed at, we recommend to starting small with a pilot implementation to gain experience and develop a show case.

How to teach in an interprofessional setting

  • Allow for enough time prior to the teaching for the organization and motivation / encouragement of stakeholders and participants
  • Allow for enough time and guidance during the course so that the participants from the different professions can get to know each other and their professions and discuss their different perspectives. This might mean that you need to calculate some extra time in addition to the suggested duration of the learning unit.
  • There may be a different understanding of clinical reasoning in the different health professions, so we recommend making participants aware of this. You could for example use and adapt activities from the learning units on the health profession roles to facilitate this.
  • Courses in an interprofessional setting should not come too early in the curriculum (not before professions have formed their own professional identity - however, this also depends on the aim of the course). 
  • Make sure you have enough participants from different professions. If possible, the facilitator could divide the participants in smaller groups with an equal distribution of professions. 
  • Similarly, you need an equal distribution of facilitators / facilitators from different professions.
  • Develop customized learning materials considering the different professions. If needed you can adapt the material and activities provided in the DID-ACT curriculum.

Further recommended reading:

van Diggele, C., Roberts, C., Burgess, A. et al. Interprofessional education: tips for design and implementation. BMC Med Educ 20, 455 (2020). (Link)

Skip to content