Quarterly Questions, the exam alternative program that will replace the DOCK beginning in 2019, uses a longitudinal assessment model where a pass/fail decision is made on diplomate performance over time. This “longitudinal assessment” method is unlike most of the other assessment formats that ophthalmologists have been exposed to in the past. Instead of choosing a single test date, sitting at a computer, and answering 100 or more questions in a single sitting, Quarterly Questions allows diplomates to engage with questions at their convenience. While it is possible to complete many questions in one sitting, it is also possible to complete just one or two, if desired. Quarterly Questions also blends assessment and learning with the use of both timed, knowledge-based questions, as well as questions based on research articles in ophthalmology. The purpose of this post is to explain why Quarterly Questions was designed in this way and how it serves as both an assessment tool as well as an educational tool.
Quarterly Questions has a performance standard, which means simply participating is not sufficient for receiving credit for the activity. Each year, a passing standard (cut score) is set for the questions that will be delivered in that calendar year. For example, in 2018 the passing standard was 60%, which means answering 60% of the knowledge-based questions correctly is required for successful completion. The reason this passing standard may be slightly different from year to year is that the difficulty of the questions may change. Although not often known to the examinees, setting different performance standards to account for differences in difficulty (in order to keep the performance standard constant over time) is used for almost all examinations. To earn a “pass” for 2018, an individual must answer at least 60% of the 40 knowledge-based questions correctly, and also answer 10 article-based questions. The article-based questions do not have a performance standard (i.e., a minimum percentage correct that must be obtained for credit).
Knowledge-based and article-based questions are both part of Quarterly Questions because they get at different competencies: the knowledge-based questions are focused on knowledge that a diplomate should know without the aid of resources, while the article-based serve to determine whether a diplomate has correctly understood a journal article. Together, these questions allow a diplomate to make a real-time self-assessment (allowing reflection on his/her strengths and weaknesses) while learning new content simultaneously.
The knowledge-based questions are used to determine whether a diplomate has been keeping up with information in the field. The content for the knowledge-based questions is based upon an outline developed by several committees of volunteer ophthalmologists in active private and academic practices. Their goal in the development of the content outline (or “blueprint”) for the knowledge-based questions was to include the content that is important to know by the majority of ophthalmologists in that subspecialty area. The five core questions delivered in quarters 1 and 2 are aimed at assessing knowledge that every ophthalmologist, regardless of subspecialty area, should know. The value in the knowledge-based questions for the diplomate is the act of attempting the question itself as well as the instant feedback (the critique and key point). Research shows that immediate feedback aids in retention of material better than simply taking a test in a single sitting and getting a score report several weeks after the fact. The diplomate is also prompted to indicate his/her confidence in the answer choice selected. This allows additional self-assessment (an incorrect answer that a diplomate was confident he/she had answered correctly might prompt more reflection than a correct answer that the diplomate was confident he/she had answered correctly).
The article-based activity is meant to encourage diplomates to engage in learning by reading important peer-reviewed studies. The value in this activity is the reading itself, while the questions serve as a “check” on the comprehension of the article. This activity was largely designed based upon diplomate input as to what activities help them to learn and grow their knowledge. Again, committees of volunteer ophthalmologists review and select “must read” journal articles. The wide variety of articles offered allows the diplomate to choose which studies are most important to read given his/her area of practice.
It is recommended that diplomates engage frequently with Quarterly Questions without taking long periods of time off between attempts. Research shows that more frequent engagement in assessment questions and review of performance on those questions leads to better long-term retention (this is called the “testing effect”). All elements of Quarterly Questions: the knowledge-based questions, the peer-reviewed article activity, as well as the ability to engage in the assessment frequently were designed with the diplomate’s experience and long-term retention in mind.
For more information on the testing effect and self-assessment, I recommend the following research articles:
Eva, K., & Regehr, G. (2005). Self-assessment in the health professions: A reformulation and research agenda. Academic Medicine, 80(10), S46-S54. doi:10.1097/00001888-200510001-00015
Larsen, D. P., Butler, A. C., & Roediger, H. L. (2008). Test-enhanced learning in medical education. Medical Education, 42(10), 959-966. doi:10.1111/j.1365-2923.2008.03124.x