Auto-Tagging for Massive Online Selection Tests: Machine Learning to the Rescue

Abstract

Difficulty Level of a question is relative to that of other questions in a test and also to the test takers, hence manually assigning Difficulty Level tags may not be accurate. There is a need to infer them from historical data pertaining to the performance of students in a test. e-Yantra Robotics Competition (eYRC) is an annual competition having around 5000 teams (20,000 students) registering in the latest edition of the competition, eYRC-2015. All four team members take a test simultaneously and each individual gets questions which are different but have a similar Difficulty Level. A Question Bank containing 1800 unique questions from 3 subjects - Aptitude, Electronics, and C-Programming - is used to generate question sets each having 30 questions. It is a challenge to ensure that each set contains questions of similar Difficulty Levels tagged manually as Easy, Medium or Hard. In this paper, we discuss a learning algorithm called Weighted Clustering that can automatically tag questions by analyzing the performance of students. We used this algorithm to analyze the performance data in eYRC-2014 for 614 questions from the Question Bank, we found that Manual Tagging accuracy was 44%. We retagged questions with Suggested Tags resulting from our analysis and used them again in eYRC-2015. When we applied the algorithm to the performance data in eYRC-2015, we found that the accuracy of tagging had significantly improved to 67%.