Sorry, you have not enough credits to pay results fee.
Present an approach. Name the field(s) of math most appropriate to the problem
Labels
Description
We have a pool of thousands of True/False questions. Players are presented 10 questions per session. Order of presentation is adaptive, so if you get a question correct, the next question is harder. Or incorrect-> easier.
I'd like to regularly rank all the items by difficulty, based on actual performance during play. We are currently using a simple "percent correct" on an interim basis, but that seems problematic because of the adaptive algorithm and other reasons.
One lay person idea I have is to log each event pair where in the same session a given player gets one item incorrect and a different item correct. Intuitively, it seems that I could use that history of differentials as a more accurate ranking of the item pool by difficulty.
But that's just one idea. The problem is to create a practical, defensible system that allows us to state with confidence and credibility the order of difficulty of the thousands of items in the item pool.
It's important that the approach be defensible to an expert audience, but only in a practical way. Experts need to find the ranking persuasive, because they want to rely on the rankings. But it doesn't need to be a bulletproof academic or theoretical defense.Of course it's not logically consistent, in that one player may get a right, b wrong; another b right c wrong; and another c right, a wrong.
We are constantly adding new items, so we need a way to handle that, perhaps by having a confidence level for each item?
We also tag items by subtype. I'm guessing I'll be able to use the same approach for ranking items within a subtype, but please address that question as well.
Sometimes a player might just fool around, for instance randomly choosing answers to see how the game works. It would be nice to be able to ignore sessions like that.
I'd like to know what field of mathematics deals with this. I thought it was psychometrics, but from a quick layman's look, that doesn't seem quite right.
I'd like a brief description of what you feel is the best approach to the problem.
Thank you.
Answers (1)
-
Oct 26, 2011Your task of finding a ranking based on performance of players can be approached from different fields of Mathematics, most of which will fall into (applied) stochastics or probability theory. The first question is chosen randomly, right? This means that if you have sufficient statistics on the first question only, you can make your ranking based on that. You don't need any tools from probability theory in this case. Otherwise you have to rely on them as you face the situation where you have some samples here and there, but many sets are not representative.
a) For something rather unexpected: You can use a ranking system for questions as it is used ... in chess or tennis for ranking players. Such systems work in practice, and it will definitely earn you a lot of respect and credibility.
b) Bayesian parameter estimation. This is about conditional probabilities: What is the chance that a player will answer A correctly if he already answered B correctly? Apparently, you have some info on these probabilities from player's behaviour. Now, a model must be developed which would predict this conditional probability based on the ranking of A and B. This will lead to a system of equations, where the rankings are unknowns.
c) Markov chains. This is about random walks: A full "game" is considered as a random walk between questions. The ranking comes in at two places: as the probability of going from A to B, and as probability of actually being in A.
d) Machine learning algorithms. Several machine learning algorithms (neuronal networks, support vector machines just to name some) can be trained to predict to some certainty whether the player will answer question 6 correctly based on his result on the first 5 questions. A ranking is computed implicitly, and can be distilled from those algorithms.
Dividing questions in subcategories should not change much. If your next-question-algorithm selects question from the same subcategory, then you will automatically have good statistics within it. So, the relative ranking may be better than the absolute one. If you decide to use a ranking similar to the one for chess players, there will be the following analogy. Since juniors play a lot against each other, their ranking relative to each other reflects typically very well their actual strength.
Finally, finding players that just fool around is strongly related to your algorithm for defining the next question. Random answers can be filtered out automatically using different machine learning algorithms.
No comments yet.
Keep your profile up to date. The launcher will look at your qualifications, experience and feedback.
Read the zomb description carefully. Ensure you answer any queries about your skills or experience
Take a look at what other members are bidding. If you're going to place a bid amount that's higher than other members, ensure you are the best candidate!