# Partial ordering

Survey questions have a maximum number of answer choices that can be displayed at one time to a respondent; five answers for multiple choice questions and two answers for image choice questions. When the number of answers choices for a question exceeds the number that can be displayed at one time, a subset of the answer choices will be shown.

The subset of answer choices will always contain the maximum number of answer choices that can be displayed.

The subset of answer choices presented are generated randomly without regard to the number of times a particular answer has been shown or to the answer combinations that were previously generated.

Survey results are weighted using the techniques described in “A Penalized Maximum Likelihood Approach for the Ranking of College Football Teams Independent of Victory Margins” by David Meese. The methodology is the same one used by the NCAA to determine the rank of teams in tournaments. Answer choices are weighted solely on its wins and losses against competing answer choices. This methodology compensates for answer bias that may occur from the randomly generated answer subsets. This bias grows smaller as the number of responses increase. The weighting can be turned off in options to show the raw scores.

Please note that we envision that customers use partial ordering to glean the top 5 answers from a list of 5+ answer choices. Once you know what the top 5 choices are, you can run a follow-up survey with just those top 5 choices, and see which answer choice is statistically the winner.