Friday, August 14, 2015

Reverse Engineering the Raw Results of a Multiple Answer Question

I had occasion recently to obtain assessment results out of a recovered database.  One question type that was a bit puzzling was the Multiple Answer where the Answers were randomized to the student.

Say I created a question:

When the student took the test, the answers were randomized, e.g.,


When I extracted the results, I got a data string like


Useful bits:
  • answer_order: 0, 3, 4, 1, 2 (the order answers were displayed to the student); note: the ordinal position is 0.
  • sequence of response_values (the actual selections)
  • score_value: 0.0 (what the student auto-scored)
Question: 

Does the sequence of response_values corresponded to the original order of answers in the question, or the order of answers displayed to the student?

Answer:

It is corresponds to the original order of answers in the question.

Deeper...

The question had five answers
  • 3 - position 0
  • 4 - position 1
  • 5 - position 2
  • 6 - position 3
  • 9 - position 4
The question was displayed to the student
  • a. 3 = position 0
  • b. 6 = position 3
  • c. 9 = position 4
  • d. 4 = position 1
  • e. 5 = position 2
That corresponds to the answer order


The student selected a. 3, b. 6, c. 9

That corresponds to the sequence




and so on.

Finally, the student received a score of 0 (in my example, they had to get all the multiple answers correct, no partial credit)