Guide to: Test Question Creation Interface

On the Quality Tab, after you have clicked "Create", you will see the following page.

• A. Show Job Instructions - displays the job instructions within the Test Question
• B. Skip Question - skip rows that do not fit Test Questions best practices
• C. Hide Answer - hide answer allows you to hide the Test Question answer from the contributor, they will not be aware that the question is a Test Question, see Secret Test Questions, below
• D. Save & Create Another - save the row as a Test Question and continue with creating Test Questions.
• E. Mode - select if you want the test question to appear in 'Quiz Mode', Work Mode' or 'Both', to use this option in the UI you'll need to have it enabled, however there is an alternative way to set the Mode using the test questions report, see Test Question Enabling and Quiz Mode and Guide To: Designating Test Questions for Quiz or Work Mode with Gold Pool
• F. Test Question Reason Textbox - enter your reason for selecting the answer here, this box will automatically appear once you have chosen an answer for the question
• G.View Unit Page

View Unit Page

This page will present you with more detailed data concerning the row selected.

Figure. 4: Important features within ‘Show Unit Page’

A. Show Data

Test Question and Source Data in JSON format.

Figure 5. The JSON format of the selected Test Question

B. Show Statistics

A specific CML question can be selected to display the answer distribution associated with it.

Figure 6. The answer distribution across all judgements for the CML Question ‘sentiment’

C. Show Contributor Info

• Worker ID
• Trust Score
• If their judgment was correct
• If their judgment was tainted
• Name of the channel the worker belongs to
• If they contested the incorrect judgement
• Each answer given by the contributor, defined by the CML name
• Each answer given by the contributor, defined by the CML name

D. Edit Test Question

Brings you back to the Test Question editing interface

Secret Test Question Feature

The primary function of Test Questions is to test the quality of contributor responses to pre-annotated data and provide feedback to them when they are annotated as incorrect, to that end Test Questions are meant to be indistinguishable from the dataset. Ideally Test Questions should not be reused across multiple jobs because long term contributors may eventually learn the specific answers they get wrong. If Test Questions will need to be reused, however, it can be useful to reserve a subset of them for which the correct answers are not revealed.

The Secret Test Question feature allows you to hide the answer to certain Test Questions in your job. After creating Test Questions, toggle the "Hide Answer" option on the Quality page for the Test Questions that you would like to hide.

Figure 7: Hide Answer on Quality page

It is recommended that Secret Test Questions are used in conjunction with the gold pool functionality. Secret Test Questions cannot be used in quiz mode to qualify contributors into the job, thus if you hide Test Questions that are set to quiz and have less than 5 Test Questions allocated to quiz, quiz mode will not work. As Secret Test Questions are only meant for Work, it is recommended that they are used in addition to regular work Test Questions. It's crucial to note that Secret Test Questions do not impact contributors' accuracy in the task, nor do they trigger any flags, rejections, or affect contributors' overall performance during the job.

To assess and calculate the accuracy of contributors on Secret Test Questions, manual review via the full report is necessary at this time. To check the rows with Secret Test Questions, see Figure 8 below where column _answers_hidden equals TRUE.

Figure 8: Full Report featuring field _answers_hidden

Hide Test Question Answers

You can choose to hide all of the Test Question answers from contributors by clicking on the toggle button on the Quality page.

Figure 9: Hide All test questions answers on the Quality Page