On the Quality Tab, after you have clicked "Create", you will see the following page.
This page will present you with more detailed data concerning the row selected.
Figure. 4: Important features within ‘Show Unit Page’
A. Show Data
Test Question and Source Data in JSON format.
Figure 5. The JSON format of the selected Test Question
B. Show Statistics
A specific CML question can be selected to display the answer distribution associated with it.
Figure 6. The answer distribution across all judgements for the CML Question ‘sentiment’
C. Show Contributor Info
D. Edit Test Question
Brings you back to the Test Question editing interface
The primary function of Test Questions is to test the quality of contributor responses to pre-annotated data and provide feedback to them when they are annotated as incorrect, to that end Test Questions are meant to be indistinguishable from the dataset. Ideally Test Questions should not be reused across multiple jobs because long term contributors may eventually learn the specific answers they get wrong. If Test Questions will need to be reused, however, it can be useful to reserve a subset of them for which the correct answers are not revealed.
The Secret Test Question feature allows you to hide the answer to certain Test Questions in your job. After creating Test Questions, toggle the "Hide Answer" option on the Quality page for the Test Questions that you would like to hide.
Figure 7: Hide Answer on Quality page
It is recommended that Secret Test Questions are used in conjunction with the gold pool functionality. Secret Test Questions cannot be used in quiz mode to qualify contributors into the job, thus if you hide Test Questions that are set to quiz and have less than 5 Test Questions allocated to quiz, quiz mode will not work. As Secret Test Questions are only meant for Work, it is recommended that they are used in addition to regular work Test Questions. It's crucial to note that Secret Test Questions do not impact contributors' accuracy in the task, nor do they trigger any flags, rejections, or affect contributors' overall performance during the job.
To assess and calculate the accuracy of contributors on Secret Test Questions, manual review via the full report is necessary at this time. To check the rows with Secret Test Questions, see Figure 8 below where column _answers_hidden equals TRUE.
Figure 8: Full Report featuring field _answers_hidden
You can choose to hide all of the Test Question answers from contributors by clicking on the toggle button on the Quality page.
Figure 9: Hide All test questions answers on the Quality Page