Quick instructions for now -- more details later.
- Make sure there is money in the M-Turk account (for the moment this is Matt's job -- so check with him on this).
Log in to http://requester.mturk.com with the lab Gmail address and password -- if you are doing this activity, you should know what those are.
- Click the 'Create' tab -- we will assume for now that you have already set up the Project in M-Turk and that you have double, triple, etc., checked that everything is correct.
- If you need to change the batch size (number of subjects), you will need to click the 'Edit' button for the Project in question and change the "Number of assignments per HIT" value on the first screen, then save the changes by hitting the 'Save' button at the bottom of the page.
- Now, back in the 'Create' tab (if you left it), click the 'Publish Batch' button.
- On the first screen, just verify that everything looks OK -- there shouldn't be anything to change if the HIT was set up correctly in the first place. After checking, click the 'Next' button.
- On the 'Confirm and Publish Batch' screen, the only thing you should need to change is the "Batch Name" field. We typically just use incrementing numbers for batch names (within a given project) -- so if it's the first batch for that project, use 'Batch01', the next batch is 'Batch02', etc.
- Double-check that the number of assignments and the money looks right -- there should be no more than 9 assignments (or else we get charged extra). Most of our studies, for now, should be at the $1.50 payment level, which with M-Turk's 20% fee should cost $1.80 per assignment.
- If all of that looks OK, click the 'Publish HITs' button.
- Now M-Turkers can see the assignment and start working on the task. You can now go to the 'Manage' tab on the M-Turk interface to check their progress. In the progress bar on the main 'Manage' page, the percent of assignments published (accepted by a user) should almost immediately go to 100%. Then the percentage for 'submitted' should gradually creep up as people complete the task.
- Now we play the waiting game. Hopefully, all the assignments should be submitted within a few hours, although occasionally people will timeout on the assignment and it will get re-assigned, which will take longer. Keep an eye on the lab Gmail for any issues (although in theory, right now most of them should go to Matt, that will change in the future -- and there might be a way for them to email the lab Gmail from within the M-Turk interface, so we should monitor both for anyone having technical issues).
When all the assignments are submitted, we need to log into the web server to check the logs and make sure they did their job correctly so we can pay them. (You can also do this just to check on how people are doing.) Generally, we should try to pay people within about 24 hours of first publishing the batch, so they stay happy and give us good reviews on M-Turker forum sites. Here's how to do that:
Log into the web server (aka Zoidberg) via SSH. (On the Mac command line: 'ssh myusername@matthewrjohnson.net' or 'ssh myusername@mturkstudi.es' -- both get you to the same place.) If it is your first time logging in, it may ask you to confirm the public cryptographic key -- just type 'yes' and hit Return. Then type your password when prompted, hit Return, and you should be in.
Change directory ('cd' command) to the data folder for M-Turk studies, which is /var/www/mturkstudi.es/data/active for the currently active study. If you list the files there ('ls' command), you should see a bunch of trial logs, a completed workers log, and a demographics log.
If you want a quick summary of how people did, you can do a 'wc *.txt' to get a "word count" of all those text files. The output will have a listing of all the text files in the directory, with the first few columns showing the number of lines, words, and characters in each file. For most of our current studies, we're requiring that people complete 50 trials correctly, which means that a worker who completed the task and did pretty well should have somewhere between 50-70 lines in their file (which would represent the 50 correct trials plus 0-20 error trials). People who bailed out of the task before finishing might have <50 lines, and should not show up in the completed worker log. People who finished but had a lot of errors might have >70 lines in their files.
- In order to actually pay people, you need to inspect the completed worker log. The 'cat' command (spits a text file out onto the command line) makes this pretty easy. So you can just do 'cat nameofstudy_completed_worker_log.txt' to see its contents. You should see M-Turk IDs and alphanumeric codes for everyone who finished the task, in the order that they finished.
- Now you need to compare those codes/IDs to what people put into M-Turk. So back in the 'Manage' tab of the M-Turk requester site, click the 'Results' button next to the batch that just finished. (Typically we should wait to pay everyone until the whole batch is finished running, which shouldn't take more than 24 hours unless something weird happened.)
- You should see a list of HITs with worker IDs and confirmation codes. Check to make sure the worker IDs and confirmation codes match up with what's in the completed worker log on the server. Usually everyone does their job right and you can check the box next to each assignment and click the 'Approve' button to pay everyone. If someone clearly fudged their confirmation code and didn't actually complete the task, you can check the box next to their name and click 'Reject'. Sometimes they'll screw something up and not enter a confirmation code, or enter the wrong thing (like their M-Turk ID or whatever was on their clipboard at the time) by accident -- in that case, don't do anything and check in with Matt for next steps.
- If everything went hunky-dory, though, and you were able to approve everyone's HITs without issue -- hurray, you're done! Now you can go ahead and start another batch, if you still want to collect more data, or get to work on the next experiment if we're all done with this one.