{"text_format":"1","workbook":"xlsx","n_sheets":3,"sheets":["Settings","KnownAnswers","README"],"not_rendered":{"charts":0,"images":0,"macros":1,"external_links":0,"defined_names":2,"comments":0,"pivot_tables":0},"unopened":[],"bounds":{"document_bytes":16777216,"sheet_cells":100000,"rows_first":2000,"rows_last":500,"rows_between":2000,"halvings":0,"cell_chars":16384,"cell_head_chars":4096},"cells_cut":0,"cells_raw":0,"read":"whole","date_system":"1900"}

{"sheet":"Settings","index":0,"n_rows":12,"n_cols":6,"rows_kept":12,"cut":false,"hidden":false,"merged":[]}
["Item","Function","Comparator","Parameter","Action","This spreadsheet has macros to help score/evaluate mechanical turk HITs that are downloaded from the Requester User Interface"]
["Answer.sel0","PctAgreement",">",0.6,"Usable","See the README tab for more instructions"]
["Answer.sel1","PctAgreement",">",0.6,"Usable"]
["Answer.sel2","PctAgreement",">",0.6,"Usable"]
["Answer.Category_id","PctAgreement",">",0.6,"Usable","Step2: adjust scoring defaults"]
["Answer.improve","Ignore",">",0.6,"Usable"]
["Answer.dd1","PctAgreement",">",0.6,"Usable"]
["KnownAnswer-Assignment","PctAgreement","<",0.8,"Reject"]
["KnownAnswer-Worker","PctAgreement","<",0.8,"None"]
["Assignment","PctAgreement",">",0.6,"Usable"]
["HIT","PctAgreement","<",0.6,"Extend"]
["Worker","PctAgreement",">",0.9,"Bonus"]

{"sheet":"KnownAnswers","index":1,"n_rows":7,"n_cols":3,"rows_kept":7,"cut":false,"hidden":false,"merged":[]}
["HitID","AnswerItem","KnownAnswer"]
["200A0RYNJ92NH3DVCXDJWUHY16A49Z","Answer.sel0","Yes"]
["200A0RYNJ92NH3DVCXDJWUHY16A49Z","Answer.sel1","No"]
["200A0RYNJ92NH3DVCXDJWUHY16A49Z","Answer.sel2","http://www.mturk.com"]
["200A0RYNJ92NH3DVCXDJWUHY16A49Z","Answer.category_id","Pet Products"]
["300A0RYNJ92NH3DVCXDJWUHY16ABCD","Answer.sel0","Yes"]
["400A0RYNJ92NH3DVCXDJWUHY168942","Answer.sel0","No"]

{"sheet":"README","index":2,"n_rows":72,"n_cols":8,"rows_kept":72,"cut":false,"hidden":false,"merged":[]}
["How this spreadsheet works:"]
["this excel file provides mechanical turk requesters helpful macros to determine the usefulness of responses to their HITs"]
["it is not an official mTurk software release, and is not supported as such. The idea is for Requesters to start with this & customize it for their purposes. Let us know if you find it useful or have feedback."]
["macros need to be enabled for this excel file (via security setting), else it will not work. It will be very helpful if you are excel/macro savvy"]
["responses to HITs are read from a CSV/XLS/Tab delimited file - usually generated by the mTurk Requester User Interface, or the Command Line tools (or maybe via API)"]
["the Input CSV file format is simple - it should have:"]
["a header row with the name of the columns (seperated by commas, of course)"]
["and 1 row per assignment (like batch results files from the Requester User Interface)"]
["each row in the CSV file should have the same columns, in the same order"]
["you can keep multiple CSV files for different HITSets as needed. Each analysis run will look at only 1 file"]
["required columns in the CSV include HITId, AssignmentID, AssignmentStatus, WorkerID, plus one column per answer in the assignment"]
["additional columns can be in the spreadsheet, but are ignored"]
["an answer column is identified by its naming convention - the column name must begin with \"Answer.\" - e.g. \"Answer.Q1CategorizeURLs\""]
["if the word \"Comment\" is part of the Answer Column name, that answer is ignored in the analysis"]
["once you select a file, the settings tab contains settings for the analysis to follow - change those settings as appropriate for your case"]
["clicking on 'Evaluate' performs the analysis & provides output in the HITResults, AssignmentResults & WorkerResults tabs"]
["Nothing is communicated with the mTurk servers. This analysis runs within Excel only, and dumps the output to a file."]
["this can be a memory hog…if your batches get large enough, it will run out of memory and cause an error. It's been tested to about 40 megs of data in the CSV"]
[]
["Explanation of settings:"]
["Item","Potential Actions",null,null,"Potential Functions",null,null,"Comments"]
["Answer.*","Usable","NonUsable",null,"Ignore","PctAgreement","NumAgreement","this setting is for each answer. It marks an answer as Usable/Nonusable depending on whether there is a certain % level of agreement (or a certain number of agreements) amongst all the provided answers for all assignments within a HIT."]
["Assignment","Approve","Reject",null,"Ignore","PctAgreement",null,"this setting is for the assignment level, and marks an assignment for approve/reject based on whether there is a certain % level of agreement across all answers within the assignment"]
["KnownAnswer-Assignment","Approve","Reject",null,"Ignore","PctAgreement",null,"this setting is for the assignment level, and marks an assignment for approve/reject based on whether there is a certain % level of agreement for all known answers within a given assignment"]
["KnownAnswer-Worker","Bonus","Block",null,"Ignore","PctAgreement",null,"this setting is for the worker level, and marks a worker for bonus/block based on whether the worker provided a certain % level of correct known answers across all the assignments for that worker (that are in the spreadsheet)"]
["HIT","Extend","Expire","Audit","Ignore","PctAgreement","NumAgreement","this setting is for the HIT level, and marks the HIT for extend/expire/audit based on whether there is a certain % level of agreement (or certain number of agreements) amongst all assignments within the HIT"]
["Worker","Bonus","Block",null,"Ignore","PctAgreement",null,"this setting is for the worker level, and marks a worker for bonus/block based on whether the worker provided a certain % level of Usable answers, across all the assignments for that worker (that are in the spreadsheet)"]
[]
["Providing Known Answers"]
["These are provided in the KnownAnswers tab, using 3 columns. The 1st column should be named the same as the CSV header column as you are using to tag a known answer. E.g. HITId"]
["The 2nd & 3rd columns are named AnswerItem & Known Answer. They should contain the specific answer within the HIT & the value for that answer that corresponds to a known answer"]
[]
[]
["Config Actions","Config Functions","ConfigComparator"]
["Usable","PctAgreement",">"]
["NonUsable","NumAgreement","<"]
["Approve","Ignore","="]
["Reject",null,">="]
["Bonus",null,"<="]
["Block"]
["Extend"]
["Expire"]
["Audit"]
[]
[]
["Sample Settings"]
["Item","Function","Comparator","Parameter","Action"]
["Answer.sel0","PctAgreement",">",0.6,"Usable"]
["Answer.sel1","PctAgreement",">",0.6,"Usable"]
["Answer.sel2","PctAgreement",">",0.6,"Usable"]
["Answer.Category_id","PctAgreement",">",0.6,"Usable"]
["Answer.improve","Ignore",">",0.6,"Usable"]
["Answer.dd1","PctAgreement",">",0.6,"Usable"]
["KnownAnswer-Assignment","PctAgreement","<",0.8,"Reject"]
["KnownAnswer-Worker","PctAgreement","<",0.8,"None"]
["Assignment","PctAgreement",">",0.6,"Usable"]
["HIT","PctAgreement","<",0.6,"Extend"]
["Worker","PctAgreement",">",0.9,"Bonus"]
[]
["Sample Known Answer Settings"]
["HitID","AnswerItem","KnownAnswer"]
["200A0RYNJ92NH3DVCXDJWUHY16A49Z","Answer.sel0","Yes"]
["200A0RYNJ92NH3DVCXDJWUHY16A49Z","Answer.sel1","No"]
["200A0RYNJ92NH3DVCXDJWUHY16A49Z","Answer.sel2","http://www.mturk.com"]
["200A0RYNJ92NH3DVCXDJWUHY16A49Z","Answer.category_id","Pet Products"]
["300A0RYNJ92NH3DVCXDJWUHY16ABCD","Answer.sel0","Yes"]
["400A0RYNJ92NH3DVCXDJWUHY168942","Answer.sel0","No"]
[]
["Sample Known Answer Settings"]
["Input.Website","AnswerItem","KnownAnswer"]
["http://www.mturk.com","Answer.category_id","On Demand Workforce"]
["http://aws.amazon.com","Answer.category_id","Cloud Computing"]