{"text_format":"3","workbook":"xlsx","n_sheets":7,"sheets":["Instructions","Data","Codebook","RelevantImages","ChangeLog","Dropdowns","For developers"],"not_rendered":{"macros":1,"defined_names":108},"read":"whole","date_system":"1900"}

{"sheet":"Instructions","index":0,"n_rows":31,"n_cols":3,"rows_kept":31}
[]
[null,"INSTRUCTIONS"]
[]
[null,"This is the Excel spreadsheet version of the EMMIE coding tool, there are several components to this tool, and these have been separated using thick borders.  You enter the data on the 'Data' tab."]
[]
[null,"Please note that for every dimension of EMMIE, there are 'evidence' (-E) codes and 'quality' (-Q) codes.  The former refers to evidence (data) extraction and comes first.  Quality codes refer to evidence appraisal and follow for each dimension."]
[]
[null,"To access a simple explanation for a  variable click on the heading (e.g. in cell 'A3' Codername).  To access the detailed explanation of a variable in the codebook click on the hyperlink immediately below the heading."]
[null,"You will need to click back on the 'Data' tab to go back to where you were working after viewing the codebook.  It should direct you back to the cell you were working from."]
[]
[null,"Some cells have been formatted as a drop-down menu to limit the answers you can enter.  Other cells have been formatted so that you may only enter integers (whole numbers) or decimals."]
[null,"When entering free text please include an asterix (*) at the beginning of each new concept."]
[]
[null,null,"Cells shaded like this will automatically populate, based on user input codes"]
[]
[null,"Coders should ALWAYS attempt to provide an answer to an EMMIE-Q code.  If there is uncertainty over an answer, then please highlight the cell in question for the moderator to check at a later date."]
[null,"If the automated -Q scores return a NULL value or you disagree with the score then you can enter the score manually by typing over the formulae in that cell.  Please discuss this in the moderation session."]
[null,"Please note, the -E codes will be used to populate the Evidence toolkit.  Please ensure that the information you insert into these codes are intelligable and useful for the end-users of the toolkit."]
[null,"If critical information is provided secondhand, via other sources, then please include this in -E codes (even if it is not a true fit with the code label).  Ensure that it is clear that the information is from another source (cite if appropriate)."]
[null,"A 'notes' row is available in row 13 - this is where you can enter page numbers where the information was found in a paper and any coding decisions made."]
[]
[null,"Start entering data in row 5."]
[]
[null,"Only the first 8 rows (after the examples) are formatted in the way described above.  This is to discourage multiple reviews from being coded in the same spreadsheet whilst retaining the ability to record notes."]
[null,"Once coded with ONE review the spreadsheet should be saved.  The naming convention for this is coders initials, hypen, short paper reference. E.g. \"LT-Example(2006).xlsx\""]
[]
[]
[]
["© ","This EMMIE coding tool was developed  at the Department of Security and Crime Science, UCL by: Lisa Tompson, Kate Bowers, Shane Johnson, Jyoti Belur"]
[null,"Crime Solutions (www.crimesolutions.gov) are acknowledged for providing many of the EMMIE effect codes."]
[null,"This spreadsheet is maintained by Lisa Tompson.  It was last updated on 23 March 2018.  If you experience any problems with the coding or have suggestions for ways we can refine the coding tool then please email l.tompson@ucl.ac.uk "]

{"sheet":"Data","index":1,"n_rows":13,"n_cols":139,"rows_kept":13,"merged":[[0,0,0,1],[0,77,0,91],[0,96,0,100],[0,123,0,137]]}
["INTERNAL REFERENCE",{"$nulls":19},"EFFECT EVIDENCE CODES",{"$nulls":16},"EFFECT QUALITY CODES",{"$nulls":15},"MECHANISM EVIDENCE CODES",{"$nulls":3},"MECHANISM QUALITY CODES",{"$nulls":14},"MODERATOR (CONTEXT) EVIDENCE CODES",{"$nulls":4},"MODERATOR (CONTEXT) QUALITY CODES",{"$nulls":15},"IMPLEMENTATION EVIDENCE CODES",null,null,"IMPLEMENTATION QUALITY CODES",{"$nulls":15},"ECONOMICS EVIDENCE CODES",{"$nulls":10},"ECONOMICS QUALITY CODES",{"$nulls":14},"INTERNAL CODE"]
[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,null,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,null,52,53,54,55,56,57,58,59,60,61,{"$nulls":7},62,null,63,64,65,66,67,68,69,70,71,72,73,74,{"$nulls":7},75,null,76,77,78,79,80,81,82,83,{"$nulls":9},84,85,86,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,{"$nulls":7},102,103]
["Codername","Reference","Intervention description","OffTreatment","Offenderfocus","Victimfocus","Placefocus","TargetedOffpop","TargetedNonpop","TargetedPopSub","CrimeOutcomes","NonCrimeOutcomes","OutcomeDataType","OutcomeProperty","OutcomeViolent","OutcomeCriminality","OutcomeDetection","OutcomeAntisocial","NoStudies","Evidence-E","EFFECT-E","EFFECTTYPE-E","EFFLOWER-E","EFFUPPER-E","EFFSE-E","EFFNSTUDIES-E","EFFNCONTROL-E","EFFCONTGRP-E","EFFNACTION-E","EFFMOD-E","EFFSTAT-E","Otherbias-E","Effectinwords","EffectInterpretation1","EffectInterpretation2","Meta-analysis","EffectSizeCalculations","EFFSearchstratALL-Q","Effectsizeapp-Q","EFFheterogeneity-Q","EFFRandomodel-Q","EFFdependency-Q","EFFweighting-Q","EFFStatconvalALL-Q","EFFpubbias-Q","EFFinterater-Q","EFFoutliers-Q","EFFriskbiasALL-Q","EFFConstvalALL-Q","EFFStudesignALL-Q","EFFunantioutALL-Q","EffectQSCORE","Mechamism-E","MECHLITSOURC-E","MECHMAP-E","MECHMOD-E","MECHSTAT-E","MECHMENT-Q","MECHLIT-Q","MECHPRAC-Q","MECHFORM-Q","MECHCOL-Q","MECHCON-Q","MECHMENT= Yes","MECHLIT &/ MECHPRAC=Yes","MECHFORM =Yes-BASIC","MECHFORM =Yes-THOROUGH","MECHCOL = Yes - BASIC","MECHCOL = Yes - THOROUGH","MECHCON =Yes","MechanismQSCORE","Moderator-E","MODLITSOURC-E","MODMOD-E","MOMMODPH-E","MODSTAT-E","MODEFF-E","MODMENT-Q","MODLIT-Q","MODPRAC-Q","MODFORM-Q","MODCOL-Q","MODSUB-Q","MODSUB-Q(APRIOR)","MODMENT=Yes","MODLIT=Yes-Basic","MODLIT=Yes-THOROUGH","MODPRAC=Yes","MODFORM=Yes","MODCOL or MODSUB=Yes","MODSUB (APRIOR)=Yes","ModeratorQSCORE","Implementation-E","IMPLITSOURC-E","IMPNECIMP-E","IMPNECREP-E","IMPMENT-Q","IMPDES-Q","IMPOBST-Q","IMPCRSUCC-Q","IMPREP-Q","IMPMENT=Yes","IMPDES= Yes-Basic","IMPDES= Yes-Thorough","IMPOBST= Yes-Basic","IMPOBST= Yes-Thorough","IMPCRSUCC= Yes-Basic","IMPCRSUCC= Yes-Thorough","IMPREP= Yes-Basic","IMPREP= Yes-Thorough","ImplementationQScore","Economics-E","ECOINPUTS-E","ECOOUTPUTS-E","ECOINTENS-E","ECOdirectCOSTIMP-E","ECOCOSTIMPSUB-E","ECOindirectCOST-E","ECOCOSTEFF-E","ECOCOSTEFSUB-E","ECONCOSTSOURCE-E","ECOCOSTBEN-E","ECOCOSTBENSUB-E","ECONdirectSETUP-Q","ECONindirectSETUP-Q","ECONINTERVPEROUT-Q","ECODIRCOSTBENEFIT-Q","ECONCOSTPEROUTCOME-Q","ECOINDIRCOSTBENE-Q","ECOINDIRCBSTAKE-Q","88=Yes","89=Yes","90=Yes","91=Yes","92=Yes","93=Yes","94=Yes","EconomicsQSCORE","CODERSNOTE"]
[{"$f":"HYPERLINK(\"#'Codebook'!\"&CELL(\"address\"),\"Codebook\")","v":"Codebook"},"Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook",null,"Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook",null,"Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook",{"$nulls":7},"Codebook",null,"Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook",{"$nulls":7},"Codebook",null,"Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook",{"$nulls":9},"Codebook",null,"Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook","Codebook",{"$nulls":7},"Codebook"]
[{"$nulls":19},{"$f":"IF(COUNTIF(U5:AE5,\"\")=0, \"Yes\", \"No\")","v":"No"},{"$nulls":23},{"$f":"IF(COUNTIF(AL5:AQ5,\"\")=COLUMNS(AL5:AQ5),\"\",IF(COUNTIF(AM5:AQ5,\"Yes\")<4,\"No\",\"Yes\"))","v":""},{"$nulls":3},{"$f":"IF(COUNTIF(AS5:AU5,\"\")=COLUMNS(AS5:AU5),\"\",IF(COUNTIF(AS5:AU5,\"Yes\")<2,\"No\",\"Yes\"))","v":""},{"$nulls":3},{"$f":"IF(COUNTIF(AV5:AY5,\"\")=COLUMNS(AV5:AY5),\"\",IF((COUNTIF(AV5:AY5,\"Yes\")+COUNTIF(AR5,\"Yes\")+COUNTIF(AL5,\"Yes\"))>=5,Dropdowns!$R$6,IF((COUNTIF(AV5:AY5,\"Yes\")+COUNTIF(AR5,\"Yes\")+COUNTIF(AL5,\"Yes\"))=4,Dropdowns!$R$5,IF((COUNTIF(AV5:AY5,\"Yes\")+COUNTIF(AR5,\"Yes\")+COUNTIF(AL5,\"Yes\"))=3,Dropdowns!$R$5,IF((COUNTIF(AV5:AY5,\"Yes\")+COUNTIF(AR5,\"Yes\")+COUNTIF(AL5,\"Yes\"))=2,Dropdowns!$R$4,IF((COUNTIF(AV5:AY5,\"Yes\")+COUNTIF(AR5,\"Yes\")+COUNTIF(AL5,\"Yes\"))=1,Dropdowns!$R$3,IF((COUNTIF(AV5:AY5,\"Yes\")+COUNTIF(AR5,\"Yes\")+COUNTIF(AL5,\"Yes\"))=0,Dropdowns!$R$2, \"NULL\")))))))","v":""},{"$f":"IF(COUNTIF(BB5:BE5, \"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":10},{"$f":"COUNTIF(BF5, \"Yes\")","v":0},{"$f":"COUNTIF(BG5:BH5, \"Yes*\")","v":0},{"$f":"COUNTIF(BI5, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(BI5, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(BJ5, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(BJ5, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(BK5, \"Yes\")","v":0},{"$f":"IF(COUNTIF(BF5:BK5,\"\")=COLUMNS(BF5:BK5),\"\",IF(AND(BL5+BM5+BO5+BR5=0),Dropdowns!$U$2,IF(AND(BL5=1,BM5>=0,BO5+BP5+BQ5+BR5=0, BN5>=0),Dropdowns!$U$3,IF(AND(BL5=1,BM5>=0,BN5+BP5=2,BO5+BQ5=0, BR5>=0),Dropdowns!$U$4,IF(AND(BL5=1,BM5>=0,BN5=0,BO5=1,BP5>=0,BQ5=0,BR5>=0),Dropdowns!$U$5,IF(AND(BL5=1,BM5>=1,BN5=0,BO5=1,BP5=0,BQ5=1,BR5>=0),Dropdowns!$U$6,\"NULL\"))))))","v":""},{"$f":"IF(COUNTIF(BU5:BY5,\"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":12},{"$f":"COUNTIF(BZ5, \"Yes*\")","v":0},{"$f":"COUNTIF(CA5, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CA5, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CB5, \"Yes*\")","v":0},{"$f":"COUNTIF(CC5, \"Yes*\")","v":0},{"$f":"COUNTIF(CD5:CE5, \"Yes\")","v":0},{"$f":"COUNTIF(CF5, \"Yes\")","v":0},{"$f":"IF(COUNTIF(BZ5:CF5,\"\")=COLUMNS(BZ5:CF5),\"\",IF(AND(CG5=0,CH5=0,CJ5=0,CK5=0,CL5=0,CM5=0),Dropdowns!$V$2,IF(AND(CG5=1,(CH5+CI5=0),CJ5=0,CK5=0,CL5=0,CM5=0),Dropdowns!$V$3,IF(AND(CG5=1,(CH5>=0),CI5=0,(CJ5>=0),(CK5>=0),CL5=2,CM5=0),Dropdowns!$V$4,IF(AND(CG5=1,(OR(CH5+CI5+CJ5>=2,CI5=1)),CK5=1,CL5>=0,CM5=0),Dropdowns!$V$5,IF(AND(CG5=1,(OR(CH5+CI5+CJ5>=2,CI5=1)),CK5=1,CL5=2,CM5=1),Dropdowns!$V$6,\"NULL\"))))))","v":""},{"$f":"IF(COUNTIF(CP5:CR5, \"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":8},{"$f":"COUNTIF(CS5, \"Yes\")","v":0},{"$f":"COUNTIF(CT5,\"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CT5, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CU5, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CU5, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CV5, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CV5, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CW5, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CW5, \"Yes - THOROUGH\")","v":0},{"$f":"IF(COUNTIF(CT5:CW5,\"\")=COLUMNS(CT5:CW5),\"\",IF(AND(CX5+CY5+CZ5+DA5+DB5+DC5+DD5+DE5+DF5=0),Dropdowns!$W$2,IF(AND(CX5=1, CY5>=0, CZ5+ DA5+DB5+DC5+DD5+DE5+DF5=0),Dropdowns!$W$3,IF(AND(CX5=1,CY5+CZ5=1,DA5+DC5=1, DB5=0,DD5=0, DE5+DF5=0),Dropdowns!$W$4,IF(AND(CX5=1,(CY5+CZ5=1),(OR(DA5+DC5+DE5=3, (DB5+DD5+DF5<2))), (DA5+DB5+DC5+DD5+DE5+DF5>=2)), Dropdowns!$W$5,IF(AND(CX5=1,CY5+CZ5=1,(DB5+DD5+DF5>=2),(DA5+DC5+DE5=1)),Dropdowns!$W$6,\"NULL\"))))))","v":""},{"$f":"IF(COUNTIF(DI5:DS5,\"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":18},{"$f":"COUNTIF(DT5,\"Yes\")","v":0},0,0,0,0,0,0,{"$f":"IF(COUNTIF(DT5:DV5,\"\")=COLUMNS(DT5:DV5),\"\",IF(AND(EA5+EB5+EC5+EG5=0),Dropdowns!$X$2,IF(AND(EA5=1,EB5+EC5+EG5=0),Dropdowns!$X$3,IF(AND(EA5+EB5=2,EC5+EG5=0),Dropdowns!$X$4,IF(AND(EA5+EB5+EC5=3,EG5=0), Dropdowns!$X$5,IF(AND(EA5+EB5+EC5+EG5=4),Dropdowns!$X$6,\"NULL\"))))))","v":""}]
[{"$nulls":19},{"$f":"IF(COUNTIF(U6:AE6,\"\")=0, \"Yes\", \"No\")","v":"No"},{"$nulls":23},{"$f":"IF(COUNTIF(AL6:AQ6,\"\")=COLUMNS(AL6:AQ6),\"\",IF(COUNTIF(AM6:AQ6,\"Yes\")<4,\"No\",\"Yes\"))","v":""},{"$nulls":3},{"$f":"IF(COUNTIF(AS6:AU6,\"\")=COLUMNS(AS6:AU6),\"\",IF(COUNTIF(AS6:AU6,\"Yes\")<2,\"No\",\"Yes\"))","v":""},{"$nulls":3},{"$f":"IF(COUNTIF(AV6:AY6,\"\")=COLUMNS(AV6:AY6),\"\",IF((COUNTIF(AV6:AY6,\"Yes\")+COUNTIF(AR6,\"Yes\")+COUNTIF(AL6,\"Yes\"))>=5,Dropdowns!$R$6,IF((COUNTIF(AV6:AY6,\"Yes\")+COUNTIF(AR6,\"Yes\")+COUNTIF(AL6,\"Yes\"))=4,Dropdowns!$R$5,IF((COUNTIF(AV6:AY6,\"Yes\")+COUNTIF(AR6,\"Yes\")+COUNTIF(AL6,\"Yes\"))=3,Dropdowns!$R$5,IF((COUNTIF(AV6:AY6,\"Yes\")+COUNTIF(AR6,\"Yes\")+COUNTIF(AL6,\"Yes\"))=2,Dropdowns!$R$4,IF((COUNTIF(AV6:AY6,\"Yes\")+COUNTIF(AR6,\"Yes\")+COUNTIF(AL6,\"Yes\"))=1,Dropdowns!$R$3,IF((COUNTIF(AV6:AY6,\"Yes\")+COUNTIF(AR6,\"Yes\")+COUNTIF(AL6,\"Yes\"))=0,Dropdowns!$R$2, \"NULL\")))))))","v":""},{"$f":"IF(COUNTIF(BB6:BE6, \"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":10},{"$f":"COUNTIF(BF6, \"Yes\")","v":0},{"$f":"COUNTIF(BG6:BH6, \"Yes*\")","v":0},{"$f":"COUNTIF(BI6, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(BI6, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(BJ6, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(BJ6, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(BK6, \"Yes\")","v":0},{"$f":"IF(COUNTIF(BF6:BK6,\"\")=COLUMNS(BF6:BK6),\"\",IF(AND(BL6+BM6+BO6+BR6=0),Dropdowns!$U$2,IF(AND(BL6=1,BM6>=0,BO6+BP6+BQ6+BR6=0, BN6>=0),Dropdowns!$U$3,IF(AND(BL6=1,BM6>=0,BN6+BP6=2,BO6+BQ6=0, BR6>=0),Dropdowns!$U$4,IF(AND(BL6=1,BM6>=0,BN6=0,BO6=1,BP6>=0,BQ6=0,BR6>=0),Dropdowns!$U$5,IF(AND(BL6=1,BM6>=1,BN6=0,BO6=1,BP6=0,BQ6=1,BR6>=0),Dropdowns!$U$6,\"NULL\"))))))","v":""},{"$f":"IF(COUNTIF(BU6:BY6,\"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":12},{"$f":"COUNTIF(BZ6, \"Yes*\")","v":0},{"$f":"COUNTIF(CA6, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CA6, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CB6, \"Yes*\")","v":0},{"$f":"COUNTIF(CC6, \"Yes*\")","v":0},{"$f":"COUNTIF(CD6:CE6, \"Yes\")","v":0},{"$f":"COUNTIF(CF6, \"Yes\")","v":0},{"$f":"IF(COUNTIF(BZ6:CF6,\"\")=COLUMNS(BZ6:CF6),\"\",IF(AND(CG6=0,CH6=0,CJ6=0,CK6=0,CL6=0,CM6=0),Dropdowns!$V$2,IF(AND(CG6=1,(CH6+CI6=0),CJ6=0,CK6=0,CL6=0,CM6=0),Dropdowns!$V$3,IF(AND(CG6=1,(CH6>=0),CI6=0,(CJ6>=0),(CK6>=0),CL6=2,CM6=0),Dropdowns!$V$4,IF(AND(CG6=1,(OR(CH6+CI6+CJ6>=2,CI6=1)),CK6=1,CL6>=0,CM6=0),Dropdowns!$V$5,IF(AND(CG6=1,(OR(CH6+CI6+CJ6>=2,CI6=1)),CK6=1,CL6=2,CM6=1),Dropdowns!$V$6,\"NULL\"))))))","v":""},{"$f":"IF(COUNTIF(CP6:CR6, \"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":8},{"$f":"COUNTIF(CS6, \"Yes\")","v":0},{"$f":"COUNTIF(CT6,\"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CT6, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CU6, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CU6, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CV6, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CV6, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CW6, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CW6, \"Yes - THOROUGH\")","v":0},{"$f":"IF(COUNTIF(CT6:CW6,\"\")=COLUMNS(CT6:CW6),\"\",IF(AND(CX6+CY6+CZ6+DA6+DB6+DC6+DD6+DE6+DF6=0),Dropdowns!$W$2,IF(AND(CX6=1, CY6>=0, CZ6+ DA6+DB6+DC6+DD6+DE6+DF6=0),Dropdowns!$W$3,IF(AND(CX6=1,CY6+CZ6=1,DA6+DC6=1, DB6=0,DD6=0, DE6+DF6=0),Dropdowns!$W$4,IF(AND(CX6=1,(CY6+CZ6=1),(OR(DA6+DC6+DE6=3, (DB6+DD6+DF6<2))), (DA6+DB6+DC6+DD6+DE6+DF6>=2)), Dropdowns!$W$5,IF(AND(CX6=1,CY6+CZ6=1,(DB6+DD6+DF6>=2),(DA6+DC6+DE6=1)),Dropdowns!$W$6,\"NULL\"))))))","v":""},{"$f":"IF(COUNTIF(DI6:DS6,\"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":18},{"$f":"COUNTIF(DT6,\"Yes\")","v":0},0,0,0,0,0,0,{"$f":"IF(COUNTIF(DT6:DV6,\"\")=COLUMNS(DT6:DV6),\"\",IF(AND(EA6+EB6+EC6+EG6=0),Dropdowns!$X$2,IF(AND(EA6=1,EB6+EC6+EG6=0),Dropdowns!$X$3,IF(AND(EA6+EB6=2,EC6+EG6=0),Dropdowns!$X$4,IF(AND(EA6+EB6+EC6=3,EG6=0), Dropdowns!$X$5,IF(AND(EA6+EB6+EC6+EG6=4),Dropdowns!$X$6,\"NULL\"))))))","v":""}]
[{"$nulls":19},{"$f":"IF(COUNTIF(U7:AE7,\"\")=0, \"Yes\", \"No\")","v":"No"},{"$nulls":23},{"$f":"IF(COUNTIF(AL7:AQ7,\"\")=COLUMNS(AL7:AQ7),\"\",IF(COUNTIF(AM7:AQ7,\"Yes\")<4,\"No\",\"Yes\"))","v":""},{"$nulls":3},{"$f":"IF(COUNTIF(AS7:AU7,\"\")=COLUMNS(AS7:AU7),\"\",IF(COUNTIF(AS7:AU7,\"Yes\")<2,\"No\",\"Yes\"))","v":""},{"$nulls":3},{"$f":"IF(COUNTIF(AV7:AY7,\"\")=COLUMNS(AV7:AY7),\"\",IF((COUNTIF(AV7:AY7,\"Yes\")+COUNTIF(AR7,\"Yes\")+COUNTIF(AL7,\"Yes\"))>=5,Dropdowns!$R$6,IF((COUNTIF(AV7:AY7,\"Yes\")+COUNTIF(AR7,\"Yes\")+COUNTIF(AL7,\"Yes\"))=4,Dropdowns!$R$5,IF((COUNTIF(AV7:AY7,\"Yes\")+COUNTIF(AR7,\"Yes\")+COUNTIF(AL7,\"Yes\"))=3,Dropdowns!$R$5,IF((COUNTIF(AV7:AY7,\"Yes\")+COUNTIF(AR7,\"Yes\")+COUNTIF(AL7,\"Yes\"))=2,Dropdowns!$R$4,IF((COUNTIF(AV7:AY7,\"Yes\")+COUNTIF(AR7,\"Yes\")+COUNTIF(AL7,\"Yes\"))=1,Dropdowns!$R$3,IF((COUNTIF(AV7:AY7,\"Yes\")+COUNTIF(AR7,\"Yes\")+COUNTIF(AL7,\"Yes\"))=0,Dropdowns!$R$2, \"NULL\")))))))","v":""},{"$f":"IF(COUNTIF(BB7:BE7, \"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":10},{"$f":"COUNTIF(BF7, \"Yes\")","v":0},{"$f":"COUNTIF(BG7:BH7, \"Yes*\")","v":0},{"$f":"COUNTIF(BI7, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(BI7, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(BJ7, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(BJ7, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(BK7, \"Yes\")","v":0},{"$f":"IF(COUNTIF(BF7:BK7,\"\")=COLUMNS(BF7:BK7),\"\",IF(AND(BL7+BM7+BO7+BR7=0),Dropdowns!$U$2,IF(AND(BL7=1,BM7>=0,BO7+BP7+BQ7+BR7=0, BN7>=0),Dropdowns!$U$3,IF(AND(BL7+BM7>=2,BN7+BP7=2,BO7+BQ7=0, BR7>=0),Dropdowns!$U$4,IF(AND(BL7=1,BM7>=0,BN7=0,BO7=1,BP7>=0,BQ7=0,BR7>=0),Dropdowns!$U$5,IF(AND(BL7+BM7>=2,BN7=0,BO7=1,BP7=0,BQ7=1,BR7>=0),Dropdowns!$U$6,\"NULL\"))))))","v":""},{"$f":"IF(COUNTIF(BU7:BY7,\"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":12},{"$f":"COUNTIF(BZ7, \"Yes*\")","v":0},{"$f":"COUNTIF(CA7, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CA7, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CB7, \"Yes*\")","v":0},{"$f":"COUNTIF(CC7, \"Yes*\")","v":0},{"$f":"COUNTIF(CD7:CE7, \"Yes\")","v":0},{"$f":"COUNTIF(CF7, \"Yes\")","v":0},{"$f":"IF(COUNTIF(BZ7:CF7,\"\")=COLUMNS(BZ7:CF7),\"\",IF(AND(CG7=0,CH7=0,CJ7=0,CK7=0,CL7=0,CM7=0),Dropdowns!$V$2,IF(AND(CG7=1,(CH7+CI7=0),CJ7=0,CK7=0,CL7=0,CM7=0),Dropdowns!$V$3,IF(AND(CG7=1,(CH7>=0),CI7=0,(CJ7>=0),(CK7>=0),CL7=2,CM7=0),Dropdowns!$V$4,IF(AND(CG7=1,(OR(CH7+CI7+CJ7>=2,CI7=1)),CK7=1,CL7>=0,CM7=0),Dropdowns!$V$5,IF(AND(CG7=1,(OR(CH7+CI7+CJ7>=2,CI7=1)),CK7=1,CL7=2,CM7=1),Dropdowns!$V$6,\"NULL\"))))))","v":""},{"$f":"IF(COUNTIF(CP7:CR7, \"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":8},{"$f":"COUNTIF(CS7, \"Yes\")","v":0},{"$f":"COUNTIF(CT7,\"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CT7, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CU7, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CU7, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CV7, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CV7, \"Yes - THOROUGH\")","v":0},{"$f":"COUNTIF(CW7, \"Yes - BASIC\")","v":0},{"$f":"COUNTIF(CW7, \"Yes - THOROUGH\")","v":0},{"$f":"IF(COUNTIF(CT7:CW7,\"\")=COLUMNS(CT7:CW7),\"\",IF(AND(CX7+CY7+CZ7+DA7+DB7+DC7+DD7+DE7+DF7=0),Dropdowns!$W$2,IF(AND(CX7=1, CY7>=0, CZ7+ DA7+DB7+DC7+DD7+DE7+DF7=0),Dropdowns!$W$3,IF(AND(CX7=1,CY7+CZ7=1,DA7+DC7=1, DB7=0,DD7=0, DE7+DF7=0),Dropdowns!$W$4,IF(AND(CX7=1,(CY7+CZ7=1),(OR(DA7+DC7+DE7=3, (DB7+DD7+DF7<2))), (DA7+DB7+DC7+DD7+DE7+DF7>=2)), Dropdowns!$W$5,IF(AND(CX7=1,CY7+CZ7=1,(DB7+DD7+DF7>=2),(DA7+DC7+DE7=1)),Dropdowns!$W$6,\"NULL\"))))))","v":""},{"$f":"IF(COUNTIF(DI7:DS7,\"\")=0,\"Yes\",\"No\")","v":"No"},{"$nulls":18},{"$f":"COUNTIF(DT7,\"Yes\")","v":0},{"$f":"COUNTIF(DU7,\"Yes\")","v":0},{"$f":"COUNTIF(DV7,\"Yes\")","v":0},{"$f":"COUNTIF(DW7,\"Yes\")","v":0},{"$f":"COUNTIF(DX7,\"Yes\")","v":0},{"$f":"COUNTIF(DY7,\"Yes\")","v":0},{"$f":"COUNTIF(DZ7,\"Yes\")","v":0},{"$f":"IF(COUNTIF(DT7:DV7,\"\")=COLUMNS(DT7:DV7),\"\",IF(AND(EA7+EB7+EC7+EG7=0),Dropdowns!$X$2,IF(AND(EA7=1,EB7+EC7+EG7=0),Dropdowns!$X$3,IF(AND(EA7+EB7=2,EC7+EG7=0),Dropdowns!$X$4,IF(AND(EA7+EB7+EC7=3,EG7=0), Dropdowns!$X$5,IF(AND(EA7+EB7+EC7+EG7=4),Dropdowns!$X$6,\"NULL\"))))))","v":""}]
[{"$nulls":19},"No",{"$nulls":23},"",{"$nulls":3},"",{"$nulls":3},{"$f":"IF(COUNTIF(AV8:AY8,\"\")=COLUMNS(AV8:AY8),\"\",IF((COUNTIF(AV8:AY8,\"Yes\")+COUNTIF(AR8,\"Yes\")+COUNTIF(AL8,\"Yes\"))>=5,Dropdowns!$R$6,IF((COUNTIF(AV8:AY8,\"Yes\")+COUNTIF(AR8,\"Yes\")+COUNTIF(AL8,\"Yes\"))=4,Dropdowns!$R$5,IF((COUNTIF(AV8:AY8,\"Yes\")+COUNTIF(AR8,\"Yes\")+COUNTIF(AL8,\"Yes\"))=3,Dropdowns!$R$5,IF((COUNTIF(AV8:AY8,\"Yes\")+COUNTIF(AR8,\"Yes\")+COUNTIF(AL8,\"Yes\"))=2,Dropdowns!$R$4,IF((COUNTIF(AV8:AY8,\"Yes\")+COUNTIF(AR8,\"Yes\")+COUNTIF(AL8,\"Yes\"))=1,Dropdowns!$R$3,IF((COUNTIF(AV8:AY8,\"Yes\")+COUNTIF(AR8,\"Yes\")+COUNTIF(AL8,\"Yes\"))=0,Dropdowns!$R$2, \"NULL\")))))))","v":""},"No",{"$nulls":10},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(BF8:BK8,\"\")=COLUMNS(BF8:BK8),\"\",IF(AND(BL8+BM8+BO8+BR8=0),Dropdowns!$U$2,IF(AND(BL8=1,BM8>=0,BO8+BP8+BQ8+BR8=0, BN8>=0),Dropdowns!$U$3,IF(AND(BL8+BM8>=2,BN8+BP8=2,BO8+BQ8=0, BR8>=0),Dropdowns!$U$4,IF(AND(BL8=1,BM8>=0,BN8=0,BO8=1,BP8>=0,BQ8=0,BR8>=0),Dropdowns!$U$5,IF(AND(BL8+BM8>=2,BN8=0,BO8=1,BP8=0,BQ8=1,BR8>=0),Dropdowns!$U$6,\"NULL\"))))))","v":""},"No",{"$nulls":12},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(BZ8:CF8,\"\")=COLUMNS(BZ8:CF8),\"\",IF(AND(CG8=0,CH8=0,CJ8=0,CK8=0,CL8=0,CM8=0),Dropdowns!$V$2,IF(AND(CG8=1,(CH8+CI8=0),CJ8=0,CK8=0,CL8=0,CM8=0),Dropdowns!$V$3,IF(AND(CG8=1,(CH8>=0),CI8=0,(CJ8>=0),(CK8>=0),CL8=2,CM8=0),Dropdowns!$V$4,IF(AND(CG8=1,(OR(CH8+CI8+CJ8>=2,CI8=1)),CK8=1,CL8>=0,CM8=0),Dropdowns!$V$5,IF(AND(CG8=1,(OR(CH8+CI8+CJ8>=2,CI8=1)),CK8=1,CL8=2,CM8=1),Dropdowns!$V$6,\"NULL\"))))))","v":""},"No",{"$nulls":8},0,0,0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(CT8:CW8,\"\")=COLUMNS(CT8:CW8),\"\",IF(AND(CX8+CY8+CZ8+DA8+DB8+DC8+DD8+DE8+DF8=0),Dropdowns!$W$2,IF(AND(CX8=1, CY8>=0, CZ8+ DA8+DB8+DC8+DD8+DE8+DF8=0),Dropdowns!$W$3,IF(AND(CX8=1,CY8+CZ8=1,DA8+DC8=1, DB8=0,DD8=0, DE8+DF8=0),Dropdowns!$W$4,IF(AND(CX8=1,(CY8+CZ8=1),(OR(DA8+DC8+DE8=3, (DB8+DD8+DF8<2))), (DA8+DB8+DC8+DD8+DE8+DF8>=2)), Dropdowns!$W$5,IF(AND(CX8=1,CY8+CZ8=1,(DB8+DD8+DF8>=2),(DA8+DC8+DE8=1)),Dropdowns!$W$6,\"NULL\"))))))","v":""},"No",{"$nulls":18},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(DT8:DV8,\"\")=COLUMNS(DT8:DV8),\"\",IF(AND(EA8+EB8+EC8+EG8=0),Dropdowns!$X$2,IF(AND(EA8=1,EB8+EC8+EG8=0),Dropdowns!$X$3,IF(AND(EA8+EB8=2,EC8+EG8=0),Dropdowns!$X$4,IF(AND(EA8+EB8+EC8=3,EG8=0), Dropdowns!$X$5,IF(AND(EA8+EB8+EC8+EG8=4),Dropdowns!$X$6,\"NULL\"))))))","v":""}]
[{"$nulls":19},"No",{"$nulls":23},"",{"$nulls":3},"",{"$nulls":3},{"$f":"IF(COUNTIF(AV9:AY9,\"\")=COLUMNS(AV9:AY9),\"\",IF((COUNTIF(AV9:AY9,\"Yes\")+COUNTIF(AR9,\"Yes\")+COUNTIF(AL9,\"Yes\"))>=5,Dropdowns!$R$6,IF((COUNTIF(AV9:AY9,\"Yes\")+COUNTIF(AR9,\"Yes\")+COUNTIF(AL9,\"Yes\"))=4,Dropdowns!$R$5,IF((COUNTIF(AV9:AY9,\"Yes\")+COUNTIF(AR9,\"Yes\")+COUNTIF(AL9,\"Yes\"))=3,Dropdowns!$R$5,IF((COUNTIF(AV9:AY9,\"Yes\")+COUNTIF(AR9,\"Yes\")+COUNTIF(AL9,\"Yes\"))=2,Dropdowns!$R$4,IF((COUNTIF(AV9:AY9,\"Yes\")+COUNTIF(AR9,\"Yes\")+COUNTIF(AL9,\"Yes\"))=1,Dropdowns!$R$3,IF((COUNTIF(AV9:AY9,\"Yes\")+COUNTIF(AR9,\"Yes\")+COUNTIF(AL9,\"Yes\"))=0,Dropdowns!$R$2, \"NULL\")))))))","v":""},"No",{"$nulls":10},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(BF9:BK9,\"\")=COLUMNS(BF9:BK9),\"\",IF(AND(BL9+BM9+BO9+BR9=0),Dropdowns!$U$2,IF(AND(BL9=1,BM9>=0,BO9+BP9+BQ9+BR9=0, BN9>=0),Dropdowns!$U$3,IF(AND(BL9+BM9>=2,BN9+BP9=2,BO9+BQ9=0, BR9>=0),Dropdowns!$U$4,IF(AND(BL9=1,BM9>=0,BN9=0,BO9=1,BP9>=0,BQ9=0,BR9>=0),Dropdowns!$U$5,IF(AND(BL9+BM9>=2,BN9=0,BO9=1,BP9=0,BQ9=1,BR9>=0),Dropdowns!$U$6,\"NULL\"))))))","v":""},"No",{"$nulls":12},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(BZ9:CF9,\"\")=COLUMNS(BZ9:CF9),\"\",IF(AND(CG9=0,CH9=0,CJ9=0,CK9=0,CL9=0,CM9=0),Dropdowns!$V$2,IF(AND(CG9=1,(CH9+CI9=0),CJ9=0,CK9=0,CL9=0,CM9=0),Dropdowns!$V$3,IF(AND(CG9=1,(CH9>=0),CI9=0,(CJ9>=0),(CK9>=0),CL9=2,CM9=0),Dropdowns!$V$4,IF(AND(CG9=1,(OR(CH9+CI9+CJ9>=2,CI9=1)),CK9=1,CL9>=0,CM9=0),Dropdowns!$V$5,IF(AND(CG9=1,(OR(CH9+CI9+CJ9>=2,CI9=1)),CK9=1,CL9=2,CM9=1),Dropdowns!$V$6,\"NULL\"))))))","v":""},"No",{"$nulls":8},0,0,0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(CT9:CW9,\"\")=COLUMNS(CT9:CW9),\"\",IF(AND(CX9+CY9+CZ9+DA9+DB9+DC9+DD9+DE9+DF9=0),Dropdowns!$W$2,IF(AND(CX9=1, CY9>=0, CZ9+ DA9+DB9+DC9+DD9+DE9+DF9=0),Dropdowns!$W$3,IF(AND(CX9=1,CY9+CZ9=1,DA9+DC9=1, DB9=0,DD9=0, DE9+DF9=0),Dropdowns!$W$4,IF(AND(CX9=1,(CY9+CZ9=1),(OR(DA9+DC9+DE9=3, (DB9+DD9+DF9<2))), (DA9+DB9+DC9+DD9+DE9+DF9>=2)), Dropdowns!$W$5,IF(AND(CX9=1,CY9+CZ9=1,(DB9+DD9+DF9>=2),(DA9+DC9+DE9=1)),Dropdowns!$W$6,\"NULL\"))))))","v":""},"No",{"$nulls":18},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(DT9:DV9,\"\")=COLUMNS(DT9:DV9),\"\",IF(AND(EA9+EB9+EC9+EG9=0),Dropdowns!$X$2,IF(AND(EA9=1,EB9+EC9+EG9=0),Dropdowns!$X$3,IF(AND(EA9+EB9=2,EC9+EG9=0),Dropdowns!$X$4,IF(AND(EA9+EB9+EC9=3,EG9=0), Dropdowns!$X$5,IF(AND(EA9+EB9+EC9+EG9=4),Dropdowns!$X$6,\"NULL\"))))))","v":""}]
[{"$nulls":19},"No",{"$nulls":23},"",{"$nulls":3},"",{"$nulls":3},{"$f":"IF(COUNTIF(AV10:AY10,\"\")=COLUMNS(AV10:AY10),\"\",IF((COUNTIF(AV10:AY10,\"Yes\")+COUNTIF(AR10,\"Yes\")+COUNTIF(AL10,\"Yes\"))>=5,Dropdowns!$R$6,IF((COUNTIF(AV10:AY10,\"Yes\")+COUNTIF(AR10,\"Yes\")+COUNTIF(AL10,\"Yes\"))=4,Dropdowns!$R$5,IF((COUNTIF(AV10:AY10,\"Yes\")+COUNTIF(AR10,\"Yes\")+COUNTIF(AL10,\"Yes\"))=3,Dropdowns!$R$5,IF((COUNTIF(AV10:AY10,\"Yes\")+COUNTIF(AR10,\"Yes\")+COUNTIF(AL10,\"Yes\"))=2,Dropdowns!$R$4,IF((COUNTIF(AV10:AY10,\"Yes\")+COUNTIF(AR10,\"Yes\")+COUNTIF(AL10,\"Yes\"))=1,Dropdowns!$R$3,IF((COUNTIF(AV10:AY10,\"Yes\")+COUNTIF(AR10,\"Yes\")+COUNTIF(AL10,\"Yes\"))=0,Dropdowns!$R$2, \"NULL\")))))))","v":""},"No",{"$nulls":10},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(BF10:BK10,\"\")=COLUMNS(BF10:BK10),\"\",IF(AND(BL10+BM10+BO10+BR10=0),Dropdowns!$U$2,IF(AND(BL10=1,BM10>=0,BO10+BP10+BQ10+BR10=0, BN10>=0),Dropdowns!$U$3,IF(AND(BL10+BM10>=2,BN10+BP10=2,BO10+BQ10=0, BR10>=0),Dropdowns!$U$4,IF(AND(BL10=1,BM10>=0,BN10=0,BO10=1,BP10>=0,BQ10=0,BR10>=0),Dropdowns!$U$5,IF(AND(BL10+BM10>=2,BN10=0,BO10=1,BP10=0,BQ10=1,BR10>=0),Dropdowns!$U$6,\"NULL\"))))))","v":""},"No",{"$nulls":12},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(BZ10:CF10,\"\")=COLUMNS(BZ10:CF10),\"\",IF(AND(CG10=0,CH10=0,CJ10=0,CK10=0,CL10=0,CM10=0),Dropdowns!$V$2,IF(AND(CG10=1,(CH10+CI10=0),CJ10=0,CK10=0,CL10=0,CM10=0),Dropdowns!$V$3,IF(AND(CG10=1,(CH10>=0),CI10=0,(CJ10>=0),(CK10>=0),CL10=2,CM10=0),Dropdowns!$V$4,IF(AND(CG10=1,(OR(CH10+CI10+CJ10>=2,CI10=1)),CK10=1,CL10>=0,CM10=0),Dropdowns!$V$5,IF(AND(CG10=1,(OR(CH10+CI10+CJ10>=2,CI10=1)),CK10=1,CL10=2,CM10=1),Dropdowns!$V$6,\"NULL\"))))))","v":""},"No",{"$nulls":8},0,0,0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(CT10:CW10,\"\")=COLUMNS(CT10:CW10),\"\",IF(AND(CX10+CY10+CZ10+DA10+DB10+DC10+DD10+DE10+DF10=0),Dropdowns!$W$2,IF(AND(CX10=1, CY10>=0, CZ10+ DA10+DB10+DC10+DD10+DE10+DF10=0),Dropdowns!$W$3,IF(AND(CX10=1,CY10+CZ10=1,DA10+DC10=1, DB10=0,DD10=0, DE10+DF10=0),Dropdowns!$W$4,IF(AND(CX10=1,(CY10+CZ10=1),(OR(DA10+DC10+DE10=3, (DB10+DD10+DF10<2))), (DA10+DB10+DC10+DD10+DE10+DF10>=2)), Dropdowns!$W$5,IF(AND(CX10=1,CY10+CZ10=1,(DB10+DD10+DF10>=2),(DA10+DC10+DE10=1)),Dropdowns!$W$6,\"NULL\"))))))","v":""},"No",{"$nulls":18},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(DT10:DV10,\"\")=COLUMNS(DT10:DV10),\"\",IF(AND(EA10+EB10+EC10+EG10=0),Dropdowns!$X$2,IF(AND(EA10=1,EB10+EC10+EG10=0),Dropdowns!$X$3,IF(AND(EA10+EB10=2,EC10+EG10=0),Dropdowns!$X$4,IF(AND(EA10+EB10+EC10=3,EG10=0), Dropdowns!$X$5,IF(AND(EA10+EB10+EC10+EG10=4),Dropdowns!$X$6,\"NULL\"))))))","v":""}]
[{"$nulls":19},"No",{"$nulls":23},"",{"$nulls":3},"",{"$nulls":3},{"$f":"IF(COUNTIF(AV11:AY11,\"\")=COLUMNS(AV11:AY11),\"\",IF((COUNTIF(AV11:AY11,\"Yes\")+COUNTIF(AR11,\"Yes\")+COUNTIF(AL11,\"Yes\"))>=5,Dropdowns!$R$6,IF((COUNTIF(AV11:AY11,\"Yes\")+COUNTIF(AR11,\"Yes\")+COUNTIF(AL11,\"Yes\"))=4,Dropdowns!$R$5,IF((COUNTIF(AV11:AY11,\"Yes\")+COUNTIF(AR11,\"Yes\")+COUNTIF(AL11,\"Yes\"))=3,Dropdowns!$R$5,IF((COUNTIF(AV11:AY11,\"Yes\")+COUNTIF(AR11,\"Yes\")+COUNTIF(AL11,\"Yes\"))=2,Dropdowns!$R$4,IF((COUNTIF(AV11:AY11,\"Yes\")+COUNTIF(AR11,\"Yes\")+COUNTIF(AL11,\"Yes\"))=1,Dropdowns!$R$3,IF((COUNTIF(AV11:AY11,\"Yes\")+COUNTIF(AR11,\"Yes\")+COUNTIF(AL11,\"Yes\"))=0,Dropdowns!$R$2, \"NULL\")))))))","v":""},"No",{"$nulls":10},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(BF11:BK11,\"\")=COLUMNS(BF11:BK11),\"\",IF(AND(BL11+BM11+BO11+BR11=0),Dropdowns!$U$2,IF(AND(BL11=1,BM11>=0,BO11+BP11+BQ11+BR11=0, BN11>=0),Dropdowns!$U$3,IF(AND(BL11+BM11>=2,BN11+BP11=2,BO11+BQ11=0, BR11>=0),Dropdowns!$U$4,IF(AND(BL11=1,BM11>=0,BN11=0,BO11=1,BP11>=0,BQ11=0,BR11>=0),Dropdowns!$U$5,IF(AND(BL11+BM11>=2,BN11=0,BO11=1,BP11=0,BQ11=1,BR11>=0),Dropdowns!$U$6,\"NULL\"))))))","v":""},"No",{"$nulls":12},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(BZ11:CF11,\"\")=COLUMNS(BZ11:CF11),\"\",IF(AND(CG11=0,CH11=0,CJ11=0,CK11=0,CL11=0,CM11=0),Dropdowns!$V$2,IF(AND(CG11=1,(CH11+CI11=0),CJ11=0,CK11=0,CL11=0,CM11=0),Dropdowns!$V$3,IF(AND(CG11=1,(CH11>=0),CI11=0,(CJ11>=0),(CK11>=0),CL11=2,CM11=0),Dropdowns!$V$4,IF(AND(CG11=1,(OR(CH11+CI11+CJ11>=2,CI11=1)),CK11=1,CL11>=0,CM11=0),Dropdowns!$V$5,IF(AND(CG11=1,(OR(CH11+CI11+CJ11>=2,CI11=1)),CK11=1,CL11=2,CM11=1),Dropdowns!$V$6,\"NULL\"))))))","v":""},"No",{"$nulls":8},0,0,0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(CT11:CW11,\"\")=COLUMNS(CT11:CW11),\"\",IF(AND(CX11+CY11+CZ11+DA11+DB11+DC11+DD11+DE11+DF11=0),Dropdowns!$W$2,IF(AND(CX11=1, CY11>=0, CZ11+ DA11+DB11+DC11+DD11+DE11+DF11=0),Dropdowns!$W$3,IF(AND(CX11=1,CY11+CZ11=1,DA11+DC11=1, DB11=0,DD11=0, DE11+DF11=0),Dropdowns!$W$4,IF(AND(CX11=1,(CY11+CZ11=1),(OR(DA11+DC11+DE11=3, (DB11+DD11+DF11<2))), (DA11+DB11+DC11+DD11+DE11+DF11>=2)), Dropdowns!$W$5,IF(AND(CX11=1,CY11+CZ11=1,(DB11+DD11+DF11>=2),(DA11+DC11+DE11=1)),Dropdowns!$W$6,\"NULL\"))))))","v":""},"No",{"$nulls":18},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(DT11:DV11,\"\")=COLUMNS(DT11:DV11),\"\",IF(AND(EA11+EB11+EC11+EG11=0),Dropdowns!$X$2,IF(AND(EA11=1,EB11+EC11+EG11=0),Dropdowns!$X$3,IF(AND(EA11+EB11=2,EC11+EG11=0),Dropdowns!$X$4,IF(AND(EA11+EB11+EC11=3,EG11=0), Dropdowns!$X$5,IF(AND(EA11+EB11+EC11+EG11=4),Dropdowns!$X$6,\"NULL\"))))))","v":""}]
[{"$nulls":19},"No",{"$nulls":23},"",{"$nulls":3},"",{"$nulls":3},{"$f":"IF(COUNTIF(AV12:AY12,\"\")=COLUMNS(AV12:AY12),\"\",IF((COUNTIF(AV12:AY12,\"Yes\")+COUNTIF(AR12,\"Yes\")+COUNTIF(AL12,\"Yes\"))>=5,Dropdowns!$R$6,IF((COUNTIF(AV12:AY12,\"Yes\")+COUNTIF(AR12,\"Yes\")+COUNTIF(AL12,\"Yes\"))=4,Dropdowns!$R$5,IF((COUNTIF(AV12:AY12,\"Yes\")+COUNTIF(AR12,\"Yes\")+COUNTIF(AL12,\"Yes\"))=3,Dropdowns!$R$5,IF((COUNTIF(AV12:AY12,\"Yes\")+COUNTIF(AR12,\"Yes\")+COUNTIF(AL12,\"Yes\"))=2,Dropdowns!$R$4,IF((COUNTIF(AV12:AY12,\"Yes\")+COUNTIF(AR12,\"Yes\")+COUNTIF(AL12,\"Yes\"))=1,Dropdowns!$R$3,IF((COUNTIF(AV12:AY12,\"Yes\")+COUNTIF(AR12,\"Yes\")+COUNTIF(AL12,\"Yes\"))=0,Dropdowns!$R$2, \"NULL\")))))))","v":""},"No",{"$nulls":10},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(BF12:BK12,\"\")=COLUMNS(BF12:BK12),\"\",IF(AND(BL12+BM12+BO12+BR12=0),Dropdowns!$U$2,IF(AND(BL12=1,BM12>=0,BO12+BP12+BQ12+BR12=0, BN12>=0),Dropdowns!$U$3,IF(AND(BL12+BM12>=2,BN12+BP12=2,BO12+BQ12=0, BR12>=0),Dropdowns!$U$4,IF(AND(BL12=1,BM12>=0,BN12=0,BO12=1,BP12>=0,BQ12=0,BR12>=0),Dropdowns!$U$5,IF(AND(BL12+BM12>=2,BN12=0,BO12=1,BP12=0,BQ12=1,BR12>=0),Dropdowns!$U$6,\"NULL\"))))))","v":""},"No",{"$nulls":12},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(BZ12:CF12,\"\")=COLUMNS(BZ12:CF12),\"\",IF(AND(CG12=0,CH12=0,CJ12=0,CK12=0,CL12=0,CM12=0),Dropdowns!$V$2,IF(AND(CG12=1,(CH12+CI12=0),CJ12=0,CK12=0,CL12=0,CM12=0),Dropdowns!$V$3,IF(AND(CG12=1,(CH12>=0),CI12=0,(CJ12>=0),(CK12>=0),CL12=2,CM12=0),Dropdowns!$V$4,IF(AND(CG12=1,(OR(CH12+CI12+CJ12>=2,CI12=1)),CK12=1,CL12>=0,CM12=0),Dropdowns!$V$5,IF(AND(CG12=1,(OR(CH12+CI12+CJ12>=2,CI12=1)),CK12=1,CL12=2,CM12=1),Dropdowns!$V$6,\"NULL\"))))))","v":""},"No",{"$nulls":8},0,0,0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(CT12:CW12,\"\")=COLUMNS(CT12:CW12),\"\",IF(AND(CX12+CY12+CZ12+DA12+DB12+DC12+DD12+DE12+DF12=0),Dropdowns!$W$2,IF(AND(CX12=1, CY12>=0, CZ12+ DA12+DB12+DC12+DD12+DE12+DF12=0),Dropdowns!$W$3,IF(AND(CX12=1,CY12+CZ12=1,DA12+DC12=1, DB12=0,DD12=0, DE12+DF12=0),Dropdowns!$W$4,IF(AND(CX12=1,(CY12+CZ12=1),(OR(DA12+DC12+DE12=3, (DB12+DD12+DF12<2))), (DA12+DB12+DC12+DD12+DE12+DF12>=2)), Dropdowns!$W$5,IF(AND(CX12=1,CY12+CZ12=1,(DB12+DD12+DF12>=2),(DA12+DC12+DE12=1)),Dropdowns!$W$6,\"NULL\"))))))","v":""},"No",{"$nulls":18},0,0,0,0,0,0,0,{"$f":"IF(COUNTIF(DT12:DV12,\"\")=COLUMNS(DT12:DV12),\"\",IF(AND(EA12+EB12+EC12+EG12=0),Dropdowns!$X$2,IF(AND(EA12=1,EB12+EC12+EG12=0),Dropdowns!$X$3,IF(AND(EA12+EB12=2,EC12+EG12=0),Dropdowns!$X$4,IF(AND(EA12+EB12+EC12=3,EG12=0), Dropdowns!$X$5,IF(AND(EA12+EB12+EC12+EG12=4),Dropdowns!$X$6,\"NULL\"))))))","v":""}]
["NOTES ROW"]

{"sheet":"Codebook","index":2,"n_rows":368,"n_cols":4,"rows_kept":368,"merged":[[2,0,5,0],[2,1,5,1],[2,2,5,2],[7,0,18,0],[7,1,18,1],[7,2,18,2],[19,0,21,0],[19,1,21,1],[19,3,21,3],[22,0,24,0],[22,1,24,1],[22,3,24,3],[25,0,27,0],[25,1,27,1],[25,3,27,3],[28,0,40,0],[28,1,40,1],[28,2,40,2],[41,0,44,0],[41,1,44,1],[41,2,44,2],[48,0,54,0],[48,1,54,1],[48,2,54,2],[55,0,63,0],[55,1,63,1],[64,0,74,0],[64,1,74,1],[75,0,84,0],[75,1,84,1],[85,0,88,0],[85,1,88,1],[89,0,97,0],[89,1,97,1],[99,2,99,3],[101,0,110,0],[101,1,110,1],[101,3,110,3],[122,0,127,0],[122,1,127,1],[122,3,127,3],[128,0,131,0],[128,1,131,1],[128,3,131,3],[132,0,134,0],[132,1,134,1],[132,3,134,3],[135,0,139,0],[135,1,139,1],[135,3,139,3],[140,0,146,0],[140,1,146,1],[147,0,151,0],[147,1,151,1],[152,0,160,0],[152,1,160,1],[161,0,164,0],[161,1,164,1],[165,0,168,0],[165,1,168,1],[169,0,171,0],[169,1,171,1],[172,0,174,0],[172,1,174,1],[172,3,174,3],[175,0,185,0],[175,1,185,1],[186,0,193,0],[186,1,193,1],[194,0,196,0],[194,1,196,1],[197,0,199,0],[197,1,199,1],[200,0,202,0],[200,1,202,1],[200,3,202,3],[203,0,205,0],[203,1,205,1],[206,0,212,0],[206,1,212,1],[213,0,220,0],[213,1,220,1],[221,2,221,3],[225,0,226,0],[225,1,226,1],[225,2,226,2],[227,0,229,0],[227,1,229,1],[230,0,232,0],[230,1,232,1],[230,3,232,3],[233,0,235,0],[233,1,235,1],[233,3,235,3],[236,0,240,0],[236,1,240,1],[241,0,245,0],[241,1,245,1],[246,0,248,0],[246,1,248,1],[246,3,248,3],[249,0,254,0],[249,1,254,1],[255,2,255,3],[257,0,258,0],[257,1,258,1],[257,2,258,2],[260,0,261,0],[260,1,261,1],[260,2,261,2],[263,0,265,0],[263,1,265,1],[263,3,265,3],[266,0,269,0],[266,1,269,1],[266,3,269,3],[270,0,272,0],[270,1,272,1],[270,3,272,3],[273,0,275,0],[273,1,275,1],[273,3,275,3],[276,0,278,0],[276,1,278,1],[276,3,278,3],[279,0,281,0],[279,1,281,1],[279,3,281,3],[282,0,284,0],[282,1,284,1],[282,3,284,3],[285,0,293,0],[285,1,293,1],[294,2,294,3],[296,0,297,0],[296,1,297,1],[296,2,297,2],[299,0,301,0],[299,1,301,1],[299,3,301,3],[302,0,305,0],[302,1,305,1],[306,0,309,0],[306,1,309,1],[310,0,313,0],[310,1,313,1],[314,0,317,0],[314,1,317,1],[318,0,326,0],[318,1,326,1],[327,2,327,3],[339,0,341,0],[339,1,341,1],[339,3,341,3],[342,0,345,0],[342,1,345,1],[342,3,345,3],[346,0,348,0],[346,1,348,1],[346,3,348,3],[349,0,351,0],[349,1,351,1],[349,3,351,3],[352,0,354,0],[352,1,354,1],[352,3,354,3],[355,0,357,0],[355,1,357,1],[358,0,360,0],[358,1,360,1],[358,3,360,3],[361,0,366,0],[361,1,366,1]]}
["Variable","No","Label","Notes"]
["Codername",1,"Write your full name (e.g. Kate Bowers)","This field is now free text to write your name in."]
["Reference",2,"Provide short paper reference - e.g. Welsh(2003)","Please use the following conventions:"]
[{"$nulls":3},"-   Author1 (Year)"]
[{"$nulls":3},"-   Author1 & Author2 (Year)"]
[{"$nulls":3},"-   Author1 et. al. (Year)  (for more than two authors)"]
["Intervention description",3,"Intervention description","Please provide a concise but informative description of the intervention being reviewed. For example, \"Cognitive behavioural therapy for domestic violence\""]
["OffTreatment",4,"Offender treatment type ","If you have selected Correctional in Intervention, then one or more of the following offender treatment codes may also be applicable.  Please select all that apply by clicking on each in turn.  Selecting an item more than once removes it from the list."]
[{"$nulls":3},"-    Behavioural treatment"]
[{"$nulls":3},"-    Biological treatment"]
[{"$nulls":3},"-    CBT - Cognitive-behavioural therapy"]
[{"$nulls":3},"-    Community based"]
[{"$nulls":3},"-    Counselling"]
[{"$nulls":3},"-    Educational & skills training"]
[{"$nulls":3},"-    Mentoring"]
[{"$nulls":3},"-    Other"]
[{"$nulls":3},"-    Pharmaceutical"]
[{"$nulls":3},"-    Physical challenge programmes"]
[{"$nulls":3},"-    Psychological therapy"]
["OffenderFocus",5,"Is the intervention focussed on offenders?","This is going to be a filter for the CoP toolkit. Please note that some interventions can have more than one focus so you can answer ‘yes’ to one or more of OffenderFocus, VictimFocus and PlaceFocus."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["VictimFocus",6,"Is the intervention focussed on victims?","This is going to be a filter for the CoP toolkit. Please note that some interventions can have more than one focus so you can answer ‘yes’ to one or more of OffenderFocus, VictimFocus and PlaceFocus."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["PlaceFocus",7,"Is the intervention focussed on places?","This is going to be a filter for the CoP toolkit. Please note that some interventions can have more than one focus so you can answer ‘yes’ to one or more of OffenderFocus, VictimFocus and PlaceFocus."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["TargetedOffpop",8,"Targeted offender population","This applies to offender population focused interventions only.  Use one or more of the following codes (which best describes the targeted offender population).  Please select all that apply by clicking on each in turn.  Selecting an item more than once removes it from the list."]
[{"$nulls":3},"~    Antisocial"]
[{"$nulls":3},"~    Domestic abusers"]
[{"$nulls":3},"~    Drink drivers"]
[{"$nulls":3},"~    Drug/alcohol abusers"]
[{"$nulls":3},"~    Gangs"]
[{"$nulls":3},"~    Mentally ill"]
[{"$nulls":3},"~    Other"]
[{"$nulls":3},"~    Prisoners"]
[{"$nulls":3},"~    Probationers"]
[{"$nulls":3},"~    Sex offenders"]
[{"$nulls":3},"~    Violent offenders"]
[{"$nulls":3},"~       General offending    NOTE: this should be used when mixed offending populations are used in a review"]
["TargetedNonPop",9,"Targeted non-offender population ","If there is a non-offender population type please select all that apply by clicking on each in turn.  Selecting an item more than once removes it from the list."]
[{"$nulls":3},"-   Other"]
[{"$nulls":3},"-   Places"]
[{"$nulls":3},"-   Victims"]
["TargetedPopSub",10,"Targeted offender population sub-type ","If the intervention information provided in TargetedOffPop can be further clarified or refined then use your discretion to update this code."]
["CrimeOutcomes",11,"Crime outcome measure","Please use this code to (briefly) capture information on CRIME outcomes.  In some cases multiple outcomes will be mentioned.  Please separate these using a '*'."]
["NonCrimeOutcomes",12,"Non-crime outcome measure","Please use this code to (briefly) capture information on NON-CRIME outcomes.  In some cases multiple outcomes will be mentioned.  Please separate these using a '*'."]
["OutcomeDataType",13,"Outcome measure- data type","Select all that apply by clicking on each in turn.  Selecting an item more than once removes it from the list."]
[{"$nulls":3},"-   Anti-social behaviour"]
[{"$nulls":3},"-   Arrests"]
[{"$nulls":3},"-   Calls for service"]
[{"$nulls":3},"-   Recorded crime"]
[{"$nulls":3},"-   Reoffending/recidivism (general)"]
[{"$nulls":3},"-   Self-reported offending"]
["OutcomeProperty",14,"Is the outcome a property crime type?","If crime types similar to these are reported as an outcome then code this as ‘Yes’:"]
[null,null,"0 = No","-    Aggravated burglary"]
[null,null,"1 = Yes","-    Arson"]
[{"$nulls":3},"-    Burglary dwelling"]
[{"$nulls":3},"-    Criminal damage"]
[{"$nulls":3},"-    Criminal damage endangering life"]
[{"$nulls":3},"-    Robbery"]
[{"$nulls":3},"-    Theft (person, vehicle)"]
[{"$nulls":3},"Please note that some reviews have more than one outcome so you can answer ‘yes’ to one or more of Outcomeproperty, Outcomeviolent, Outcomecriminality, Outcomedetection and Outcomeantisocial."]
["OutcomeViolent",15,"Is the outcome a violent crime type?","If crime types similar to these are reported as an outcome then code this as ‘Yes’:"]
[null,null,"0 = No","-    Aggravated burglary"]
[null,null,"1 = Yes","-    Assault without injury"]
[{"$nulls":3},"-    Attempted murder"]
[{"$nulls":3},"-    Criminal damage endangering life"]
[{"$nulls":3},"-    Grievous bodily harm/ actual bodily harm"]
[{"$nulls":3},"-    Homicide"]
[{"$nulls":3},"-    Rape "]
[{"$nulls":3},"-    Robbery"]
[{"$nulls":3},"-    Sexual Assault"]
[{"$nulls":3},"Please note that some reviews have more than one outcome so you can answer ‘yes’ to one or more of Outcomeproperty, Outcomeviolent, Outcomecriminality, Outcomedetection and Outcomeantisocial."]
["OutcomeCriminality",16,"Is the outcome a measure of criminality?","Criminality is subtly different than crime, and is worth coding for if it is applicable. If outcomes similar to these are reported then code this as ‘Yes’:"]
[null,null,"0 = No","-    Alcohol-use"]
[null,null,"1 = Yes","-    Delinquency "]
[{"$nulls":3},"-    Driving offences"]
[{"$nulls":3},"-    Driving under the influence"]
[{"$nulls":3},"-    Drug-use"]
[{"$nulls":3},"-    Rearrest"]
[{"$nulls":3},"-    Recidivism"]
[{"$nulls":3},"-    Reconviction"]
[{"$nulls":3},"Please note that some reviews have more than one outcome so you can answer ‘yes’ to one or more of Outcomeproperty, Outcomeviolent, Outcomecriminality, Outcomedetection and Outcomeantisocial."]
["OutcomeDetection",17,"Is the outcome a measure of detection rate?","If outcomes similar to these are reported then code this as ‘Yes’:"]
[null,null,"0 = No","-    Arrest rate"]
[null,null,"1 = Yes","-    Clearance rate"]
[{"$nulls":3},"Please note that some reviews have more than one outcome so you can answer ‘yes’ to one or more of Outcomeproperty, Outcomeviolent, Outcomecriminality, Outcomedetection and Outcomeantisocial."]
["OutcomeAntisocial",18,"Is the outcome a measure of anti-social behaviour? (i.e. it is not criminal but widely considered to be undesirable)","If outcomes similar to these are reported then code this as ‘Yes’:"]
[null,null,"0 = No","-    Aggressive behaviour"]
[null,null,"1 = Yes","-    Begging/panhandling"]
[{"$nulls":3},"-    Disorder"]
[{"$nulls":3},"-    Fare evasion"]
[{"$nulls":3},"-    Graffiti"]
[{"$nulls":3},"-    Noise"]
[{"$nulls":3},"-    Public urination"]
[{"$nulls":3},"Please note that some reviews have more than one outcome so you can answer ‘yes’ to one or more of Outcomeproperty, Outcomeviolent, Outcomecriminality, Outcomedetection and Outcomeantisocial."]
["NoStudies",19,"Total number of studies used in REVIEW as a whole (not just meta-analysis)","This refers to the sample of studies used in the review overall (not just those used in a meta-analysis).  "]
["EFFECT SECTION",null,"The effect section refers to the aspects of a review that reports findings, that is, what impact the intervention had on the participants (or area). Predominantly this section collects information about quantitative effects that aim to measure the impact of an intervention. There are also codes that encourage the collection of qualitative statements or assessments of impact."]
["EFFECT-E",20,"Reported overall mean effect size and number of studies in analysis","If no overall effect size is calculated for ALL the studies in the sample, but mean effect sizes are reported for subgroups then please report all of the moderator analysis results. Note: This should be numeric. If the effect is qualitative or in words please fill in code (Effectinwords)."]
["EFFECTTYPE-E",21,"Type of effect size","This is the type of effect size reported in code (EFFECT-E).  Please select from the dropdown menu. For percentage change please use the 'proportional change' option."]
[null,null,"Proportional change"]
[null,null,"Odds ratio"]
[null,null,"Standardised mean difference"]
[null,null,"Relative risk ratio"]
[null,null,"correlation coefficient"]
[null,null,"phi coefficient"]
[null,null,"unstandardised mean difference"]
[null,null,"log odds ratio"]
[null,null,"log relative risk ratio"]
["EFFLOWER-E",23,"Lower 95% confidence interval","This relates to code (EFFECT-E).  Note: This is numeric. "]
["EFFUPPER-E",22,"Upper 95% confidence interval","This relates to code (EFFECT-E).  Note: This is numeric. "]
["EFFSE-E",24,"Standard Error of mean effect size","This relates to code (EFFECT-E).  Note: This is numeric. "]
["EFFNSTUDIES-E",25,"Number of studies used in meta-analysis","This refers to the sample of studies used in the mean overall effect size reported in code (EFFECT-E).  "]
["EFFNCONTROL-E",26,"Number of participants across control groups","This refers to the number of participants (people or places) in the control group for the mean overall effect size reported in code (EFFECT-E).  "]
["EFFCONTGRP-E",27,"What did the control group consist of? ","Control groups can take many forms; they may be groups not receiving treatment, groups receiving 'business as usual' treatment or non-comparable groups.  Please briefly describe the control group - i.e. what the effect size in code (EFFECT-E) is calculated relative to."]
["EFFNACTION-E",28,"Number of participants across action groups","This refers to the number of participants (people or places) in the action (experimental) group for the mean overall effect size reported in code (EFFECT-E).  "]
["EFFMOD-E",29,"Effect-based subgroup analysis","This relates to subgroup analysis on aspects of the implementation or evaluation; such as outcome type (e.g. arrests vs. convictions), study design (e.g. interrupted time-series vs. quasi-experimental), follow-up period (i.e. point at which outcome was measured) or other methodological criteria.  Please separate items with an '*'."]
["EFFSTAT-E",30,"Statements qualifying variations in EFFMOD-E","Qualitative statements/conclusions of the analysis in EFFMOD-E. For example, the impact of study design on effect, the impact of outcome data on effect. On occasion these might be speculative, rather than involve analysis - these are still worth recording at this stage.  Please separate items with an '*'"]
["Otherbias-E",31,"Other risks of bias mentioned by the review authors","  Use this code to record any other threats of bias mentioned by the review authors.  For example, if the review authors identify risks of bias in the primary studies in the review, include those here.  Please use a '*' to separate the items."]
["Effectinwords",32,"Simple text summary of the effect","  Please fill this in even if there is a quantitative effect size.  Please separate items with an '*'."]
["EffectInterpretation1",33,"Were increases in crime, decreases in crime, or both observed across the primary studies?","This code relates to the direction of the (statistically significant) effects observed across individual primary studies.  Coders must therefore look for summary tables and/or forest plots in the review and assess these effects as a whole.  If the primary study effects (or effect sizes - ES) are not reported, or the statistical significance of primary study effects are not present then select the appropriate option.  The latter two scenarios results in no 'inner ticks and crosses' on the online toolkit."]
[null,null,"Increases in crime observed"]
[null,null,"Decreases in crime observed"]
[null,null,"Both directions observed"]
[null,null,"Primary study ES not reported"]
[null,null,"Statistical significance of ES not reported"]
["EffectInterpretation2",34,"This relates to the direction of the OVERALL effect (however calculated)","This code relates to the direction of the (statistically significant)overall effect described by the review authors.  This could be produced by meta-analysis, descriptive analysis or a narrative synthesis."]
[null,null,"Overall increase in crime"]
[null,null,"Overall decrease in crime"]
[null,null,"No overall effect"]
["Meta-analysis",35,"Did the review use meta-analysis?","To score 1, the review authors must explicitly state that they used meta-analysis, OR report an overall effect size calculated via meta-analysis techniques.  Providing descriptive statistics or another means of summarising the effect sizes is acceptable."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["EffectSizeCalculations",36,"Relates to the type of effect size/s presented in the review","Evidence syntheses present effects in various ways.  Please select the option that most closely resembles what the review authors present.  Descriptive statistics can include percentages or summary statistics (min, median, max, inter-quartile range).  Please note that options for this code are in development and if you wish to inform us of more options then please email l.tompson@ucl.ac.uk."]
[null,null,"Descriptive (e.g. %)"]
[null,null,"Inferential statistics from primary studies"]
[null,null,"Meta-analytic findings"]
[null,null,"Unclear"]
["EFFSearchstratALL-Q",37,"• A transparent and well-designed search strategy","The review is based on a comprehensive and systematic search of the literature. To score 1, the search must include both of the following components:"]
[null,null,"0 = No","a.  At least two sources for identifying potentially eligible research must be used. Sources include, for example, different electronic bibliographies, Google-type internet searches, hand searches of journals, contacting relevant researchers in the field, and reviewing the references in pertinent studies and prior reviews."]
[null,null,"1 = Yes","  i)  Two clearly specified different electronic databases (e.g., PsycInfo and Dissertation Abstracts International) can be counted as two different sources."]
[{"$nulls":3}," ii)  Meta-search engines, such as EBSCO Host, ProQuest, or Dialog count as two (or more) different sources only if the meta-analysis clearly identifies two or more sources within the meta-search engine that were searched."]
[{"$nulls":3},"iii) Google Scholar as the sole search strategy is not sufficient to meet the two-source minimum criterion."]
[{"$nulls":3},"b.  There is some indication that unpublished or “grey” literature was sought during the search, whether or not any unpublished research reports eventually met the eligibility criteria for inclusion in the meta-analysis. Unpublished and grey literature are defined here as literature that is not controlled by commercial publishers, including for example, conference presentations, government and non-governmental agency technical reports, and theses/dissertations. Meta-analyses that explicitly state that only published or peer-reviewed research was included are ineligible. Meta-analyses that explicitly or actively excluded unpublished or non-peer-reviewed research are ineligible."]
[{"$nulls":3},"Definitely rate as zero if: The review relies on only two (or less) sources of literature (this could be electronic databases or other types of literature sources) and excludes the grey literature. "]
["Effectsizeapp-Q",38,"      o Subcategory-  Calculation of appropriate effect sizes","Effect Size reporting assesses whether the meta-analysis reported an aggregate mean effect size that synthesized (averaged) effect sizes across one or more sets of multiple studies, and whether the meta-analysis also provided estimates of precision around the point estimate(s). Preference is given to meta-analyses that report mean effect sizes for the key findings along with confidence intervals around the mean, or standard errors of the mean effect size that would allow calculation of the confidence interval around the mean."]
[null,null,"0 = No","To score 1, the meta-analysis should report mean effect sizes with confidence intervals or standard errors for each analysis conducted, or at least for the key outcomes of interest."]
[null,null,"1 = Yes","Note that if a key part of the analysis/findings is related to mean effect sizes within certain subgroups (e.g., quality subgroups, population subgroups, intervention type subgroups), then it is essential that the meta-analysis report confidence intervals or standard errors for those subgroups, or at minimum, provide results from statistical tests (generally meta-regression coefficients, or a Q-test for between-group heterogeneity) comparing the difference in means across those groups. "]
[{"$nulls":3},"Meta-analysts should be transparent about how they deal with missing data, and the extent to which any effect sizes estimates were approximated from partially reported data."]
[{"$nulls":3},"Definitely rate as zero if: The meta-analysis does not report a mean effect size but reports individual study effect sizes only, or a mean effect size is reported without confidence intervals. If no meta-analysis is conducted also rate as 0."]
["EFFheterogeneity-Q",39,"     o Subcategory- The analysis of heterogeneity","Heterogeneity analysis rates a meta-analysis on whether the authors were aware of and attentive to heterogeneity (i.e., variability) in the effect size estimates from the studies in the meta-analysis. This rating specifically assesses whether authors conducted analysis to examine the heterogeneity in the distribution of effect sizes. A meta-analysis will commonly report one or more of the following heterogeneity statistics:"]
[null,null,"0 = No","-   ·τ2 — often reported as tau-squared, other names are random effects variance component, between-studies variance component, estimate of between-study variability, variance of the distribution of effect size estimates."]
[null,null,"1 = Yes","-   ·τ — often reported as tau, random effects standard deviation component, between-studies standard deviation, standard deviation of the distribution of effect size estimates."]
[{"$nulls":3},"-   ·Q — often reported as Q, χ2, Cochran’s Q, Q-total, Q-test for homogeneity, Q-test for heterogeneity. This Q refers to the weighted sum of squares of the effect sizes for all of the studies included in a meta-analysis."]
[{"$nulls":3},"-   ·I2 — often reported as I-squared, may also be called the percentage of variability in effect sizes due to heterogeneity or between-study differences."]
[{"$nulls":3},"These tests estimate whether any variation in effect sizes across studies reflect more than sampling error alone (see next variable)."]
[{"$nulls":3},"(This element is associated with the random effects and moderator variables)."]
[{"$nulls":3},"If no meta-analysis is conducted, but the review authors discuss heterogeneity (without the above statistical tests) then rate as 1."]
[{"$nulls":3},"Definitely rate as zero if: The meta-analysis does not make reference to effect size heterogeneity or reports any assessment of heterogeneity. "]
["EFFRandomodel-Q",40,"     o Subcategory- Use of a random effects model where appropriate","A study mean effect can be calculated using a Fixed Effects or a Random Effects statistical model.  For a fixed effects model, it is assumed that any differences in effect sizes across studies reflect sampling error alone (random variation).  However, if it is reasonable to assume that the differences observed across studies reflect more than this - that is, that there are “real” differences across studies due to (for example) the intervention being tested on systematically different populations or types of context - then a random effects model should be employed.  The latter type of model is more conservative as the standard errors (which include an estimate of sampling error and between study error) will be larger than those for a fixed effects model (which only includes an estimate of sampling error)."]
[null,null,"0 = No","Disagnostic tests should be first be used to assess whether the heterogeneity observed across individual studies exceeds what would be expected assuming that any differences observed across studies can be explained by sampling error alone.  These tests include the Q-test, tau, tau-squared, and I-squared.  Where the test value indicates that there is more heterogeneity than would be expected, a random effects model should be employed."]
[null,null,"1 = Yes","Studies that discuss this issue and use an appropriate model, or use both a fixed effects and random effects model and discuss the findings, are considered to have employed an appropriate statistical model, and would therefore be scored as 1.  As are those that provide a valid reason for using a fixed effects model."]
[{"$nulls":3},"Definitely rate as zero if: The meta-analysis does not clearly specify whether a random effects or fixed effect model was used to estimate mean effect sizes, and it is not possible to determine based on the reported information.   If no meta-analysis is conducted also rate as 0."]
["EFFdependency-Q",41,"     o Subcategory-  Attention to the issue of dependency","Rates a meta-analysis based on its appropriate analysis of effect sizes. The most common methods assume that all effect sizes within a given analysis are statistically independent, meaning that only one effect size per study sample should be included within a given analysis. Acceptable methods for handling statistically dependent effect sizes include (a) selecting only one effect size per study sample based on some decision criteria; (b) creating one effect size per study sample by averaging all or a selected set of effect sizes for that sample; (c) using multivariate methods that allow inclusion of multiple effect sizes per study and account for the covariance between effect sizes within a study sample; (d) using robust standard error procedures that allow inclusion of multiple effect sizes per study sample; and (e) running a series of smaller meta-analyses based on subsets of effect sizes such that only one effect size per study sample is included within a given analysis. "]
[null,null,"0 = No","Meta-analyses that have potential statistical dependencies among effect sizes and ignore them or do not describe how they were handled are assumed to produce incorrect standard errors and erroneous results for the associated statistical significance tests and confidence intervals."]
[null,null,"1 = Yes","In some cases, even if the authors do not explicitly state how dependent effect sizes were handled, it may be possible to ascertain by examining forest plots or tables that list the studies included within a given analysis. If each of the studies contributing to a given analysis appear to be unique, it is reasonable to infer that independent effect sizes were used. In other cases the forest plot or other itemizations may show multiple effect sizes for a given study reference or citation, which may or may not be appropriate (e.g., some study reports provide data for multiple independent subsamples). In those instances, the coder will need to examine the text of the meta-analysis to determine if those apparent dependencies are explained."]
[{"$nulls":3},"Definitely rate as zero if: The meta-analysis explicitly includes multiple effect sizes per study within a single analysis, and those dependencies are ignored in the analysis.  "]
["EFFweighting-Q",42,"     o Subcategory- Use of appropriate weighting of the individual effect sizes in the calculation of mean effect sizes","An appropriate weighting scheme should be applied when the mean effect size is computed.  That is, the mean effect size is calculated by taking an average across identified studies.  However, some studies (those with more reliable results) should be weighted more than others (those with less reliable results).  To do this, a good meta-analyst will weight the individual effect sizes accordingly.  Appropriate weightings include inverse variance weighting and sample size weighting (For a detailed discussion, see Lipsey and Wilson, 2001). Inappropriate weighting schemes include weighting effect sizes by quality scores or some other arbitrary author-defined function, or using no weights at all."]
[null,null,"0 = No","Definitely rate as zero if: The meta-analysis does not use inverse variance or sample size weighting, but instead does not use any weighting, or uses some other type of index, such as a quality rating, to weight some or all effect size estimates (instead of, or in combination with other appropriate weighting schemes). "]
[null,null,"1 = Yes"]
["EFFStatconvalALL-Q",43,"• High statistical conclusion validity (at least four of the preceding are necessary for a study to be considered sufficient):","This is calculated based on choices for the preceding five subcategory codes and is automatically populated on the spreadsheet."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["EFFpubbias-Q",44,"    o Subcategory- Assessment of potential publication bias","Publication Bias rates the extent to which a meta-analysis investigates the potential for publication bias in the sample of included studies. Publication bias broadly refers to the idea that published results are more likely to include large and/or statistically significant effects, whereas unpublished results are more likely to include null, small, or “negative” (i.e., opposite of what would be predicted) effects. "]
[null,null,"0 = No","The purpose of this rating is to assess whether the meta-analyst was aware of and sensitive to the possibility that publication bias could influence the results of their analysis. Note that it is not sufficient for a meta-analysis to discuss publication bias in terms of the literature search, e.g., whether unpublished results were included or an attempt was made to find them. This rating refers to whether the meta-analysis descriptively or statistically assessed the possibility of publication bias in the results (this will primarily be reported in the results section, although occasionally may be present in the methods section of the meta-analysis). There are several techniques meta-analysts may use to statistically assess the possibility of publication bias. The most common techniques include:"]
[null,null,"1 = Yes","-   funnel plot graphs or contour-enhanced funnel plot graphs"]
[{"$nulls":3},"-   regression-based tests for funnel plot asymmetry (Egger test, Peters test, Harbord test, Begg rank correlation)"]
[{"$nulls":3},"-   cumulative meta-analysis graphs"]
[{"$nulls":3},"-   selection modeling approaches"]
[{"$nulls":3},"-   trim and fill analysis"]
[{"$nulls":3},"-   statistical comparison of effect sizes for published and unpublished studies"]
[{"$nulls":3},"-   inclusion of publication status in multivariate analysis, e.g. meta-regression"]
[{"$nulls":3},"Code as a one if review authors state an intention to assess publication bias under ideal conditions (e.g. a sample of studies that would permit analysis).  Also code as one if the review authors have explored (analytically) the effect of publication bias without performing meta-analysis."]
[{"$nulls":3},"Definitely rate as zero if: The meta-analysis does not mention the possibility of publication bias in the data. "]
["EFFinterater-Q",45,"    o Subcategory- Consideration of inter-rater reliability","Inter-rate reliability assesses how the authors of the meta-analysis handled reliability of data extraction from the primary research reports. Ideally, two or more coders would extract all pieces of information from each eligible research report and reliability statistics would be used to assess coder reliability and/or consensus would be reached on all items. "]
[null,null,"0 = No","Sometimes double-coding is not feasible due to time or financial restraints, or authors may only double-code subsets of the entire meta-analysis dataset. The goal of this rating is to give higher scores to those meta-analyses that adequately assess and document reliability of study coding."]
[null,null,"1 = Yes","Code as a one if there were two or more coders for at least a sample of studies and it is reported that reliability of coding was quality assured. "]
[{"$nulls":3},"Definitely code as zero if: The meta-analysis"]
[{"$nulls":3},"(a) does not report how many coders were used to extract data and/or there is no information on inter-coder reliability,"]
[{"$nulls":3},"OR"]
[{"$nulls":3},"(b) only had one coder.  "]
[{"$nulls":3},"If no meta-analysis is conducted also rate as 0."]
["EFFoutliers-Q",46,"    o Subcategory- Consideration of the influence of statistical outliers","Outlier Analysis assesses whether the meta-analysis checks for effect size outliers in the data. Note that this item refers to outlying effect sizes included as data points in the meta-analysis, not outlying data in the primary research studies that contributed to the meta-analysis. Extreme outliers can potentially distort the overall mean effect size estimate as well as the results of other analyses. An extreme effect size with a large weight (i.e., from a large study) is more problematic than an extreme effect size with a relatively small weight. Thus, a visual outlier on a forest plot with a very large confidence interval is less problematic than one with a small confidence interval. "]
[null,null,"0 = No","Note that it is possible for a meta-analysis to have a large amount of variability or heterogeneity in the effect sizes estimates, yet have no outliers. The degree of heterogeneity in a meta-analysis therefore does not inherently provide information about the presence of outliers. Although not all meta-analyses may have effect size outliers, this rating assumes that meta-analyses are of higher quality if they check for outliers and, if found, make some attempt to assess or control their influence."]
[null,null,"1 = Yes","Definitely code as zero if: The meta-analysis does not consider outliers or  explicitly reports outlying effect sizes, but does not report how, or if, they were handled in the analysis. If no meta-analysis is conducted also rate as 0."]
["EFFriskbiasALL-Q",47,"• Sufficient assessment of the risk of bias (at least two of the preceding are necessary for sufficient consideration)","This is calculated based on choices for the preceding three subcategory codes and is automatically populated on the spreadsheet."]
[null,null,"0 = No","It will be coded as zero if: only one of the above (outliers, inter-rate reliability, and publication bias) was considered."]
[null,null,"1 = Yes"]
["EFFConstvalALL-Q",48,"• Attention to the validity of the outcome constructs, with only comparable outcomes combined AND/OR exploration of the implications of combining outcome constructs.","Different sources of data (recorded crime data, victim survey data, arrest rates, etc) have different strengths and weaknesses. These differences can generate contrasting accounts of the extent and/or patterns of crime. To score 1 the review authors must have discussed the outcome constructs and, if appropriate, only combined comparable outcomes in the analysis AND/OR explored the implications of combining difference outcome constructs. If the authors have given sufficient attention to the types of data, and decided not to do meta-analysis because the data are incomparable, then score this 1."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["EFFStudesignALL-Q",49,"• Assessment of the influence of study design for any of the following that are applicable: study type (e.g. experimental and quasi-experimental design), outcome type (such as recorded crime and victimisation surveys) and follow-up measurement periods.","This is a form of sensitivity analysis; therefore decisions made by the author during the data collection and analysis phase may influence the results and conclusions.  Efforts to explore whether those decisions had an appreciable impact on the findings assist in minimising bias.  We would expect the researchers to explore the impact of these decisions on the statistical conclusions reached through additional analysis (although they may only comment on whether this changed the results, not present the detailed analysis)."]
[null,null,"0 = No"]
[null,null,"1 = Yes","If there are NO different study designs (e.g. all RCTs) or there is only one outcome type analysed then this is not necessary- so please code as a YES.   Otherwise, if the answer is no to any of the stipulated conditions (study design, outcome type or follow-up periods) then code as NO."]
["EFFunantioutALL-Q",50,"• Assessment of the influence of unanticipated outcomes or spin-offs on the size of the effect (e.g. quantification of displacement or diffusion of benefit)","To get a YES there needs to be an explicit attempt to quantify any side-effects of intervention. Purely saying that displacement or diffusion is a possibility is not sufficient here. Unanticipated outcomes can take various forms including:"]
[null,null,"0 = No","~   Displacement"]
[null,null,"1 = Yes","~   Anticipatory benefit"]
[{"$nulls":3},"~   Residual deterrence"]
[{"$nulls":3},"~   Changes in reporting"]
[{"$nulls":3},"~   An increase in fear /anxiety levels regarding vicitimisation"]
[{"$nulls":3},"~   Changes in outcomes in other domains (improvements say in health, school attendance or on psychometric scores)"]
["EffectQSCORE",51,"Overall rating of quality of EFFECT DIMENSION","Note: this should be automatically populated when the codes feeding into it have been correctly completed."]
[null,null,"0 = Insufficient consideration of validity elements listed above"]
[null,null,"1 = Sufficient consideration of one element of validity","This will be calculated using the following rules:"]
[null,null,"2 = Sufficient consideration of two elements of validity"," 0 = none from EFFSearchstratALL-Q, EFFStatconvalALL-Q, EFFriskbiasALL-Q, EFFConstvalALL-Q, EFFStudesignALL-Q or EFFunantioutALL-Q are coded ‘Yes’"]
[null,null,"3 = Sufficient consideration of three or four elements of validity"," 1 = one from EFFSearchstratALL-Q, EFFStatconvalALL-Q, EFFriskbiasALL-Q, EFFConstvalALL-Q, EFFStudesignALL-Q or EFFunantioutALL-Q are coded ‘Yes’"]
[null,null,"4 = Sufficient consideration of five or six elements of validity"," 2 = two from EFFSearchstratALL-Q, EFFStatconvalALL-Q, EFFriskbiasALL-Q, EFFConstvalALL-Q, EFFStudesignALL-Q or EFFunantioutALL-Q are coded ‘Yes’"]
[{"$nulls":3}," 3 = three or four from EFFSearchstratALL-Q, EFFStatconvalALL-Q, EFFriskbiasALL-Q, EFFConstvalALL-Q, EFFStudesignALL-Q or EFFunantioutALL-Q are coded ‘Yes’"]
[{"$nulls":3}," 4 = five or six from EFFSearchstratALL-Q, EFFStatconvalALL-Q, EFFriskbiasALL-Q, EFFConstvalALL-Q, EFFStudesignALL-Q or EFFunantioutALL-Q are coded ‘Yes’"]
["MECHANISM SECTION",null,"The mechanism section refers to articulations of the theory of change, or the sequence of events, that occur to produce the effect. Put differently, mechanism refers to the causal pathway through which an intervention makes an impact on the participants or area. These are often referred to very superficially in the literature -e.g. the intervention was believed to deter offenders (the implication being that it was making the offender believe that they had a greater chance of being caught, which is a mechanism known as 'increasing the risk' in the situational crime prevention field)."]
["MECHLITSOURC-E",52,"Information on some of the sources cited concerning mechanisms","Please just give references here, separated by an '*'."]
["MECHMAP-E",53,"Map of possible mechanisms/ Logic Maps","When presented in graphic form insert image in ‘relevant images’ sheet of the spreadsheet.  "]
["MECHMOD-E",54,"Mechanism-based subgroup analysis","Please provide details of any subgroup analysis that was undertaken to test any mechanisms (loosely defined). For example, a meta-analysis on street lighting tested for the mechanism of community pride by using 24-hour outcome data, and the mechanism of increased visibility by comparing the night-time crime trends with the day-time crime trends. Hence the two subgroups were analysed to provide an assessment of which mechanism was more plausible. Include information on the pooled effect size and number of studies in the analysis. Where more than one subgroup analysis has been reported please separate items with an '*'."]
["MECHSTAT-E",55,"Possible mechanisms","These can be mentioned implicitly or explicity in the review. Please separate items with an '*'. For example: "]
[{"$nulls":3},"* Increases the perceived risk of detection * Increases the true probability of detection * Increases usage and hence affects real and perceived guardianship * Encourages victims to take security precautions * Directs police intervention * Increases community pride and intervention * GIves a false sense of security * Increases reporting * Causes displacement * Threatens privacy and civil liberties."]
["MECHMENT-Q",56,"• Mention of mechanisms","This might take several forms. At a minimum it might be referred to as, say, an ‘increasing effort’ mechanism, describing how a crime prevention measure is designed to produce the sought-after outcome. By contrast, a review might provide a detailed narrative of ‘how’ an intervention is hypothesised to work without necessarily using the term ‘mechanism’. "]
[null,null,"0 = No","A score of 1 is not contingent on using the exact word ‘mechanism’ but refers to the reporting of ‘how’ the intervention under review was hypothesized to work. This might be quite superficial - e.g. the intervention was believed to deter offenders."]
[null,null,"1 = Yes"]
["MECHLIT-Q",57,"• Search for literature relating to relevant mechanisms","This is just a binary yes or no variable. At this point we are purely interested in whether any attempt was made to find prior research relating to how the intervention might work. "]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["MECHPRAC-Q",58,"• Discussion with stakeholders about mechanisms and subsequent reporting of these in the review","To receive a score of 1 the review must cite whether any relevant parties have been consulted as to how the intervention under review is expected to work. "]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["MECHFORM-Q",59,"• Articulation of the theory of change, including sets of testable intermediate (or proxy) variables and patterns that would be observed in the data.","This means that there is consideration of the steps involved in firing a particular mechanism. For example- "]
[{"$nulls":3},"-   Description of the chains of event that occur between the treatment, policy or programme and the ultimate effects of interest that are produced (e.g. lighting leads to increased use of a street leading to an increase in informal surveillance and consequently a reduction in crime). "]
[null,null,"0 = No","-   Considering  the steps between a sign informing cyclists on how to lock their bikes most effectively to stands and reduced rates of cycle theft: the cyclist notices the sign – the cyclist reads the sign – the cyclist has the resources to lock their bicycle more securely – the cyclist actually locks their bicycle in the suggested way – the rate of bicycle theft falls."]
[null,null,"1 = Yes – BASIC","-   Another example is ‘Fight clubs’ that aim to reduce gang membership whereby the theory of change is as follows: martial arts or boxing clubs are used as a hook to attract young people - their needs are assessed and they are given personal development coaching or intensive one-to-one mentoring - martial arts changes their personal appearance and inculcates sportsman spirit, values of fairness, leadership skills and gender sensitivity -   development classes changes attitudes and the provision of educational and vocational opportunities increases their life chances of an alternative lifestyle away from crime/violence - all these factors aid better decision-making -  companionship and social interaction provided by the fight club reduces the attraction of gang membership. "]
[null,null,"2 = Yes - THOROUGH","In all these examples, mediator/ intermediate variables can be articulated and ultimately could also be measured. Is there increased use of the street? Do the cyclists lock their bikes more securely? Do the young people attending fight clubs exhibit a change in attitudes, behaviour and educational attainment?  A basic account of these intermediate variables would score 1.  A thorough account of these intermediate variables would score 2."]
["MECHCOL-Q",60,"• Collection and analysis of data that test whether the mediators and mechanisms are operating as expected","This is likely to be the next step on from code (MECHFORMQ). To continue with the cycle theft example, a review which scores Yes would provide evidence that data has been collected and analysed with respect to:"]
[{"$nulls":3},"-   whether cyclists did or did not notice the installed signs,"]
[{"$nulls":3},"-   whether cyclists read the signs,"]
[null,null,"1 = Yes – BASIC","-   whether their locking practice altered where signs were present, etc."]
[null,null,"2 = Yes - THOROUGH","Descriptive analysis of intermediate outcome measures that are predicted to be important in achieving crime reduction would receive a score of 1.\nMeta-analysis of intermediate outcome measures that are predicted to be important in achieving crime reduction would receive a score of 2."]
["MECHCON-Q",61,"• Conclusions that corroborate, falsify or suggest refinements to the mediator or mechanisms theories","To receive a 1 the review must critically appraise the proposed mechanism(s) in light of the review findings. For example, lets say the prevailing view is that property marking works to reduce burglary because the associated publicity installed at homes where the property is marked serves to deter burglars. Then imagine that the results of the systematic review show that burglary decreased in places that a) did not receive the property marking publicity and b) did but did not install it, then the review should challenge the accuracy of this proposed causal mechanism, and articulate other more convincing (rival) mechanism theories (such as marked property being harder to dispose of due to increased vigilance on the part of retailers).    "]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["MechanismQSCORE",62,"Overall rating of quality/ extent? of MECHANISM DIMENSION","This will be calculated using the following rules: "]
[null,null,"0 = No reference to theory - simple black box","0 = MECHMENT-Q to MECHCON-Q are coded ‘No’"]
[null,null,"1 = General statement of assumed theory","1 = MECHMENT-Q is coded ‘yes’, MECHLIT-Q and MECHPRAC-Q can be 'Yes' or 'No', MECHFORM-Q can be 'Yes - BASIC' or 'No', MECHCOL-Q and MECHCON-Q are coded ‘No’"]
[null,null,"2 = Basic tests of the assumed mechanism using variables that are at hand","2 =  MECHMENT-Q is coded ‘yes’, MECHLIT-Q and MECHPRAC-Q can be 'Yes' or 'No', MECHFORM-Q is 'Yes - BASIC', MECHCOL-Q is 'Yes - BASIC', MECHCON-Q can be 'Yes' or ‘No’"]
[null,null,"3 = Full description of the theory of change and testable predictions generated from it","3 =MECHMENT-Q is coded ‘yes’, MECHLIT-Q and MECHPRAC-Q can be 'Yes' or 'No', MECHFORM is 'Yes - THOROUGH', MECHCOL can be 'No' or 'Yes - BASIC', MECHCON-Q can be 'Yes' or ‘No’"]
[null,null,"4 = Full description of the theory of change and robust analysis of whether this is operating as expected","4 = MECHMENT-Q is coded ‘yes’, MECHLIT-Q and MECHPRAC-Q can be 'Yes' or 'No', MECHFORM is 'Yes - THOROUGH', MECHCOL is 'Yes - THOROUGH', MECHCON-Q can be 'Yes' or ‘No’"]
["MODERATOR SECTION",null,"The moderator section refers to those conditions that support the firing of the mechanism (or the causal chain of events that lead to the effect). For example, the mechanism of 'shaming' for mandatory arrest policies for domestic violence seems to be moderated by the type of population. It appears to fire for middle-class populations who have a loss of status by the public shaming, whereas it appears to backfire for other populations, who arguably have less to lose. Other moderating conditions might be geographical area, crime type, time period, age group, gender, to name a few. Remember, moderators are PRE-EXISTING CHARACTERISTICS, that cannot be modified by the implementers of the intervention, or the evaluation team. Factors that are able to be manipulated by these actors can be considered to fall more in the IMPLEMENTATION section (although there is a fine line between these)."]
["MODLITSOURC-E",63,"Information on some of the sources cited concerning moderators/contexts.","This is to provide evidence that moderator analysis was theoretically informed (i.e. done a priori). The references are not usually used in the narrative. Please just give references here, separated by an '*'."]
["MODMOD-E",64,"A Priori context-based moderator analysis/subgroup analysis (analysis testing the difference context make to outcome- theoretically driven). Include pooled effect sizes and number of studies in analysis.","Please separate items with an '*'. For example:"]
[{"$nulls":3},"Significant moderator analysis- *Crime in car parks decreased by 51% in the experimental group compared to the control *Vehicle crimes decreased by 26% in the experimental group compared to the control * Crime decrease in British scheme by 19% in the experimental group compared to the control."]
["MODMODPH-E",65,"Post-Hoc context-based moderator analysis/ subgroup analysis (analysis testing the difference context makes to outcome- done due to data availablility/ not theorectically driven/ not mentioned prior to analysis). Include pooled effect sizes and number of studies in analysis.","Please separate items with an '*'."]
["MODSTAT-E",66,"Statements qualifying contextual variations (these can be narratives)","Please separate items with an '*'. For example: "]
[{"$nulls":3},"Recognition of the moderating effect of * Type of setting- more effective in car parks than public housing, public transport or town centres *Type of crime- more effective for vehicle crimes than violence *Country- more effective in UK than elsewhere (USA, Canada, Norway and Sweden)."]
["MODEFF-E",67,"Effect sizes (or percentage changes) calculated in analysis of moderators/contexts","When moderator analysis has been undertaken, insert the effect sizes or percentage changes reported by the review authors."]
["MODMENT-Q",68,"• Mention of moderator (context)","To receive a 1 the review must mention factors that are considered causally relevant to the activation of the mechanism(s) through which the intervention under review is expected to work.  These are factors that are a pre-existing condition (i.e. age, gender of the participants) rather than a factor to do with the study.  The term causally relevant is key: reference to factors that are not reasonably expected to influence the outcome patterns (such as signs of the zodiac on responsivity to drugs) would not score a 1. "]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["MODLIT-Q",69,"• Search of literature for causally significant contexts/moderators","Note: if the researchers just cite (or self-cite) one paper as containing all the information on context: if this appears to be a definitive/ persuasive source then you can give this a score of 2. In other situations if only one source is used then this would be scored a 1."]
[null,null,"0 = No"]
[null,null,"1 = Yes – BASIC"]
[null,null,"2 = Yes - THOROUGH"]
["MODPRAC-Q",70,"• Consultation with relevant others (including practitioners) about the contextual factors which might matter.","An example would be evidence that a focus group or survey had been carried out to make sense of why the intervention worked in one setting but not another. No indication of consultation would receive a score of 0."]
[null,null,"0 = No"]
[null,null,"1 = Yes "]
["MODFORM-Q",71,"• Pre-specification of theoretically or empirically derived moderators/contexts","It is important  for meta-analysts to identify potentially relevant moderators a priori, first so that this can be captured during the data collection (coding) phase, but also to minimise stumbling across moderators at the analysis phase that might capitalise on chance."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["MODCOL-Q",72,"• Collection of data enabling moderators to be tested","Reviews might theorise about particular moderators but lack the data from the primary studies to examine these theories. This code records whether the review authors were able to collect any data to test (even vaguely) whether certain conditions moderated the effect. Here for example, we might see the collection of data on times and places that interventions were implemented, or particular characteristics of (human) participants. Any environmental variables apply here. For instance, research has found that CCTV tends to work better in car parks than in city centres. Here type of place is the environmental variable on which moderator analysis is being conducted."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["MODSUB-Q",73,"• Sub-group analysis undertaken","This looks at the way that the outcome differs for different levels of the moderator variable. It is typically done statistically, reporting for example two separate effect size for the group of CCTV interventions implemented in car parks and those implemented in city centres."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["MODSUB-Q(APRIOR)",74,"• Sub-group analysis in view of the data collection and the theoretically-derived pre-specified expected variations","Note that you have to have a Yes on items MODFORM-Q, MODCOL-Q and MODSUB-Q to get a YES on item MODSUB-Q(APRIOR)"]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["ModeratorQSCORE",75,"Overall rating of quality of MODERATOR DIMENSION","Note: this should be automatically populated when the codes feeding into it have been correctly completed."]
[null,null,"0 = No reference to relevant contextual conditions that may be necessary","This will be calculated using the following rules: "]
[null,null,"1 = Ad hoc description of possible relevant contextual conditions","0 = MODMENT-Q to MODSUB-Q(APRIOR) are coded ‘No’"]
[null,null,"2 = Tests of the effects of contextual conditions defined post hoc using variables that are at hand ","1 = MODMENT-Q is coded ‘yes’, MODLIT is coded 'No' or 'Yes-BASIC', and MODPRAC-Q to MODSUB-Q(APRIOR) are coded ‘No’ "]
[null,null,"3 = Theoretically grounded description of relevant contextual conditions ","2 = MODMENT-Q is coded ‘yes’, MODLIT is coded 'No' or 'Yes-BASIC', MODPRAC-Q is coded ‘Yes’ or ‘No’, MODFORM-Q is coded ‘Yes’ or ‘No’, MODCOL-Q and MODSUB-Q are coded ‘Yes’ and all other codes are coded ‘No'"]
[null,null,"4 = Collection and analysis of relevant data relating to theoretically grounded moderators and contexts ","3 = MODMENT-Q is coded ‘yes’, MODLIT-Q and/or MODPRAC-Q are coded ‘Yes’ (MODLIT needs to be 'Yes-THOROUGH' unless MODPRAC is also 'Yes'), MODFORM-Q is coded ‘Yes’, and  all other codes are coded ‘No’."]
[{"$nulls":3},"4 = MODMENT-Q is coded ‘yes’, MODLIT-Q and/or MODPRAC-Q are coded ‘Yes’ (MODLIT needs to be 'Yes-THOROUGH' unless MODPRAC is also 'Yes'), MODFORM-Q to MODSUB-Q(APRIOR) are coded ‘Yes’."]
[{"$nulls":3},"Assumptions for automated calculation:"]
[{"$nulls":3},"~Item MODSUB-Q(APRIOR) coded as ‘Yes’ depends on items MODFORM-Q, MODCOL-Q and MODSUB-Q being coded as ‘Yes’."]
["IMPLEMENTATION SECTION",null,"The implementation section refers to the process in turning a working programme theory into a practical successfully-running programme. They therefore, in a roundabout way, help the mechanism to fire (either directly or indirectly through affecting the moderating conditions). This can be enormously complex and there are many things that might cause implementation failure along the way (e.g. resources, logistics, legalities, community support, to name a few). Implementation factors can relate to the context in which the programme is to be implemented, those that are expected to implement the programme, those that are intended to receive it, as of course the programme itself. Factors of the programme that are able to be manipulated by the implementers or actors (e.g. study design, follow up period, outcome data, what areas/people, what blinding procedure, what 'buy-in' from front-line staff) can be considered as IMPLEMENTATION factors. Those conditions that are pre-exising (i.e. not manipulatable) are considered as MODERATORS. Implementation factors may be discussed at any point in the view but they are commonly raised in the discussion section at the end."]
["IMPLITSOURC-E",76,"Information on some of the sources cited concerning implementation.","Please just give references here, separated by an '*'."]
["IMPNECIMP-E",77,"A list/statement of key components necessary for implementation for reviewed interventions","Please separate items with an '*'. For example: "]
[{"$nulls":3},"*Implementation appeared more successful when combined with other interventions (improved lighting , fencing, better parking payment schemes, security personnel, youth inclusion)  * Active monitoring of cameras appears common practice * Success appears related to stronger public support * Public support can vary by context (e.g. higher support in car parks)"]
["IMPNECREP-E",78,"A list/statement of key components deemed necessary by the review authors for replication elsewhere","The review authors need to identify these.  Please separate items with an '*'."]
["IMPMENT-Q",79,"• Mention of implementation","  To receive a 1 the review must mention implementation in relation to the intervention or programme.  Implementation can be thought of as dynamic factors that are introduced by the intervention (e.g. to do with the implementers or the delivery of the intervention).  It is the who did what and how. Mentioning implementation can be quite superficial - it may be a passing reference that does not even count as a '1' in IMPDES-Q, for example."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["IMPDES-Q",80,"• Description of what was delivered in practice","Most crime reduction schemes involve several partners and occur in multiple stages. Miscommunication, misunderstanding, misspecification and combinations thereof at any stage in this process can result in sub-optimal results, and in extreme cases backfire effects. Understanding the process involved in turning a working theory into a practical successfully-running programme is therefore very useful information, particularly for the purposes of scale-ups and replications."]
[null,null,"0 = No","To receive a 1 a review must provide some basic details of the key stages involved in implementing a particular scheme effectively. To receive a 2, a review would provide extensive information on both the key stages but also the people involved at different stages and some of the challenges and resolutions encountered at such stages. "]
[null,null,"1 = Yes – BASIC","Note that implementation issues involve actions and conditions that are under the control of the actors involved (in other words are things that are dependent on the action of people). We are meaning here the choices and actions of the people actually responsible for producing the required output. "]
[null,null,"2 = Yes - THOROUGH"]
["IMPOBST-Q",82,"• Identification (by review authors) of enablers and obstacles encountered in attempting to implement policy, program, treatment or intervention","Between the initial formulation of a crime reduction programme and the ultimate end goal of (at a minimum) reducing crime, stands much work and many challenges. Many factors might arise which prevent or halt the progress from idea generation to crime reduction. These can relate to the context in which the programme is to be implemented, those that are expected to implement the programme, those that are intended to receive it, as of course the programme itself. For example, if the programme is alleygates to reduce domestic burglary, then a common obstacle is getting all affected residents to sign the necessary paperwork confirming they are happy for gates to be implemented behind their property. Failure to do so stalls implementation. "]
[null,null,"0 = No","By contrast, some factors might make implementation easier. If a key component of the intervention under review is to engage with high-risk youths in communities with an enduring distrust of the police, then forming coalitions with groups familiar with and trusted by the community of interest is likely to be more effective than an exclusively police-led operation. "]
[null,null,"1 = Yes – BASIC","To receive a 1 the review authors must provide some basic details of what the key obstacles and enablers to implementation might be (not rely on other sources). "]
[null,null,"2 = Yes - THOROUGH","To receive a 2, review authors need to provide extensive information on both what the key obstacles and enablers are, but also why they are so integral to the implementation process and how they might best be overcome. "]
["IMPCRSUCC-Q",81,"• Specification of what is crucial to the successful implementation of the policy, program, treatment or intervention","Crucial here refers to those things that a) are within the control of those responsible for implementing the intervention under review and b) are causally related to the activation of the mechanism(s) judged to be responsible for the sought-after outcome patterns. For example, consider an initiative to reduce copper cable theft whereby scrap metal dealers are to be visited and requested to a) demand ID from all customers and b) scan all metals they process for signs of property marking (such as smartwater under a UV light). It is well known that the price of metals is positively correlated with levels of metal theft. However, this is not something that can be controlled by any one person or agency (point a). By contrast, if it is shown that the implementers of this scheme a) failed to visit all the known scrap metal dealers and b) did not provide them all with the necessary UV lamps to check for signs of property marking, then this is something that they are responsible for (point a) and failure to do so might reasonably be expected to deactivate the mechanism through which the scheme was expected to work (i.e. deter cable thieves from attempting to sell stolen metal at scrap metal yards). "]
[null,null,"0 = No","To receive a 1 a review must provide some basic details of what the crucial actionable components of implementation are. To receive a 2, a review would provide extensive information on what the crucial and actionable components of implementation are and how best to ensure they happen. "]
[null,null,"1 = Yes – BASIC"]
[null,null,"2 = Yes - THOROUGH"]
["IMPREP-Q",82,"• Specification of what would (and would not) comprise a replication of the treatment, programme, or intervention in question.","The review authors need to identify this information. "]
[null,null,"0 = No"]
[null,null,"1 = Yes – BASIC"]
[null,null,"2 = Yes - THOROUGH"]
["ImplementationQSCORE",83,"Overall rating of quality of IMPLEMENTATION DIMENSION","Note: this should be automatically populated when the codes feeding into it have been correctly completed."]
[null,null,"0 = No account of implementation or implementation challenges","This will be calculated using the following rules: "]
[null,null,"1 = Ad hoc comments on implementation or implementation challenges","0 = IMPDES-Q to IMPREP-Q are coded ‘No’"]
[null,null,"2 = Concerted efforts to document implementation or implementation challenges","1 = IMPDES-Q is coded ‘Yes’ (basic or thorough), the rest are coded ‘No’"]
[null,null,"3 = Detailed evidence-based account of implementation or implementation challenges","2 = IMPDES-Q is coded ‘Yes’ (basic or thorough), IMPOBST-Q or IMPCRSUCC-Q are coded ‘Yes’ (basic or thorough), IMPREP-Q is coded ‘No’."]
[null,null,"4 = Complete evidence-based account of implementation or implementation challenges and specification of what would be necessary for replication elsewhere","3 = IMPDES-Q is coded ‘Yes’ (basic or thorough), and at least two of IMPOBST-Q, IMPCRSUCC-Q or IMPREP-Q are coded ‘Yes’ (basic or thorough)"]
[{"$nulls":3},"4= IMPDES-Q is coded ‘Yes’ (basic or thorough), and all three of IMPOBST-Q, IMPCRSUCC-Q or IMPREP-Q are coded ‘Yes’, with at least two as ‘Yes-thorough’"]
[{"$nulls":3},"Assumptions for automated calculation:"]
[{"$nulls":3}," ~Item IMPREP-Q coded as ‘Yes’ depends on items IMPOBST-Q or IMPCRSUCC-Q being coded as ‘Yes’."]
["ECONOMICS SECTION",null,"The economics section, predictably, refers to the costs and benefits of running the intervention. These may fall on a number of stakeholders/actors, and take a large variety of forms. Preferably these costs and benefits are monetised - that is, a cash value is associated with them - but this is not as common as we would like. Please record any mention of costs and/or benefits to any stakeholder in this section. If the review refers to another document for costs then please note that down, as sometimes that is the only source of economic information."]
["ECONINPUTS-E",84,"Quantification of inputs to the scheme","This should detail what resources, and how much of them were involved in the implementation of the program.  Please detail all inputs (e.g. staff time, hardware purchased, etc). Please separate items with an '*'.  For example, an alleygating scheme might have - *200 Police officer hours *200 alleygates *100 Local authority staff hours."]
["ECONOUTPUTS-E",85,"Quantification of outputs from the scheme","This should detail what the outputs on the ground were.  For example, in the alleygating example, this would be the number of gates installed.   Please separate items with an '*'. "]
["ECONINTENS-E",86,"Quantification of intensity (e.g. spend per unit of output)","Note if this is reported in the review.  Please separate items with an '*'. "]
["ECONdirectCOSTIMP-E",87,"Estimate of direct  financial cost of implementation","Record the direct costs of implementation incurred by those implementing the intervention.  For example, in the alleygating example, list the total cost of the police officer hours, the costs of the gates and their installation, and the local authority staff hours if these are not already included.  Indicate if costs are based on standard or national costs, or based on primary (local) data collected for the evaluation. Please separate items with an '*'. "]
["ECONCOSTIMPSUB-E",88,"Estimate of direct financial cost of implementation by subgroup","As above but by subgroup.  For example, costs might be provided for different types of context (car parks, residential areas, city centres, and so on).  This is unlikely to be mentioned. Please separate items with an '*'. "]
["ECONindirectCOST-E",89,"Estimate indirect financial costs of implementation","Record any estimated indirect costs associated with the scheme.  For example, in the allegating example, if those not in receipt of treatment had to install other security measures this would be a cost.  An increase in the fear of crime (due to the gates) might also be estimated in some evaluations and a monetized cost reported.   The latter would be an intangible cost. Please separate items with an '*'. "]
["ECONCOSTEFF-E",90,"Estimate of cost-effectiveness per unit output","If reported in the review, please report any data supplied.  If reported, cost effectiveness estimates should be reported for two or more scenarios (e.g. the status quo or two different interventions) and might include: 1) the estimated cost of reducing one crime; 2) the estimated cost of delivering one output (e.g. one hour of offender supervision).  Please also indicate if the estimates of cost were full or partial estimates (i.e. did the review include all of the costs associated with intervention or only some of them).  Please record any data provided using an '*' to separate items."]
["ECONCOSTEFSUB-E",91,"Estimate of cost-effectiveness by subgroup","As above but by subgroup, if reported.  For example, costs might be provided for different types of context (car parks, residential areas, city centres, and so on).  This is unlikely to be mentioned. Please separate items with an '*'. "]
["ECONCOSTSOURCE-E",92,"Source of the costs of crime information","List the source of the data used to monetise crime reduction benefits.  Different estimates of the cost of crime have been produced by different researchers (e.g. the Home Office).  If evaluators (or systematic reviewers) use different estimates of the cost of crime then the monetised benefits will differ across reviews, making it difficult to compare monetised benefits across reviews.  Consequently, we need to record which estimates were used to translate estimated benefits (e.g. crimes reduced) into monetised ones."]
["ECONCOSTBEN-E",93,"Estimate of cost-benefit","If reported in the review, please report any data supplied.  Cost benefit analysis compares the cost of implementation with the monetized benefit of the estimated outcomes (crime or reoffending reduced).  In a review (e.g. McDougal et al.s (2008) review of the costs of sentencing) the authors may just report the cost benefit ratio - which compares the value of the benefits realized to the cost of intervention.  They are usually expressed as A:1, where A is the realized benefit of intervention for every one pound spent (where spend may include all direct and indirect monetized costs).  Please indicate which costs were available to produce the calculations.  For example, are they partial estimates of costs or benefits, or do they include the full monetized costs of implementation and of the benefits realized.  For example, you might report that the cost benefit ratio was 2:1 (there was a monetized benefit of £2 for every £1 spent) and that the full costs of intervention and outcomes were estimated."]
["ECONCOSTBENSUB-E",94,"Estimate of cost-benefit by subgroup","As above but by subgroup.  For example, costs might be provided for different types of context (car parks, residential areas, city centres, and so on).  This is unlikely to be mentioned.  "]
["ECONdirectSETUP-Q",95,"•Collection of data to assess the direct financial revenue and set-up costs of the policy, measure, intervention or treatment, as incurred by the provider","To score 1, the review should detail the direct financial costs incurred by the provider to implement the intervention.  This should be at the review level, rather than reported by a single primary study. The costs could be local (ie for the specific intervention) but it should be clear that they detail all of the likely costs incurred, rather than being an unsubstantiated estimate."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["ECONindirectSETUP-Q",96,"Collection of data to assess the indirect financial costs of the policy, measure, intervention or treatment.","To score 1, the review would detail those additional (monetized) costs borne by victims, such as the loss of productivity, fear of crime etc.  This should be at the review level, rather than reported by a single primary study.  It might also include additional costs incurred by potential victims to avoid any victimization that might occur as a consequence of victimization (costs incurred by residents in area adjacent to a treatment area)."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
[]
["ECONINTERVPEROUT-Q",97,"•Evidence-based quantified estimate of the direct, financial costs of the intervention to the provider per unit of output (marginal costs)","To score 1, a review should document the cost of intervention by unit of output (e.g. the cost per CCTV camera), so that rather than just informing how much it would cost to replicate an existing program (e.g. with the same number of cameras) the cost of implementing (say) 10 CCTV cameras could be calculated, as could the cost of 100 (i.e. this is because the unit costs have been explicitly documented).  To be evidence-based, it should be clear that the estimates are based on data collected from a particular intervention (or set of them) rather than being an unsubstantiated estimate.  "]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["ECONDIRCOSTBENEFIT-Q",98,"•Evidence-based quantified estimate of the overall outcome-related financial benefits and costs in relation to direct financial costs to the provider","To score 1, the reviewer would need to: 1) Estimate the number of crimes (or other outcomes) prevented using an appropriate quantitative approach (using an appropriate counterfactual and analytic approach); 2) multiply this estimate by an estimate of the monetized value of those crimes; 3) estimate the direct costs of intervention (see above)."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["ECONCOSTPEROUTCOME-Q",99,"•Evidence-based quantified estimate of the direct, financial provider costs per unit of (positive and negative) outcome","To score 1, a review should provide a quantitative estimate of the financial cost per unit of estimated crime reduction outcome.  This would require the reviewer(s) to (say) estimate the number of crimes prevented - or caused in the case of negative outcomes - using an appropriate quantitative approach (using an appropriate counterfactual and analytic approach), and to divide the total financial costs of intervention by this estimate.  To be evidence-based, the data should have been collected for a specific intervention rather than being an unsubstantiated estimate."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["ECONINDIRCOSTBENE-Q",100,"•Evidence-based estimate of monetized financial and non-financial costs and benefits per unit of monetized financial and non-financial unit of intended and unintended outcome","To score 1, a review would need to: 1) Place a monetary value on the direct financial costs of intervention (to estimate the cost of intervention); 2) Estimate the number of crimes (or reoffending) directly prevented by the intervention and add or subtract any associated unintended effects (e.g. displacement or a diffusion of crime control benefits) to provide an estimate of the net effect of intervention. 3) Place a monetary value on the direct financial costs of crimes prevented (or caused) such as the cost of replacement goods etc; 4) Place a monetary value on the indirect costs of intervention (such as the costs incurred by people in the anticipation of crime due to the intervention, such as the installation of personally funded security measures, productivity loss or changes in the fear of crime); 5) apply 3)-4) to the estimates calculated in part 2). "]
[null,null,"0 = No","Elements 1 and 4 are intended to estimate the monetized cost of intervention. Elements 2-3 are intended to estimate the monetized benefits of intervention."]
[null,null,"1 = Yes"]
["ECONINDIRCBSTAKE-Q",101,"•Evidence-based estimate of the distribution by stakeholder of direct and indirect costs and benefits","To score 1, the review would need to: 1) identify the stakeholders involved in an intervention; and, 2) estimate how the costs and benefits are distributed across those stakeholders."]
[null,null,"0 = No"]
[null,null,"1 = Yes"]
["EconomicsQScore",102,"Overall rating of quality of ECONOMICS DIMENSION","Note: this should be automatically populated when the codes feeding into it have been correctly completed."]
[null,null,"0 = No mention of costs (and/or benefits) or no explicit details","1 – if, ECONdirectSETUP-Q=yes, ECONCOSTPEROUTCOME-Q, ECOINDIRCOSTBEN-Q and ECONIDIRCBSTAKE-Q can be yes or no (see below), all else no ﻿﻿"]
[null,null," 1 = Only direct or explicit costs (and/or benefits) estimated (this would allow CEA)"," 2 –if, ECONdirectSETUP-Q=yes, ECONindirectSETUP-Q=yes, ECONCOSTPEROUTCOME-Q, ECOINDIRCOSTBEN-Q and ECONIDIRCBSTAKE-Q can be yes or no (see below), all else no"]
[null,null," 2 = Direct or explicit monetized costs and benefits estimated (this would allow CEA or simple CBA)"," 3 – if, ECONdirectSETUP-Q=yes, ECONindirectSETUP-Q=yes,  ECODIRCOSTBENEFIT-Q=yes, ECONCOSTPEROUTCOME-Q, ECOINDIRCOSTBEN-Q and ECONIDIRCBSTAKE-Q can be yes or no, all else no"]
[null,null,"3 = Direct/explicit and indirect costs and benefits, and marginal costs (and/or benefits) estimated"," 4 - if, ECONdirectSETUP-Q=yes, ECONindirectSETUP-Q=yes,  ECODIRCOSTBENEFIT-Q=yes, 94=yes, ECONCOSTPEROUTCOME-Q, ECOINDIRCOSTBEN-Q and ECONIDIRCBSTAKE-Q can be yes or no"]
[null,null," 4 = Direct/explicit and indirect costs and benefits, and marginal costs (and/or benefits) estimated by bearer (or recipient)","** Codes ECONCOSTPEROUTCOME-Q to ECONIDIRCBSTAKE-Q relate to cost benefit analysis (CBA).  We have agreed that the EMMIE-E scale relates to the cost of implementation, and so these codes do not inform the EMMIE ratings.  However, it is important to know where CBA data are presented as we will include any such information in the summary screens of the toolkit.  These codes will also allow us to assess the extent to which CBA data are available in systematic reviews and what data are available."]
["CODERSNOTE",103,"Record any important coding decisions or judgement calls here"]

{"sheet":"RelevantImages","index":3,"n_rows":0,"n_cols":0,"rows_kept":0}

{"sheet":"ChangeLog","index":4,"n_rows":22,"n_cols":2,"rows_kept":22,"hidden":true}
["2014-10-30","Removed Aiden from the development team"]
["2014-10-30","Changes to EFFECT-Q codes for statistical conclusion validity issues outside of a meta-analysis."]
["2014-10-30","New codes inserted to capture direction of effects (individual and overall)"]
["2014-11-01","Data validation changed in cell AA3:12 so that the entry had to be over zero"]
["2014-11-29","Logic changed for Mechanism, Moderator and Implementation"]
["2014-11-29","New code inserted for implementation"]
["2014-11-29","Codebook updated"]
["2015-11-30","Codes renumbered"]
["2015-11-30","Developer's sheet amended"]
["2015-01-27","Extra codes added for EFFMOD-E and EFFSTAT-E"]
["2015-01-27","Codebook updated"]
["2015-01-27","Codes renumbered"]
["2015-04-01","Data validation changed in Effect Size and confidence interval cells so negative decimals permitted"]
["2015-04-01","Hyperlinks to codebook restored in some cells"]
["2015-04-01","Options for EffectInterpretation1 updated to reflect no information in primary studies"]
["2015-04-01","Codebook for EffectInterpretation1 updated"]
["2015-04-01","Codebook tidied up"]
["2015-04-01","Label changed for TargetedPopSub"]
["2015-04-01","Codebook clarified for Economics, Moderators and Implementation"]
["2016-11-01","codebook updated"]
["2016-12-04","Codes streamlined on feedback from WP1+ team, codebook updated"]
["2018-03-26","Code 48 updated in the codebook"]

{"sheet":"Dropdowns","index":5,"n_rows":14,"n_cols":24,"rows_kept":14,"hidden":true}
["EPPI codes",{"$nulls":5},"Coder name",{"$nulls":7},"Yes/No","Yes/No extended",null,"EffectQSCORE","effecttypeE","Effectinterpretation","MechanismQSCORE","ModeratorQSCORE","ImplementationQSCORE","EconomicsQScore"]
["Intervention","Offtreatment","TargetedOffpop","TargetedNonpop","Outcome3datatype",null,"Aiden Sidebottom",{"$nulls":7},"No","No",null,"0 = No sufficient consideration of validity elements listed above","Proportional change (%)","No significant effect","0 = No reference to theory - simple black box","0 = No reference to relevant contextual conditions that may be necessary","0 = No account of implementation or implementation challenges","0 = No mention of costs (and/or benefits)"]
["* Correctional ","* Behavioural treatment ","* Antisocial ","* Other ","* Anti-social behaviour ",null,"Gloria Laycock",{"$nulls":7},"Yes","Yes - BASIC",null,"1 = Sufficient consideration of one element of validity","Odds ratio","A modest but significant reduction","1 = General statement of assumed theory","1 = Ad hoc description of possible relevant contextual conditions","1 = Ad hoc comments on implementation or implementation challenges","1 = Only direct or explicit costs (and/or benefits) estimated"]
["* Community ","* Biological treatment ","* Domestic abusers ","* Places ","* Arrests ",null,"Jyoti Belur",{"$nulls":8},"Yes - THOROUGH",null,"2 = Sufficient consideration of two elements of validity","Standardised mean difference","A large and significant reduction","2 = Basic tests of the assumed mechanism using variables that are at hand","2 = Tests of the effects of contextual conditions defined post hoc using variables that are at hand ","2 = Concerted efforts to document implementation or implementation challenges","2 = Direct or explicit and indirect and implicit costs (and/or benefits) estimated"]
["* Developmental and social prevention ","* CBT - Cognitive-behavioural therapy ","* Drink drivers ","* Victims ","* Calls for service ",null,"Kate Bowers",{"$nulls":10},"3 = Sufficient consideration of three or four elements of validity","Relative risk ratio","A modest but significant increase","3 = Full description of the theory of change and testable predictions generated from it","3 = Theoretically grounded description of relevant contextual conditions","3 = Evidence-based account of levels of fidelity to program, policy or treatment plans","3 = Marginal or total or opportunity costs (and/or benefits) estimated"]
["* Drug treatment ","* Community based ","* Drug/alcohol abusers ",null,"* Recorded crime ",null,"Ken Pease",{"$nulls":10},"4 = Sufficient consideration of five or six elements of validity","Correlation coefficient","A large and significant increase","4 = Full description of the theory of change and robust analysis of whether this is operating as expected","4 = Collection and analysis of relevant data relating to theoretically grounded moderators and contexts ","4 = Analysis of structural/legal sources of departure from/fidelity to program plans","4 = Marginal or total or opportunity costs (and/or benefits) by bearer (or recipient) estimated"]
["* Educational ","* Counselling ","* Gangs ",null,"* Reoffending/recidivism (general) ",null,"Lisa Tompson",{"$nulls":11},"Phi coefficient"]
["* Policing and partnership ","* Educational & skills training ","* Mentally ill ",null,"* Self-reported offending ",null,"Lucia Summers",{"$nulls":11},"Unstandardised mean difference"]
["* Publicity ","* Mentoring ","* Other ",null,"*Self-reported victimisation",null,"Shane Johnson",{"$nulls":11},"Log odds ratio"]
["* Restorative justice ","* Other ","* Prisoners ",null,"* Victimisation survey",null,"Michelle Rogerson",{"$nulls":11},"Log relative risk ratio"]
["* Sentencing and deterrence ","* Pharmaceutical ","* Probationers ",{"$nulls":15},"Relative effect size (RES)"]
["* Situational prevention ","* Physical challenge programmes ","* Sex offenders "]
["* Other","* Psychological therapy ","* Violent offenders "]
[null,null,"* General offending "]

{"sheet":"For developers","index":6,"n_rows":2,"n_cols":20,"rows_kept":2,"hidden":true}
["Titletoshow","Offenderfocus","Victimfocus","Placefocus","Outcomeproperty","Outcomeviolent","Effect highlighted?","EffectQSCORE","Mechanism-E","MechanismQSCORE","MechanismQScore in words","Moderator-E","ModeratorQSCORE","ModeratorQScore in words","Implementation-E","ImplementationQSCORE","ImplementationQSCORE in words","Economics-E","EconomicsQSCORE","EconomicsQSCORE in words"]
[{"$f":"Data!#REF!","v":{"$error":"#REF!"}},{"$f":"Data!E5","v":0},{"$f":"Data!F5","v":0},{"$f":"Data!G5","v":0},{"$f":"Data!N5","v":0},{"$f":"Data!O5","v":0},null,{"$f":"Data!AZ5","v":""},{"$f":"Data!BA13","v":0},{"$f":"Data!BS13","v":0},null,{"$f":"Data!BT13","v":0},{"$f":"Data!CN13","v":0},null,{"$f":"Data!CQ13","v":0},{"$f":"Data!DI13","v":0},null,{"$f":"Data!DJ13","v":0},{"$f":"Data!EJ13","v":0}]