[
{
"key": "QW9F567L",
"version": 51,
"library": {
"type": "group",
"id": 314328,
"name": "Confidence Levels for Monitoring",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring",
"type": "text/html"
}
}
},
"links": {
"self": {
"href": "https://api.zotero.org/groups/314328/items/QW9F567L",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring/items/QW9F567L",
"type": "text/html"
}
},
"meta": {
"createdByUser": {
"id": 75759,
"username": "jkarl",
"name": "Jason Karl",
"links": {
"alternate": {
"href": "https://www.zotero.org/jkarl",
"type": "text/html"
}
}
},
"creatorSummary": "McDonald-Madden et al.",
"parsedDate": "2010",
"numChildren": 0
},
"data": {
"key": "QW9F567L",
"version": 51,
"itemType": "journalArticle",
"title": "Monitoring does not always count",
"creators": [
{
"creatorType": "author",
"firstName": "Eve",
"lastName": "McDonald-Madden"
},
{
"creatorType": "author",
"firstName": "Peter W.J.",
"lastName": "Baxter"
},
{
"creatorType": "author",
"firstName": "Richard A.",
"lastName": "Fuller"
},
{
"creatorType": "author",
"firstName": "Tara G.",
"lastName": "Martin"
},
{
"creatorType": "author",
"firstName": "Edward T.",
"lastName": "Game"
},
{
"creatorType": "author",
"firstName": "Jensen",
"lastName": "Montambault"
},
{
"creatorType": "author",
"firstName": "Hugh P.",
"lastName": "Possingham"
}
],
"abstractNote": "",
"publicationTitle": "Trends in Ecology & Evolution",
"publisher": "",
"place": "",
"date": "10/2010",
"volume": "25",
"issue": "10",
"section": "",
"partNumber": "",
"partTitle": "",
"pages": "547-550",
"series": "",
"seriesTitle": "",
"seriesText": "",
"journalAbbreviation": "",
"DOI": "10/b4h3gc",
"citationKey": "",
"url": "http://linkinghub.elsevier.com/retrieve/pii/S0169534710001576",
"accessDate": "2018-09-13T17:21:44Z",
"PMID": "",
"PMCID": "",
"ISSN": "01695347",
"archive": "",
"archiveLocation": "",
"shortTitle": "",
"language": "en",
"libraryCatalog": "Crossref",
"callNumber": "",
"rights": "",
"extra": "",
"tags": [],
"collections": [],
"relations": {},
"dateAdded": "2021-09-01T18:03:44Z",
"dateModified": "2021-09-01T18:03:45Z"
}
},
{
"key": "MWG3LV5K",
"version": 50,
"library": {
"type": "group",
"id": 314328,
"name": "Confidence Levels for Monitoring",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring",
"type": "text/html"
}
}
},
"links": {
"self": {
"href": "https://api.zotero.org/groups/314328/items/MWG3LV5K",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring/items/MWG3LV5K",
"type": "text/html"
}
},
"meta": {
"createdByUser": {
"id": 75759,
"username": "jkarl",
"name": "Jason Karl",
"links": {
"alternate": {
"href": "https://www.zotero.org/jkarl",
"type": "text/html"
}
}
},
"creatorSummary": "Elliot",
"parsedDate": "2013",
"numChildren": 0
},
"data": {
"key": "MWG3LV5K",
"version": 50,
"itemType": "journalArticle",
"title": "Combining Data from Probability and Non- Probability Samples Using Pseudo-Weights",
"creators": [
{
"creatorType": "author",
"firstName": "Michael R",
"lastName": "Elliot"
}
],
"abstractNote": "",
"publicationTitle": "Survey Practice; Vol 2, No 6 (2009)",
"publisher": "",
"place": "",
"date": "2013",
"volume": "",
"issue": "",
"section": "",
"partNumber": "",
"partTitle": "",
"pages": "",
"series": "",
"seriesTitle": "",
"seriesText": "",
"journalAbbreviation": "",
"DOI": "",
"citationKey": "",
"url": "http://surveypractice.org/index.php/SurveyPractice/article/view/185",
"accessDate": "",
"PMID": "",
"PMCID": "",
"ISSN": "",
"archive": "",
"archiveLocation": "",
"shortTitle": "",
"language": "",
"libraryCatalog": "",
"callNumber": "",
"rights": "",
"extra": "",
"tags": [
{
"tag": "⛔ No DOI found",
"type": 1
}
],
"collections": [],
"relations": {},
"dateAdded": "2021-09-01T18:01:13Z",
"dateModified": "2021-09-01T18:01:15Z"
}
},
{
"key": "6QLW2JA9",
"version": 50,
"library": {
"type": "group",
"id": 314328,
"name": "Confidence Levels for Monitoring",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring",
"type": "text/html"
}
}
},
"links": {
"self": {
"href": "https://api.zotero.org/groups/314328/items/6QLW2JA9",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring/items/6QLW2JA9",
"type": "text/html"
}
},
"meta": {
"createdByUser": {
"id": 75759,
"username": "jkarl",
"name": "Jason Karl",
"links": {
"alternate": {
"href": "https://www.zotero.org/jkarl",
"type": "text/html"
}
}
},
"creatorSummary": "White et al.",
"parsedDate": "2017",
"numChildren": 0
},
"data": {
"key": "6QLW2JA9",
"version": 50,
"itemType": "journalArticle",
"title": "Two scales are better than one: Monitoring multiple-use northern temperate forests",
"creators": [
{
"creatorType": "author",
"firstName": "Mark A.",
"lastName": "White"
},
{
"creatorType": "author",
"firstName": "Meredith W.",
"lastName": "Cornett"
},
{
"creatorType": "author",
"firstName": "Peter T.",
"lastName": "Wolter"
}
],
"abstractNote": "",
"publicationTitle": "Forest Ecology and Management",
"publisher": "",
"place": "",
"date": "01/2017",
"volume": "384",
"issue": "",
"section": "",
"partNumber": "",
"partTitle": "",
"pages": "44-53",
"series": "",
"seriesTitle": "",
"seriesText": "",
"journalAbbreviation": "",
"DOI": "10/gf59fr",
"citationKey": "",
"url": "http://linkinghub.elsevier.com/retrieve/pii/S0378112716307964",
"accessDate": "2017-05-11T19:00:50Z",
"PMID": "",
"PMCID": "",
"ISSN": "03781127",
"archive": "",
"archiveLocation": "",
"shortTitle": "Two scales are better than one",
"language": "en",
"libraryCatalog": "CrossRef",
"callNumber": "",
"rights": "",
"extra": "",
"tags": [],
"collections": [],
"relations": {},
"dateAdded": "2021-09-01T18:01:13Z",
"dateModified": "2021-09-01T18:01:14Z"
}
},
{
"key": "3DZ4DKUI",
"version": 47,
"library": {
"type": "group",
"id": 314328,
"name": "Confidence Levels for Monitoring",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring",
"type": "text/html"
}
}
},
"links": {
"self": {
"href": "https://api.zotero.org/groups/314328/items/3DZ4DKUI",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring/items/3DZ4DKUI",
"type": "text/html"
},
"up": {
"href": "https://api.zotero.org/groups/314328/items/BX8JN55T",
"type": "application/json"
}
},
"meta": {
"createdByUser": {
"id": 75759,
"username": "jkarl",
"name": "Jason Karl",
"links": {
"alternate": {
"href": "https://www.zotero.org/jkarl",
"type": "text/html"
}
}
},
"numChildren": 0
},
"data": {
"key": "3DZ4DKUI",
"version": 47,
"parentItem": "BX8JN55T",
"itemType": "note",
"note": "\"Active adaptive management (Shea et al. 2002) presupposes a monitoring regime that is capable of detecting environmental trends in spite of the levels of process and observation uncertainty inherent in the system. In other words, it implies that analysis of monitoring data will achieve sufficient statistical power to enable informed decision-making.\"
\n\"In the frequentist statistical paradigm, the traditional answer [to question of what is sufficient power] has been found by adherence to the 'five-eighty convention' (Di Stefano 2003).... In practive, however, this power target is rarely achieved (Sedlmeier and Gigerenzer 1989; Anderson et al. 2000; Jennions and Moller 2003\"
\n\"The simple message is that when the economic costs of Type II errors are as overwhelmingly high as they are for koalas, monitoring using a frequentist hypothesis test is not a cost-effective way of deciding whether management intervention is needed.\"
\n\"...there exists a narrow window of V:R rations - between 1 and 4.85 for the baseline parameter settings - for which an optimal alpha-value exists. Within this range, optimal alpha increases approximately linearly, and the cost saving by using the optimal alpha increases either side of V:R=1.07.
\nSee Dayton 2001 paper.
",
"tags": [],
"relations": {},
"dateAdded": "2015-01-09T04:57:55Z",
"dateModified": "2015-01-09T05:14:46Z"
}
},
{
"key": "QHWQ7D48",
"version": 33,
"library": {
"type": "group",
"id": 314328,
"name": "Confidence Levels for Monitoring",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring",
"type": "text/html"
}
}
},
"links": {
"self": {
"href": "https://api.zotero.org/groups/314328/items/QHWQ7D48",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring/items/QHWQ7D48",
"type": "text/html"
},
"up": {
"href": "https://api.zotero.org/groups/314328/items/RZ6K6VDH",
"type": "application/json"
}
},
"meta": {
"createdByUser": {
"id": 75759,
"username": "jkarl",
"name": "Jason Karl",
"links": {
"alternate": {
"href": "https://www.zotero.org/jkarl",
"type": "text/html"
}
}
},
"numChildren": 0
},
"data": {
"key": "QHWQ7D48",
"version": 33,
"parentItem": "RZ6K6VDH",
"itemType": "note",
"note": "Canadian EEM monitoring program uses an adaptive approach where monitoring results are used to modify subsequent monitoring efforts.
\n\"For changes in the level of monitoring, this statistical effect must also exceed a predefined critical effect size, and be confirmed (in the same direction) in the next cycle of monitoring. When an effect has been statistically detected, exceeds the critical effect size, and is confirmed, industry enters either “focused monitoring” to understand the extent and magnitude of the effect, or more often begins “investigation of cause” studies to determine whether the effluent is the cause of the observed difference.\"
\n\"The critical effect sizes have been defined in EEM as a percent difference in the measured end point (10% for condition and 25% for gonad size and liver size) between the exposure and reference site, relative to the reference site.11\"
\n
\n
",
"tags": [],
"relations": {},
"dateAdded": "2014-12-31T20:08:59Z",
"dateModified": "2014-12-31T23:53:07Z"
}
},
{
"key": "4SJA9UU6",
"version": 33,
"library": {
"type": "group",
"id": 314328,
"name": "Confidence Levels for Monitoring",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring",
"type": "text/html"
}
}
},
"links": {
"self": {
"href": "https://api.zotero.org/groups/314328/items/4SJA9UU6",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring/items/4SJA9UU6",
"type": "text/html"
},
"up": {
"href": "https://api.zotero.org/groups/314328/items/RZ6K6VDH",
"type": "application/json"
}
},
"meta": {
"createdByUser": {
"id": 75759,
"username": "jkarl",
"name": "Jason Karl",
"links": {
"alternate": {
"href": "https://www.zotero.org/jkarl",
"type": "text/html"
}
}
},
"numChildren": 0
},
"data": {
"key": "4SJA9UU6",
"version": 33,
"parentItem": "RZ6K6VDH",
"itemType": "note",
"note": "Optimal alpha is not the alpha-level where alpha and beta are equal, but the alpha-level that minimizes the average of alpha and beta.
\nEven with excellent experimental design, optimal alpha cannot be determined before data are collected because realized sample size and variability often differ from predicted.
\nHigher variability increases probability of type II error and results in the average of type I and II errors typically being minimized at a higher optimal alpha level.
\nUsing a standard alpha level (0.05) that is lower than the optimal alpha results in implied cost ratios (typeI/TypeII) being greater than 1 - meaning type I errors are implied to be approximately 1.5 times more important than type II errors.
\n \"Adjustments to α levels have been shown to yield optimal environmental management decisions under situations where the prior probabilities and costs of management actions and errors are known and under certain circumstances α levels as extreme as 0 or 1 have been shown to be most cost-effective (meaning that either no effect or an effect should sometimes be assumed, rather than sampling and statistically testing for an effect).5\"
",
"tags": [],
"relations": {},
"dateAdded": "2014-12-31T20:12:03Z",
"dateModified": "2014-12-31T23:53:03Z"
}
},
{
"key": "G69KJSC6",
"version": 33,
"library": {
"type": "group",
"id": 314328,
"name": "Confidence Levels for Monitoring",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring",
"type": "text/html"
}
}
},
"links": {
"self": {
"href": "https://api.zotero.org/groups/314328/items/G69KJSC6",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring/items/G69KJSC6",
"type": "text/html"
},
"up": {
"href": "https://api.zotero.org/groups/314328/items/RZ6K6VDH",
"type": "application/json"
}
},
"meta": {
"createdByUser": {
"id": 75759,
"username": "jkarl",
"name": "Jason Karl",
"links": {
"alternate": {
"href": "https://www.zotero.org/jkarl",
"type": "text/html"
}
}
},
"numChildren": 0
},
"data": {
"key": "G69KJSC6",
"version": 33,
"parentItem": "RZ6K6VDH",
"itemType": "note",
"note": "Setting appropriate relative Type I and Type II costs
\n\"there are two reasonable rules of thumb that can be used to facilitate the estimation of Type I/II error cost ratios. One, when using multiple end points α should be set so that the Type I/II cost ratios are consistent across end points, and two, if there is no clear rationale (economic, social, environ- mental, or health) for setting the costs of one type of error to be higher than the other, the relative costs of Type I and Type II error should be assumed to be equal.\"
\nUnequal costs of Type I and II errors are implied whenever alpha is set at a livel that does not minimize the average of alpha and beta.
\nHinds (REF) suggested to set alpha = beta in study design for environmental monitoring such that the risk to industry (Type I error - conclude effect when no effect is present) is equal to the risk to the environment (Type II error - conclude no effect when effect is present). This approach, however, does not minimize risk and favors \"fairness\" between industry and environment over having lowest overall risk.
\n
\n\"If relative costs of errors are known (or can be estimated), the goal should be to minimize errors given their relative costs (equivalent to minimizing overall costs of errors), rather than simply requiring error rates to be proportional to their relative costs.\"
\n\"We recommend that unless there is a clear rationale and consensus for unequal costs, they should be assumed equal, in favor of minimizing the combined probabilities of error.\"
\n\"One explanation for the lack of attention to the relative costs of Type I and II errors is that it imposes the subjective concept of valuation into a process that scientists prefer to think of as objective. However, this avoids the fundamental reality that all decision-making incorporates valuation. In some cases, economic estimates of (i) the costs of environmental degradation when real effects are not detected and (ii) the costs of efforts to mitigate or prevent effects that are not occurring may provide a strong foundation for quantitative estimates of the relative costs of Type I and II errors.\"
\n
\n
\n
",
"tags": [],
"relations": {},
"dateAdded": "2014-12-31T20:27:47Z",
"dateModified": "2014-12-31T23:52:58Z"
}
},
{
"key": "DH76XF8G",
"version": 33,
"library": {
"type": "group",
"id": 314328,
"name": "Confidence Levels for Monitoring",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring",
"type": "text/html"
}
}
},
"links": {
"self": {
"href": "https://api.zotero.org/groups/314328/items/DH76XF8G",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring/items/DH76XF8G",
"type": "text/html"
},
"up": {
"href": "https://api.zotero.org/groups/314328/items/RZ6K6VDH",
"type": "application/json"
}
},
"meta": {
"createdByUser": {
"id": 75759,
"username": "jkarl",
"name": "Jason Karl",
"links": {
"alternate": {
"href": "https://www.zotero.org/jkarl",
"type": "text/html"
}
}
},
"numChildren": 0
},
"data": {
"key": "DH76XF8G",
"version": 33,
"parentItem": "RZ6K6VDH",
"itemType": "note",
"note": "Setting Effect Size
\n\"We argue that monitoring programs that have not identified a critical effect size have a significant weakness in their rationale for any decisions that are made using monitoring data and should place a high priority on developing critical effect sizes that can be used to trigger decisions. While critical effect sizes are under development we recommend identifying small, medium, and large effect sizes, and providing optimal α levels for each of these effect sizes so that interpretations can be made based on different potentially relevant effect sizes, along with their associated error rates.\"
",
"tags": [],
"relations": {},
"dateAdded": "2014-12-31T20:33:58Z",
"dateModified": "2014-12-31T23:52:53Z"
}
},
{
"key": "9VRJ2KVZ",
"version": 33,
"library": {
"type": "group",
"id": 314328,
"name": "Confidence Levels for Monitoring",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring",
"type": "text/html"
}
}
},
"links": {
"self": {
"href": "https://api.zotero.org/groups/314328/items/9VRJ2KVZ",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring/items/9VRJ2KVZ",
"type": "text/html"
},
"up": {
"href": "https://api.zotero.org/groups/314328/items/9QW6X23Z",
"type": "application/json"
}
},
"meta": {
"createdByUser": {
"id": 75759,
"username": "jkarl",
"name": "Jason Karl",
"links": {
"alternate": {
"href": "https://www.zotero.org/jkarl",
"type": "text/html"
}
}
},
"numChildren": 0
},
"data": {
"key": "9VRJ2KVZ",
"version": 33,
"parentItem": "9QW6X23Z",
"itemType": "note",
"note": "\"One of the most obvious problems with null hypothesis significance testing in ecology is the dogmatic adherence to the arbitrary α level of 0.05 as the significance threshold for decision-making. Two consequences of always applying this arbitrary standard are (1) the decoupling of statistical significance (or lack thereof) and biological significance (or lack thereof) and (2) radical inconsistencies in statistical power to detect biologically relevant effects between studies (ranging from near 0% to near 100%).\"
\n
\n\"If a researcher can quantify 2 things that I believe should always be important considerations for any ecological research question; (1) the level of effect that would be considered biologically meaningful if it were to exist, and (2) the relative seriousness of Type I vs. Type II errors, it becomes possible to set an optimal, study-specific, α level for decision-making that minimizes either the combined probabilities or costs of both Type I errors and biologically relevant Type II errors.\"
\n
\n\"... it should be noted that implicit and unexamined decisions about biologically meaningful effect sizes and the relative importance of Type I and II error are made when α is set at 0.05. You can’t avoid making these decisions, you can only ignore that you’ve made them.\"
\n
",
"tags": [],
"relations": {},
"dateAdded": "2014-12-15T22:04:33Z",
"dateModified": "2014-12-31T23:52:45Z"
}
},
{
"key": "A294FVFC",
"version": 33,
"library": {
"type": "group",
"id": 314328,
"name": "Confidence Levels for Monitoring",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring",
"type": "text/html"
}
}
},
"links": {
"self": {
"href": "https://api.zotero.org/groups/314328/items/A294FVFC",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring/items/A294FVFC",
"type": "text/html"
},
"up": {
"href": "https://api.zotero.org/groups/314328/items/IXCP6I8G",
"type": "application/json"
}
},
"meta": {
"createdByUser": {
"id": 75759,
"username": "jkarl",
"name": "Jason Karl",
"links": {
"alternate": {
"href": "https://www.zotero.org/jkarl",
"type": "text/html"
}
}
},
"numChildren": 0
},
"data": {
"key": "A294FVFC",
"version": 33,
"parentItem": "IXCP6I8G",
"itemType": "note",
"note": "Interpretation of results with optimal alpha
\n\"So, if the objective is to minimize the probability of making an error (Type I or II) – and there is rarely, if ever, a different and rational objective – then the author made the wrong conclusion regardless of his choice of effect size, and should have concluded that there was an insufficient number of samples to be able to make any strong conclusion, but that there was better evidence to for a difference between lake types than for no difference between lake types.\"
\n
\nThis interpretation is significant. Conclusion was that more samples were needed, but that the data suggested a difference. This kind of interpretation would be helpful for broad-scale monitoring studies.
",
"tags": [],
"relations": {},
"dateAdded": "2014-12-31T20:42:23Z",
"dateModified": "2014-12-31T23:52:35Z"
}
},
{
"key": "NRTU8W55",
"version": 33,
"library": {
"type": "group",
"id": 314328,
"name": "Confidence Levels for Monitoring",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring",
"type": "text/html"
}
}
},
"links": {
"self": {
"href": "https://api.zotero.org/groups/314328/items/NRTU8W55",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/confidence_levels_for_monitoring/items/NRTU8W55",
"type": "text/html"
},
"up": {
"href": "https://api.zotero.org/groups/314328/items/IXCP6I8G",
"type": "application/json"
}
},
"meta": {
"createdByUser": {
"id": 75759,
"username": "jkarl",
"name": "Jason Karl",
"links": {
"alternate": {
"href": "https://www.zotero.org/jkarl",
"type": "text/html"
}
}
},
"numChildren": 0
},
"data": {
"key": "NRTU8W55",
"version": 33,
"parentItem": "IXCP6I8G",
"itemType": "note",
"note": "\n
\n
\n
the logical decision-making significance threshold, a, should be the value that minimizes the probability, or occasionally, the cost of making any relevant error.
\n
\n
\n
\n
Considering critical effect sizes and the relative consequences of Type I and Type II errors should not be perceived as an unnecessary extra step of statistical testing that can be avoided, because decisions about these factors are unwittingly made using the traditional approach of setting a = 0.05.
\n
\n
\n
\n
\n
\n
\n
Turning a blind eye to Type II errors, in favor of controlling a Type I error rate, allows b to fluctuate as a function of the sample size and variability.
\n
\n
\n
\n
Current practice involves a bias against accepting a falsehood (Type 1 error), where rejecting a truth (Type II error) is regarded as ‘‘healthy scepticism’’ [6], implying that there is much lower cost associated with Type II errors relative to the cost associated with Type I error [7].
\n
\n
\n
\n
The acceptability or magnitude of the potential for Type II errors should be an area of concern for all researchers, no matter the type of study performed. There are many cases where we believe that failing to detect a real effect, should be considered at least as serious, if not more serious than falsely detecting a non- existent effect. Should natural resource industries be encouraged to design low power environmental impact studies that would never have any chance of detecting anything but extreme environmental impacts using a=0.05?
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
in almost all contexts, the goal of statistical testing is to aid us in making conclusions that limit the probabilities of making mistakes, whether they be Type I or II errors. We think a strong case can be made that in most studies (and perhaps all) a should be set with the objective of either minimizing the combined probabilities of making Type I or Type II errors at a critical effect size, or minimizing the overall cost associated with Type I and Type II errors given their respective probabilities.
\n
\n
\n
\n
\n
The optimal a level for any null hypothesis significance test depends on i) the prior probabilities that the null and alternate hypotheses are true, which are typically unknown but can therefore be assumed to be equal under Laplace’s principle of indifference [9], ii) the relative costs of Type I and II errors, which are also often unknown but are assumed equal when the goal is to minimize combined probabilities of Type I and Type II error, iii) the critical effect size, iv) the sample size of the study, and v) the variability in the data.
\n
\n
\n
\n
\n
Due to the nonlinear but negative and monotonic nature of the relationship between a and b (Figure 1), it is possible, through iterative examination of v over a range of a values, to identify a unique combination of a and b that minimizes the combined probability of Type I and Type II error (v) for a desired critical effect size, in a study with a given sample size
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
Setting decision-making thresholds in a way that accounts for the relative costs of Type I and Type II errors is an area of enquiry that has been largely unexplored in many scientific fields and it is particularly relevant to research where the costs of Type I and II errors are seen as potentially ‘estimable’ (e.g. environmental effects monitoring, pharmaceutical testing, or disease treatment efficacy). Minimizing overall cost of error implies that while making conclusions based on evidence is still a priority, some of the overall confidence in the conclusion should be sacrificed to make it more likely that when errors do occur, they are the least costly type of error (see Field et al. [13] for a discussion of this technique for environmental monitoring and management).
\n
\n
\n
\n
\n
If there is no clear rationale for a specific effect size, multiple critical effect sizes could be used and results could be presented for ‘small’, ‘intermediate’ and ‘large’ critical effect sizes.
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n