Students do not learn more from professors with higher student evaluation of teaching (SET) ratings.
\nPrevius meta-analyses of SET/learning correlations in multisection studies are not interprettable.
\nRe-analyses of previous meta-analyses of multisection studies indicate that SET ratings explain at most 1% of variability in measures of student learning.
\nNew meta-analyses of multisection studies show that SET ratings are unrelated to student learning.
\nStudent evaluation of teaching (SET) ratings are used to evaluate faculty's teaching effectiveness based on a widespread belief that students learn more from highly rated professors. The key evidence cited in support of this belief are meta-analyses of multisection studies showing small-to-moderate correlations between SET ratings and student achievement (e.g., 0095 and 0100; Feldman, 1989). We re-analyzed previously published meta-analyses of the multisection studies and found that their findings were an artifact of small sample sized studies and publication bias. Whereas the small sample sized studies showed large and moderate correlation, the large sample sized studies showed no or only minimal correlation between SET ratings and learning. Our up-to-date meta-analysis of all multisection studies revealed no significant correlations between the SET ratings and learning. These findings suggest that institutions focused on student learning and career success may want to abandon SET ratings as a measure of faculty's teaching effectiveness.
\n© 2016 Elsevier Ltd. All rights reserved.
", "contentType": "text/html", "charset": "utf-8", "filename": "S0191491X16300323.html", "md5": "6613ce918a7842f66a2c1152751e742b", "mtime": 1474488949654, "tags": [], "relations": {}, "dateAdded": "2016-09-21T20:15:48Z", "dateModified": "2016-09-21T20:17:00Z" } }, { "key": "WPV84ES2", "version": 27, "library": { "type": "group", "id": 367123, "name": "Higher Education Topics and MCCCD", "links": { "alternate": { "href": "https://www.zotero.org/groups/higher_education_topics_and_mcccd", "type": "text/html" } } }, "links": { "self": { "href": "https://api.zotero.org/groups/367123/items/WPV84ES2", "type": "application/json" }, "alternate": { "href": "https://www.zotero.org/groups/higher_education_topics_and_mcccd/items/WPV84ES2", "type": "text/html" } }, "meta": { "createdByUser": { "id": 1372082, "username": "sethgoodman", "name": "Seth Goodman", "links": { "alternate": { "href": "https://www.zotero.org/sethgoodman", "type": "text/html" } } }, "creatorSummary": "Uttl et al.", "numChildren": 1 }, "data": { "key": "WPV84ES2", "version": 27, "itemType": "journalArticle", "title": "Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related", "creators": [ { "creatorType": "author", "firstName": "Bob", "lastName": "Uttl" }, { "creatorType": "author", "firstName": "Carmela A.", "lastName": "White" }, { "creatorType": "author", "firstName": "Daniela Wong", "lastName": "Gonzalez" } ], "abstractNote": "Student evaluation of teaching (SET) ratings are used to evaluate faculty's teaching effectiveness based on a widespread belief that students learn more from highly rated professors. The key evidence cited in support of this belief are meta-analyses of multisection studies showing small-to-moderate correlations between SET ratings and student achievement (e.g., Cohen, 1980, 1981; Feldman, 1989). We re-analyzed previously published meta-analyses of the multisection studies and found that their findings were an artifact of small sample sized studies and publication bias. Whereas the small sample sized studies showed large and moderate correlation, the large sample sized studies showed no or only minimal correlation between SET ratings and learning. Our up-to-date meta-analysis of all multisection studies revealed no significant correlations between the SET ratings and learning. These findings suggest that institutions focused on student learning and career success may want to abandon SET ratings as a measure of faculty's teaching effectiveness.", "publicationTitle": "Studies in Educational Evaluation", "publisher": "", "place": "", "date": "", "volume": "", "issue": "", "section": "", "partNumber": "", "partTitle": "", "pages": "", "series": "", "seriesTitle": "", "seriesText": "", "journalAbbreviation": "Studies in Educational Evaluation", "DOI": "10.1016/j.stueduc.2016.08.007", "citationKey": "", "url": "http://www.sciencedirect.com/science/article/pii/S0191491X16300323", "accessDate": "2016-09-21T20:15:42Z", "PMID": "", "PMCID": "", "ISSN": "0191-491X", "archive": "", "archiveLocation": "", "shortTitle": "Meta-analysis of faculty's teaching effectiveness", "language": "", "libraryCatalog": "ScienceDirect", "callNumber": "", "rights": "", "extra": "", "tags": [ { "tag": "Evaluation of faculty", "type": 1 }, { "tag": "Meta-analysis of student evaluation of teaching", "type": 1 }, { "tag": "Multisection studies", "type": 1 }, { "tag": "SET and learning correlations", "type": 1 }, { "tag": "Teaching effectiveness", "type": 1 }, { "tag": "Validity", "type": 1 } ], "collections": [], "relations": {}, "dateAdded": "2016-09-21T20:15:42Z", "dateModified": "2016-09-21T20:15:42Z" } }, { "key": "RUAAKMKC", "version": 26, "library": { "type": "group", "id": 367123, "name": "Higher Education Topics and MCCCD", "links": { "alternate": { "href": "https://www.zotero.org/groups/higher_education_topics_and_mcccd", "type": "text/html" } } }, "links": { "self": { "href": "https://api.zotero.org/groups/367123/items/RUAAKMKC", "type": "application/json" }, "alternate": { "href": "https://www.zotero.org/groups/higher_education_topics_and_mcccd/items/RUAAKMKC", "type": "text/html" }, "up": { "href": "https://api.zotero.org/groups/367123/items/ZAMGDESF", "type": "application/json" } }, "meta": { "createdByUser": { "id": 1372082, "username": "sethgoodman", "name": "Seth Goodman", "links": { "alternate": { "href": "https://www.zotero.org/sethgoodman", "type": "text/html" } } } }, "data": { "key": "RUAAKMKC", "version": 26, "parentItem": "ZAMGDESF", "itemType": "attachment", "linkMode": "imported_url", "title": "Snapshot", "accessDate": "2016-09-21T20:12:57Z", "url": "https://www.insidehighered.com/news/2016/09/21/new-study-could-be-another-nail-coffin-validity-student-evaluations-teaching", "note": "\n
New study adds to evidence that student reviews of professors have limited validity.
\nA number of studies suggest that student evaluations of teaching are unreliable due to various kinds of biases against instructors. (Here’s one addressinggender.) Yet conventional wisdom remains that students learn best from highly rated instructors; tenure cases have even hinged on it.
\nWhat if the data backing up conventional wisdom were off? A new study suggests that past analyses linking student achievement to high student teaching evaluation ratings are flawed, a mere “artifact of small sample sized studies and publication bias.”
\n“Whereas the small sample sized studies showed large and moderate correlation, the large sample sized studies showed no or only minimal correlation between [student evaluations of teaching, or SET] ratings and learning,” reads the study, in press with Studies in Educational Evaluation. “Our up-to-date meta-analysis of all multisection studies revealed no significant correlations between [evaluation] ratings and learning.”
\nThese findings “suggest that institutions focused on student learning and career success may want to abandon SET ratings as a measure of faculty's teaching effectiveness,” the study says.
\nThe paper considered end-of-course evaluations, not arguably more subjective ratings found on ratings websites.
\nAuthors of the new paper scrutinized data taken from seven studies that have been cited over time as evidence of the effectiveness of student evaluations. Some of the data, for example, come from a 1981 meta-analysis of multisection validity studies. That analysis, based on 41 studies reporting on 68 multisection courses, found a significant link between overall instructor course rating and student achievement, especially on measures of skill and structure. It endorsed student ratings as valid measures of teacher effectiveness.
\nYet, according to the new analysis, that paper and others like it “suffer from multiple critical methodological flaws that render their conclusions unwarranted.” Namely, the studies fail to do some or all of the following: provide basic information about the primary-level data, such as effect and sample size; ensure the data’s accuracy, such as by checking how they’re coded; or, perhaps most importantly, consider small sample size bias. The latter occurs when statistical results that may not be representative of the sample as a whole are gathered or reported in such a way that shows significant -- and therefore more likely to be published -- results.
\nThe 1981 study, for example, did briefly consider sample size, in terms of course sections, but reported it was not a significant factor. A few pages later, the same study dismissed reviewers “concerned that rating/achievement correlations vary according to the number of sections used in the study,” but then somewhat inexplicably said a “number of sections correlated significantly with the absolute value of effect size.\" Correlation size was not reported.
\nA rerunning of that study’s original, available data found that the number of sections included in multisection studies was generally small, with the number of multisection studies based on as few as five sections, and that “many impossibly high correlations (r > 0.90) were obtained in multisection studies with a small number of sections.” It also found that the majority of reported rating-achievement correlations were not statistically significant, and that the magnitude of evaluation-achievement correlations decreased for larger-sized studies in a predictable pattern.
\nThe study says that the best evidence -- its own meta-analysis of SET-learning correlations when prior learning and ability are taken into account -- indicates that the SET-learning correlation is actually zero, and that it’s “astonishing” that poor data have driven the conversation around evaluations for some 30 years. The paper advises universities to begin giving teaching evaluations appropriate “weight” in personnel and other decisions.
\n“The entire notion that we could measure professors' teaching effectiveness by simple ways such as asking students to answer a few questions about their perceptions of their course experiences, instructors' knowledge and the like seems unrealistic given well-established findings from cognitive sciences such as strong associations between learning and individual differences including prior knowledge, intelligence, motivation and interest,” the paper says. “Individual differences in knowledge and intelligence are likely to influence how much students learn in the same course taught by the same professor.”
\nSmall sample size bias concerns aren’t unique to student evaluations of teaching -- it’s a concern in neuroscience, for example, and many other fields. But the new analysis is one more reason for critics to question the validity of student evaluations of teaching as effective measures of quality. A recent Stanford University investigation of meta-analyses also found them to be problematic. \"Currently, there is massive production of unnecessary, misleading, and conflicted systematic reviews and meta-analyses,\" that paper says. \"Instead of promoting evidence-based medicine and health care, these instruments often serve mostly as easily produced publishable units or marketing tools.\"
\n“Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related” was written by Bob Uttl, professor of psychology at Mount Royal University; Carmela A. White, a graduate student in psychology at the University of British Columbia; and Daniela Wong Gonzalez, a graduate student at the University of Windsor, all in Canada. Most of the studies analyzed were based on U.S. data.
\nPhilip B. Stark, associate dean of the Division of Mathematical and Physical Sciences and a professor of statistics at Stanford, is a vocal critic of teaching evaluations used as high-stakes measures of teaching effectiveness (he did not write the recent study on meta-analysis). He said Uttl's and his colleagues' paper \"pays much more attention than usual to the quality of the underlying studies, and gives a circumspect review of previous meta-analyses.\"
\nGiven what \"the best randomized, controlled experiments have shown, it is not surprising that this study finds no meaningful correlation between SET and learning,\" he said. And given the \"strong association between SET and instructor gender, this adds evidence to the argument that institutions that care about learning should abandon SET as a measure of teaching effectiveness.\"
\nUttl said that contrary to popular belief, \"the multisection studies do not support validity of SET ratings as measure of faculty’s teaching effectiveness. They indicate that students do not learn more from professors with higher SET ratings.\"
\n", "contentType": "text/html", "charset": "utf-8", "filename": "new-study-could-be-another-nail-coffin-validity-student-evaluations-teaching.html", "md5": "62f7438699766496ff195b2757959477", "mtime": 1474488778795, "tags": [], "relations": {}, "dateAdded": "2016-09-21T20:12:57Z", "dateModified": "2016-09-21T20:15:11Z" } }, { "key": "ZAMGDESF", "version": 23, "library": { "type": "group", "id": 367123, "name": "Higher Education Topics and MCCCD", "links": { "alternate": { "href": "https://www.zotero.org/groups/higher_education_topics_and_mcccd", "type": "text/html" } } }, "links": { "self": { "href": "https://api.zotero.org/groups/367123/items/ZAMGDESF", "type": "application/json" }, "alternate": { "href": "https://www.zotero.org/groups/higher_education_topics_and_mcccd/items/ZAMGDESF", "type": "text/html" } }, "meta": { "createdByUser": { "id": 1372082, "username": "sethgoodman", "name": "Seth Goodman", "links": { "alternate": { "href": "https://www.zotero.org/sethgoodman", "type": "text/html" } } }, "numChildren": 1 }, "data": { "key": "ZAMGDESF", "version": 23, "itemType": "webpage", "title": "New study could be another nail in the coffin for the validity of student evaluations of teaching", "creators": [], "abstractNote": "New study adds to evidence that student reviews of professors have limited validity.", "websiteTitle": "", "websiteType": "", "date": "", "publisher": "", "place": "", "DOI": "", "citationKey": "", "url": "https://www.insidehighered.com/news/2016/09/21/new-study-could-be-another-nail-coffin-validity-student-evaluations-teaching", "accessDate": "2016-09-21T20:12:55Z", "shortTitle": "", "language": "", "rights": "", "extra": "", "tags": [], "collections": [], "relations": {}, "dateAdded": "2016-09-21T20:12:55Z", "dateModified": "2016-09-21T20:12:55Z" } }, { "key": "RNMXI2U6", "version": 20, "library": { "type": "group", "id": 367123, "name": "Higher Education Topics and MCCCD", "links": { "alternate": { "href": "https://www.zotero.org/groups/higher_education_topics_and_mcccd", "type": "text/html" } } }, "links": { "self": { "href": "https://api.zotero.org/groups/367123/items/RNMXI2U6", "type": "application/json" }, "alternate": { "href": "https://www.zotero.org/groups/higher_education_topics_and_mcccd/items/RNMXI2U6", "type": "text/html" } }, "meta": { "createdByUser": { "id": 1372082, "username": "sethgoodman", "name": "Seth Goodman", "links": { "alternate": { "href": "https://www.zotero.org/sethgoodman", "type": "text/html" } } }, "numChildren": 1 }, "data": { "key": "RNMXI2U6", "version": 20, "itemType": "webpage", "title": "9-ways-colleges-should-support-underrepresented-students.pdf", "creators": [], "abstractNote": "", "websiteTitle": "", "websiteType": "", "date": "", "publisher": "", "place": "", "DOI": "", "citationKey": "", "url": "https://s3-us-west-2.amazonaws.com/production.tcf.org/app/uploads/2016/05/18185630/9-ways-colleges-should-support-underrepresented-students.pdf", "accessDate": "2016-09-21T19:35:03Z", "shortTitle": "", "language": "", "rights": "", "extra": "", "tags": [], "collections": [], "relations": {}, "dateAdded": "2016-09-21T19:36:05Z", "dateModified": "2016-09-21T19:36:05Z" } } ]