THIS DOI METADATA WAS LAST UPDATED: 2024-01-10 22:48
Full Title
SOCIOLOGIA E POLITICHE SOCIALI
Publisher
FrancoAngeli
ISSN
1591-2027 (Printed Journal)
1972-5116 (Online Journal)
Journal Issue Number
2
Journal Issue Designation
2
Journal Issue Date (YYYY/MM)
2015/09
Full Title
Severità di giudizio: dinamiche valutative nell'area della sociologia nella VQR 2004-2010
By (author)
First Page
9
Last Page
43
Language of text
Italian
Publication Date
2015/09
Copyright
2015 FrancoAngeli srl
Introduction or preface
The results of the Italian Quality Research Assessment VQR 2004–10 in the area of Sociology were characterized by the scarcity of excellent ratings and a lack of consensus among referees. These two aspects, which affected the assessment of all social sciences and humanities, descend directly from the use of the peer review as a method of evaluation. As a matter of fact, peer review, although irreplaceable, poses different issues on which critical thinking is needed. In this paper, I aim to contribute to this discussion. Firstly, I focus on the system framework, i.e. an assessment exercise where in total almost 200,000 scientific publications from universities and research institutes have been analyzed. Then, the rules of the game have been reviewed and also the specific characters of the macro-area "14" where Sociology is located. The judgments formulated by the referees and the level of inter-rater agreement were descriptively considered, considering the main features of the referees, the publications and the authors. Finally, some possible interpretations are outlined together with recommendations for future improvements.
Unstructured Citation
ANVUR. 2011. Bando di partecipazione, Valutazione della Qualità della Ricerca 2004-2010 (VQR 2004-2010).
Unstructured Citation
— 2012a. Criteri per la valutazione dei prodotti di ricerca del Gruppo di Esperti della Valutazione dell’area 14 (GEV 14), Valutazione della Qualità della Ricerca 2004-2010.
Unstructured Citation
— 2012b. Documento di lavoro sulla classificazione delle riviste scientifiche italiane dell’area Scienze politiche e sociali del Gruppo di Esperti della Valutazione dell’area 14 (GEV 14), Valutazione della Qualità della Ricerca 2004-2010.
Unstructured Citation
— 2012c. Potenzialità e limiti della analisi bibliometrica nelle aree umanistiche e sociali.
Unstructured Citation
— 2012d. Rapporto finale di area. Gruppo di Esperti della Valutazione dell’Area 14 (GEV14), Valutazione della Qualità della Ricerca 2004-2010.
Unstructured Citation
— 2012e. Rapporto finale, Valutazione della Qualità della Ricerca 2004-2010
Unstructured Citation
— 2013. Criteri di assegnazione della classe di merito nel caso di valutazioni peer review con valutazioni non coincidenti da parte dei referee.
Unstructured Citation
Blasi, B. 2013. I prodotti dei Sociologi delle università italiane nella Valutazione della Qualità della Ricerca 2004-2010. Relazione presentata al X Convegno dell’Associazione Italiana di Sociologia “La qualità del sapere sociologico”, Firenze, 11 Ottobre 2013
Unstructured Citation
Bertocchi, G. and Gambardella, A. and Jappelli, T. and Nappi C. A. and Peracchi F. 2015. Bibliometric evaluation vs. informed peer review: Evidence from Italy.Research Policy, 44 (2), pp. 451–466.
Unstructured Citation
Chubin, D. E. and Hackett, E. J. 1990. Peerless science: peer review and U.S. science policy, Albany State: University of New York Press.
Unstructured Citation
CIVR. 2007b. Relazione finale. Valutazione Triennale della Ricerca 2001-2003
Unstructured Citation
— 2007a. Relazione finale di Area. Panel: 14 - Scienze politiche e sociali. Valutazione Triennale della Ricerca 2001-2003 (VTR 2001-2003).
Unstructured Citation
Cicero, T. and Malgarini, M. and Nappi C. A. and Peracchi F. 2013. Bibliometric and peer review methods for research evaluation: a methodological appraisement. MPRA Paper No. 50470 http://mpra.ub.uni-muenchen.de/50470/
Unstructured Citation
Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, pp. 37-46
https://doi.org/10.1177/001316446002000104
Unstructured Citation
Crane, D. 1976. Reward Systems in Art, Science and Religion. American Behavioral Scientist, 19 (6), pp. 719–735.
Unstructured Citation
Dinges, M. 2006. Pressure, deception and peer review. Plattform Fteval, 28, pp. 11–22.
Unstructured Citation
Hackett, E. J. and Chubin, D. E. 2003. Peer Review for the 21st Century: Applications to Educational Research. National Research Council Workshop, Washington. 25 February.
Unstructured Citation
Hicks, D. 2004. The four literatures of social science in Moed, H. and Glänzel, W. and Schmoch, U. (eds) 2004. Handbook of Quantitative Social Science and Technology Research. Dordrecht: Kluwer Academic Publishers.
Unstructured Citation
Horrobin, D. F. 1990. The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical
Association, 263(10), pp. 1438-1441.
https://doi.org/10.1001/jama.1990.03440100162024
Unstructured Citation
Knorr Cetina, K. 1999. Epistemic Cultures: How the Sciences Make Knowledge. Cambridge MA: Harvard University Press.
Unstructured Citation
Lamont, M. 2009. How Professor Think: Inside the Curious World of Academic Judgment. Cambridge MA: Harvard University Press.
https://doi.org/10.4159/9780674054158
Unstructured Citation
Latour, B. and Woolgar, S.1979. Laboratory Life: the social construction of scientific facts. Beverly Hills: Sage Publications.
Unstructured Citation
Mahoney, M. J. 1977. Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1, pp. 161–75.
Unstructured Citation
Malgarini, M. and Cicero, T. and Benedetto S. 2014. Research quality, characteristics of publications and sociodemographic features of Universities and Researchers: evidence from the Italian VQR 2004-2010 evaluation exercise. Proceedings of the science and technology indicators conference 2014 Leiden. http://sti2014.cwts.nl/Program
Unstructured Citation
Martin, B. R. and Irvine J. 1983. Assessing basic research: some partial indicators of scientific progress in radio astronomy. Research Policy, 12, pp. 61–90. Merton, R. K. 1985. Der Matthäus-Effekt in der Wissenschaft. in (ders.) Entwicklung und Wandel von Forschungsinteressen. Aufsätze zur Wissenschaftssoziologie, pp. 147- 171, Suhrkamp.
Unstructured Citation
Peters, D.P. and Ceci, S.J. 1982. Peer-review practices of psychological journals: the fate of published articles, submitted
again. Behavioral and Brain Sciences, 5, pp. 187-195.
https://doi.org/10.1017/S0140525X00011183
Unstructured Citation
Rinia, E. J., and van Leeuwen, T. N., and van Vuren, H. G. and van Raan, A. F. J. 2001. Influence of interdisciplinarity on peer review and bibliometric evaluations in physics research. Research Policy, 30, pp. 357–361
Unstructured Citation
Reyneri, E. Is Italian sociology in the doldrums? Una breve nota sulla produzione scientifica dei sociologi italiani nella VQR 2004-2010.
Unstructured Citation
Rinia, E. J., and van Leeuwen Th. N., and van Vuren H. G., and van Raan A. F. J. 1998. Comparative analysis of a set of bibliometric indicators and central peer review criteria. Research Policy, 27, pp. 95–107
Unstructured Citation
Smith, R. 1997. Peer review: reform or revolution? Time to open the black box of peer review. British Medical Journal, 315, pp.759–60
Unstructured Citation
Wennerås, C. and Wold A. 1997. Nepotism and sexism in peer-review. Nature 387, pp. 341-343.