Measuring Research Impact
Vladimir Ponczek1 Cristine Pinto1
1S ˜ao Paulo School of Economics, FGV
Introduction
Main Ideas:
Dimensions
Measures
Aggregations
Dimensions
Impact to academia
Impact to policy-making
Public dissemination
Dimensions
Impact to Academia
Peer-reviewed publications.
Thesis and dissertations (students participation).
Dimensions
Impact to Policy-Making
Policy papers.
Participation in policy-making workshops and congresses.
Courses and training targeting policy-makers.
Dimensions
Public Dissemination
Events (seminars, workshops, lectures) open to general public.
Courses open to general public.
Media Coverage.
Dimensions
Financial Funding
Funding from research agencies.
Funding from private sector.
Measures
Impact to Academia
Peer-reviewed publications.
Journal ranking (ranking CL - Based onInferring Missing Citations: A Quantitative Multi-Criteria Ranking of all Journal in Economics by Combes and Linnemer):
Focuses on Economics (EconLit), but considers other fields (multidisciplinary).
Impact factors based on JCR (Thomson-Reuters) and Google H-Index.
Takes into consideration self-citation, page numbers and number of co-authors.
Measures
Impact to Academia
Table :CL-Ranking
Journal CLm Final Ranking Quarterly Journal of Economics 100.00 1
Econometrica 98.42 2
American Economic Review 96.14 3
Journal of Finance 94.21 4
Review of Economic Studies 86.37 5 Journal of Political Economy 82.93 6 Journal of Financial Economics 82.56 7 Review of Financial Studies 78.41 8 Review of Economics and Statistics 68.50 9 Journal of Econometrics 66.55 10 Journal of Economic Theory 65.27 11
Economic Journal 59.94 12
Measures
Impact to Academia
Thesis and Dissertations
Number of thesis and dissertation defenses.
Number of thesis and dissertation published in peer-reviewed journals.
Academic Seminar, Congresses and Workshops.
Number of papers of students or/and professors accepted in congresses and workshops.
Number of academic seminars presented in other institutions by students or professors.
Measures
Impact to Policy-making
Policy papers
Number of working papers or papers produced by professors and students published in thinks-tanks, government journals,
working-paper series, etc.
Participation in policy-making workshops and congresses.
Number of attendances in workshops and debates with policy-makers.
Number of workshops and seminars promoted by the Centers with policy-makers.
Courses and training targeting policy-makers
Number of lectures in courses targeting policy-makers. Number of courses promoted by the Centers targeting policy-makers.
Number of courses’ participants.
Project development to the public sector
Measures
Public Dissemination
Events open to general public
Number of attendances in workshops and debates with the general public.
Number of events promoted by the Centers open to the general public.
Number of events’ participants.
Media coverage
Media Metrics
Online dissemination
Number of website accesses Social media metrics
Courses and training targeting general public
Measures
Financial Funding
Funding from research agencies
Research Grants (in R$). Student scholarships.
Professor and student fellowships.
Funding from private sector
Research Grants (in R$).
Aggregation
One size does not fit all!
Centers may have different objectives and aggregation must reflect them!
Aggregation
Aggregation
Centers are ranked in each sub-dimension.
Aggregate each dimension using different weights based on Center type.
Aggregation
Table :Suggestions of Weights
Center focus Academia Policy-making
Academia 60% 15%
Peer reviewed Publication 70% 60% Thesis and dissertations 20% 20% Congress and workshops 10% 20%
Policy-making 15% 60%
Policy papers 25% 30%
Congress and workshops 25% 15%
Courses 25% 15%
Project Development 25% 40%
Public Dissemination 10% 10%
Congress and workshops 25% 15%
Courses 25% 15%
Media Coverage 30% 50%
Online dissemination 20% 20%
Financial funding 15% 15%
Research Agencies 60% 50%
Aggregation
Table :Example for policy-making Centers
Center A Center B Center C
Academia 15% 2 1.4 2.6
Peer reviewed Publication 60% 2 1 3 Thesis and dissertations 20% 1 2 3 Congress and workshops 20% 3 2 1
Policy-making 60% 2.1 2 1.9
Policy papers 30% 1 2 3
Congress and workshops 15% 2 1 3
Courses 15% 2 3 1
Project Development 40% 3 2 1
Public Dissemination 10% 2.2 2.3 1.5
Congress and workshops 15% 3 2 1
Courses 15% 1 2 3
Online dissemination 20% 3 1 2
Media Coverage 50% 2 3 1
Financial funding 15% 2 2 2
Research Agencies 50% 2 3 1
Private sector 50% 2 1 3
Final Index 2.08 1.94 1.98
Next Steps
Internal discussion with faculty members.
New specialized staff:
Data collection and measuring of impact.
