Towards measuring research supervision quality

Richard S J Tol 20 April 2014

a

A

Academics do many things. We teach, research, run departments, engage in public debate, and train young researchers. The outputs of some of these activities are routinely measured and thus play an important role in rewards and promotion. The standard and mostly widely followed measures involve publication outputs (see Hudson 2013 and Conley et al. 2011). Other activities are rarely assessed and thus left to the intrinsic motivation of the academic. Research supervision is one.

Our recent work – summarised in this column – proposes a measure of the quality of advice to PhD candidates, the h1-index. Our proposed indicator of research supervision excellence shows that, by and large, good researchers make good advisers; however, there are many people who excel at research but are less good at supervision, and many more that make great supervisors without being well published themselves.

This index could be a useful additional measure of university quality and transparency by assisting students in choosing a university and PhD supervisor. Moreover, if with measurement comes saliency, this could be an additional motivating factor for academics.

The index

A well-known indicator for research output is the h-index, where h is the highest number of papers a researcher has published that have each been cited h times or more. In other words, an academic has an h-index of 5, if she has published 5 papers which have each been cited at least 5 times. The h-index is a simple way to combine the quality and quantity of research output. Quality without quantity is punished, as is quantity without quality. Invented only in 2005 by Jorge Hirsch (Hirsch 2005), it has taken the world of research assessment by storm.

The h-index naturally lends itself to evaluating collectives. Frances Ruane and I (2009) introduce the h1-index to quantify the quality of research supervision. A professor is a good PhD adviser if she trains many good researchers. We define a professor’s h1-index based on the h-indices of her supervisees; a professor with an h1-index of 4 has had 4 PhD graduates each with an h-index of 4 or more.

Data are a challenge. All universities keep records on PhD completions, but few do so in an accessible way, let alone a structured database. Fortunately, academics are snobs and take pride in their lineage. (Jan Tinbergen was my adviser’s adviser’s adviser, so there.) Academic genealogies can be found. We used Alan Deardorff’s family tree of trade economists. There are 65 professors with 4 or more supervisees, for a total of 356 PhDs. Publication and citation data were taken from the Web of Science. The 12 best professors are listed in Table 1.

Table 1. Selected characteristics of the 12 most successful advisers in trade economics.

 
 
h
1st publication
# supervisees
h1
1st graduation
Johnson, Harry G.
U Chicago
18
1956
14
9
1956
Bhagwati, Jagdish N.
Colombia U
25
1957
25
8
1969
Findlay, Ronald
Colombia U
11
1959
13
7
1981
Baldwin, Robert E.
U Wisconsin
12
1966
16
7
1969
Jones, Ronald W.
U Rochester
19
1956
22
7
1962
Kindleberger, Charles P.
MIT
8
1956
11
7
1956
Samuelson, Paul A.
MIT
33
1942
8
7
1956
Grossman, Gene M.
Princeton U
28
1978
13
6
1985
Deardorff, Alan V.
U Michigan
12
1970
24
6
1978
Krugman, Paul R.
Princeton U
31
1976
10
6
1974
Ethier, Wilfred J.
U Penn
13
1970
11
6
1973
Haberler, Gottfried
Harvard U
3
1956
10
6
1941

Like the h-index, the h1-index is a measure of lifetime achievement. It thus favours those with an earlier career start. Like the h-index is roughly proportional to the number of years since first publication, the h1-index is roughly proportional to the number of years since first graduation – see Figure 1. We can thus compare academics of different vintages by their h- and h1-rate, defined as the index over the number of active years.

Figure 1. The h1-index as a function of the year of first graduation

Note: dots are observations; thick central line is the expected value; thin lines are the 95% confidence interval.

Three professors stand out in that they graduate more high quality supervisees per year than most: Tony Venables, Gene Grossman, and Ron Findlay.

By and large, good researchers make good advisers. There is a weak but positive correlation (0.41) between the h-rate and the h1-rate – see Figure 2. However, there are people who are excellent researchers but less good at research supervision. There are also excellent PhD advisers whose own research is nothing special. People good at neither are excluded from the sample.

Figure 2. The h1-rate as a function of the h-rate

Note: dots are observations; line is the least squares fit.

The flamboyant Harry Johnson stands out. He made important contributions to our understanding of international trade. He also trained a fair number of trade economists who themselves made important contributions. And he taught them how to be good PhD advisers themselves. Johnson’s students tend to have high h-indices as well as high h1-indices. Johnson thus has a high h-, h1- and h2-index; he was an excellent researcher and a grand professor.

There are two key problems with the analysis. Restricting attention to trade economists disadvantages those who also work in other fields. We did not control for student quality at the start of the PhD. That is, we did not measure the value added of research advice but rather intrinsic quality plus all education. We can get some idea, however, by looking at the institutions in Table 1. Some teach at top schools and have first pick from PhD candidates. Others do not, but nevertheless succeed in graduating excellent PhDs.

The results here merely illustrate our proposed indicator of excellence in research supervision. More systematic data collection is needed (see http://ideas.repec.org/top/top.person.students.html for another attempt) not only for a more comprehensive assessment but also for a better understanding of what makes a good PhD adviser.

Policy conclusions

Supervising PhD candidates is an important task. Yet, evaluations rarely go beyond the number of PhD completions with a few anecdotes about placements thrown in. The quality of research supervision can be measured, however, and should – at the moment, quantity is rewarded, and the incentive is thus to lower standards. The indicator proposed here should be interpreted with care. It takes time for a PhD candidate to manifest himself as a good researcher, and thus for a professor to manifest herself as a good supervisor. The h1-index is fine for celebrating past achievements. It should be used for current decisions only to the extent that present quality correlates with past glory.

References

Conley, John P, Ali Sina Önder, Mario Crucini, and Robert Driskill (2011), “Publication lags and young economists’ publication lags”, VoxEU.org, 24 October.

Deardoff, Alan (undated), “Family Tree of Trade Economists”.

Hirsch, J E (2005), “An index to quantify an individual’s scientific research output”, PNAS, 102: 16569–16572.

Hudson, John (2013), “Challenging times in academia”, VoxEU.org, 11 November.

Ruane, Frances P, and Richard S J Tol (2009), “A Hirsch measure for the quality of research supervision, and an illustration with trade economists”, Scientometrics, 80: 613–624.

 

This column first appeared on the website of the Economics of Higher Education Network: economicsofhe.org/

a

A

Topics:  Frontiers of economic research

Tags:  economic research, research quality, supervision quality

Professor or Economics, University of Sussex; and Professor of the Economics of Climate Change, Vrije Universiteit Amsterdam