Skip to Content

Thomas C. Skalak

Thomas C. Skalak

Contact Us

P.O. Box 400301
Charlottesville, VA 22904

Phone:
434-924-3606
Fax:
434-924-3667
Email:
vpresearch@virginia.edu

National Research Council: Data-Based Assessment of Research Doctorate Programs


Summary of Methodology

The last NRC assessment was conducted in 1993 and released in 1995.  The resulting rankings were based on peer assessment of faculty quality.  The current assessment, conducted in 2006/7 and scheduled to be released on Sept. 28, 2010, is significantly more data-driven than the previous study.  Institutions, doctoral programs, faculty, and students completed separate questionnaires that solicited various data elements (e.g. time-to-degree, student financial support, etc.) not included in the previous assessment.

The NRC identified 20 key variables that, in their judgment, are indicative of the quality of a Ph.D. program.  Values for these variables were obtained from the questionnaires and existing data sources. Two exercises were conducted to calculate the relative importance (i.e. weight) for each variable.  First, participating faculty were asked to select the variables that contribute most to program quality.  Second, a sample of faculty was asked to rate a sample of programs in their field.  The NRC then used statistical methods (i.e. a regression model) to relate these reputational ratings to the 20 key variables.  Results of both approaches will be reported as two overall ranges of rankings. 

Program Rankings

Upon release, the NRC will list, alphabetically by institution, a range of rankings (e.g. 31st to 79th) for each program within a particular discipline. The range of rankings, rather than a single value, accounts for the statistical uncertainty inherent in an assessment of program quality. In total, the NRC will publish five ranges of rankings into which an individual program falls.

The five sets of rankings include:

(1) An overall range of rankings, called the survey-based (S-based) rankings, derived from weights calculated from responses of faculty to a survey that asked them to select those factors most indicative of program quality.
(2) An overall range of rankings, called the regression-based (R-based) rankings, derived from weights calculated from responses to a reputational survey sent to a sample of faculty from each discipline.
(3) A range of rankings based on research activity.
(4) A range of rankings based on student support and outcomes.
(5) A range of rankings based on diversity of the academic environment.

The NRC notes, ”typically the weights for the R-based rankings are larger for program size, measured by PhDs produced, than in the S-based rankings, which usually give more emphasis (larger weights) to measures of per capita research activity.”