and reliable (Marshall et al., 2011; Amrein-Beardsley & Popp, 2012).
Within the RTOP, 25 items describe three scales: lesson design and
implementation, content, and class culture.
Two trained raters independently scored four to eight recorded
class sessions for each of the 13 course sections (Holt et al., 2015).
Inter-rater reliability was high among RTOP raters (generalizability
coefficient = 0.787; see Holt et al., 2015). Raters categorized the
total RTOP score for each class session into one of five levels,
dependent on their raw score out of 100 (Sawada et al., 2003;
Ebert-May et al., 2011). If the two raters independently scored
class sessions to the same RTOP level, the raters’ raw scores were
averaged. A blind “tie-breaking” rater was used if initial scoring
by the two original raters yielded two different RTOP levels; in this
case, the two raters’ raw scores sharing an RTOP level were
averaged. Within each section, rater-averaged RTOP scores from
multiple days were averaged to better represent the range of
learner-centeredness over the semester. A single continuous RTOP
score, averaged among raters and among several representative
class sessions, for each instructor was used in analyses to reflect
learner-centeredness measured by external observers. We should
note that with the exception of one instructor whose two course sections had greater than a 10-point difference in RTOP scores, the
average RTOP scores of instructors who taught two course sections
generally did not differ by more than two points. We expected that
more learner-centered components of a syllabus (e.g., learning objectives) would correlate with a higher RTOP score, as this would indicate that more student-focused instructors were engaging in more
learner-centered practices, based on expert observations.
Data & Analyses
Descriptive statistics described the faculty sample, examined distributions and frequencies of the data, and assessed the suitability of
the items to be included in later regression models. We used t-tests
and analysis of variance to examine differences in independent and
dependent variables across demographic characteristics and conducted correlations to measure the relationship between ATI and
RTOP variables and the total syllabus rubric score and scale scores
therein ( i.e., Learning Goals and Objectives, Assessment Activities,
Schedule, and Overall Learning Environment).
Hierarchical linear regression was used to answer the second
research question, with total syllabus rubric score as the dependent
variable, and variables of interest ( i.e., RTOP, CCSF, and ITTF
scores) acting as independent variables in the model. We examined
R2 values, and associated P-values and F-tests, for each linear regression model to evaluate model performance. Assumptions of linearity
were met, as evidenced through normal probability plots and standardized residual plots. Hierarchical linear regression was used to
determine the contribution of two demographic characteristics ( i.e.,
gender and years of teaching experience) in explaining syllabus
rubric scores; this regression model served as a control for the potentially extraneous effect of demographic variables on instructors’
syllabus rubric scores. For these demographic items, response options
were categorical and therefore had to be dummy-coded prior to
inclusion in the regression models. Variables were entered into linear
regression models in two steps, with demographic variables tested at
step 1, and the three independent variables of interest ( i.e., RTOP,
CCSF, and ITTF scores) added at step 2. All analyses were conducted using SPSS Statistics 24.0 (IBM, Somers, NY).
RESULTS
Descriptive Analyses
The student enrollment of participating course sections ranged
from 23 to 391 (mean = 115), and courses were scheduled at both
traditional ( i.e., weekday mornings and afternoons) and nontraditional ( i.e., weekends and weekday evenings) times. We explored
correlations between syllabus score and enrollment as well as syllabus score and class time but found no clear trends; thus, these findings are not reported here. Across instructors, the mean RTOP
score was 41.1 ( i.e., teacher-centered lecture with limited demonstrations and student participation), while the mean ATI score
was 23.3 on the ITTF scale and 26.5 on the CCSF scale ( i.e., transitional and approaching learner-centeredness, respectively).
The mean syllabus rubric score across our eight instructors was
13.1 ( i.e., teacher-centered). Five of the instructors’ total syllabus
scores fell within the teacher-centered range, three instructors’
syllabus scores fell within the transitional range, but no scored syllabi
were categorized as learner-centered ( i.e., score of 31–46). Across
scales, Assessment Activities (α = 0.616) and Overall Learning
Environment (α = 0.698) were moderately reliable, while Learning
Goals and Objectives had low reliability (α = 0.436). As Schedule
was only composed of one item, reliability could not be calculated
for this scale. Low to moderate reliability across scales could most
likely be accounted for by (1) a low sample size of eight instructors
and (2) few items per scale, especially for Learning Goals and
Objectives (n = 2).
Simple Comparisons
Total syllabus rubric scores did not strongly correlate with ITTF
scores (r = −0.065); however, RTOP scores (r = 0.593; Figure 1)
and CCSF scores (r = 0.664) correlated independently and positively with total syllabus rubric scores (Table 1). We further found
strong positive correlations (r > 0.5) between RTOP score and the
Learning Goals and Objectives scale score (r = 0.655); ITTF score
and Schedule scale score (r = 0.713); CCSF and Learning Goals
and Objectives scale score (r = 0.855); and CCSF and Overall
Learning Environment scale score (r = 0.514; Table 1).
We detected no significant difference between male and female
instructors in total syllabus rubric scores (t6 = 0.055, P = 0.96),
nor in any syllabus scale scores (Learning Goals and Objectives:
t6 = −1.083, P = 0.32; Assessment Activities: t6 = 0.928, P = 0.39;
Schedule: t6 = 0, P = 1.00; Overall Learning Environment:
t6 = 0.378, P = 0.72). Further, total syllabus rubric scores were statistically similar across categories of teaching experience, including
1–10 years, 11–20 years, and 21+ years (F2, 5 = 0.621, P = 0.57).
Likewise, syllabus scale scores were also statistically similar across
our three categories of teaching experience (Learning Goals and
Objectives: F2, 5 = 2.41, P = 0.18; Assessment Activities: F2, 5 =
1.05, P = 0.42; Schedule: F2, 5 = 2.50, P = 0.18; Overall Learning
Environment: F2, 5 = 0.292, P = 0.76).
Hierarchical Linear Regression
The three variables of interest ( i.e., RTOP, CCSF, and ITTF scores)
simultaneously added at step 2 (F3, 2 = 5.376, P = 0.164, R2 =
0.931) improved the fit of the model beyond what was explained
by the demographic variables ( i.e., gender and years of teaching
