Speech evaluation rubrics: how many levels should be on the scale, and which way should it point?

Back on May 8, 2010 I blogged about Rubrics and figuring out where you are. The fourth Merriam-Webster dictionary definition for a rubric is:

“a guide listing specific criteria for grading or scoring academic papers, projects or tests”

An evaluation rubric includes several questions about both content and delivery, which are scored on some sort of a scale. How many levels should be on the scale? At least two, but would either three, four, or five be better, as is shown above. What names should be given to those levels?

A week ago on LinkedIn at The Official Toastmasters International Members Group ChenKeat Fan posted on Evaluation sheet for evaluator in Pathways: what’s your opinion of its usefulness? So far there have been over twenty comments (including one from me). He complained that having to quantify on a 1 to 5 scale is cumbersome as compared to the previous scale with three levels. (But Toastmasters actually used a five-level scale before in their course on The Art of Effective Evaluation).

I got curious and looked up some history about speech evaluations. Back in 1981 there was a 47-page booklet by Douglas G. Bock and E. Hope-Bock on Evaluating Classroom Speaking. You can download it as Document ED 214 213 on the Education Resources Information Center (ERIC) web site. That booklet ends with a section showing 13 sample evaluation forms. The preceding section on constructing an evaluation instrument discusses the topic of Controlling Rating Errors on page 22. It says that:

“The error of central tendency can be controlled by the number of scale values used on the continuum. For example, if only three numbers are use, most raters are going to use the middle category. It has been found that a five-step scale usually results in three steps being used. A seven-step scale uses about four. A ten-step scale usually produces five. One way to get raters to use more of the scale is to have more steps.”

A second question is which way the scale should run. As shown above, graphs displayed with Cartesian coordinates typically have an x-axis using the right-hand rule, but the left-hand rule also could be used. A scale with the right-hand rule would have the worst category at the left, and the best at the right. Since we read English from left to right, there might be a primacy effect where we would overuse that left category. Those using languages read from right to left (Arabic, Hebrew, Persian, or Urdu) might have a different bias than English speakers.

A third question is where the scale should start, as shown above. Should there be a zero? Some might object that it should begin at 1, since we all are heroes - not zeroes. One way to organize a speech evaluation form with five levels is to alternately list a question and the scale (so the score for it can be circled). Other pages of the form can show details of what those five levels mean.

One well-known rubric is The NCA Competent Speaker Speech Evaluation Form (2nd edition, 2007) from the U. S. National Communication Association, as shown above. It is described in a 49-page Acrobat .pdf file you can download free from their web site. This rubric lists eight competencies – four for Content and four for Delivery (in four columns) rated on a right-hand rule axis with three levels: Unsatisfactory, Satisfactory, or Excellent. I have numbered them 0, 1, 2 – although the form omits that detail but nevertheless has a bottom row for General Comments and a Summative Score. The form also has a three-page explanation for those competencies. There is another NCA Competent Speaker Holistic Speech Evaluation Form that combines each set of four into just two categories labeled Preparation and Content, and Presentation and Delivery.  

In 2012 Communication Education magazine had an article by Lisa M. Schreiber, Gregory D. Paul, and Lisa R. Shibley titled The Development and Test of the Public Speaking Competence Rubric (PSCR) which you can download. As shown above, this rubric has eleven categories rated on a left-hand rule axis numbered from 4 to 0 and titled 4 = Advanced, 3 = Proficient, 2 =Basic, 1 = Minimal, or 0 = Deficient. You also can download a single-page Table with a detailed explanation for each item. I blogged about the PSCR in a July 9, 2012 post titled A new scale (rubric) for evaluating speeches.

   

In their Success Communication series, Toastmasters International has a two-hour course titled The Art of Effective Evaluation (Item 251). It has an Individual Speech Evaluation Form (Item 251D), which you can find at the end of a handout for it from the Park City club. As shown above, the form has 12 explicit categories (and room for two optional ones). The right-hand rule axis has five levels which from left to right are labeled 1 = Needs Considerable Improvement, 2 = Needs Some Improvement, 3 = Acceptable, 4 = Very Good, 5 = Excellent. The form is organized into three columns. Each row has a Category, followed by a Rating (1 to 5) and Recommendations for Improvement.   

In the new Pathways educational program from Toastmasters International, speeches are evaluated using a three-page form. The first page has a Purpose Statement, Notes for the Evaluator, and General Comments (with three categories – You excelled at, You may want to work on, and To challenge yourself). As shown above for the Icebreaker Speech, the second page in the form has seven categories on a left-hand rule axis with five levels labeled from left to right as 5 = Exemplary, 4 = Excels, 3 = Accomplished, 2 = Emerging, and 1 = Developing. Although those labels are  explained in detail on the third page of that form, I think they are way more obscure than those used in the PSCR.

As is shown above, perhaps a more honest revised set of labels would be 5 = Outstanding, 4 = Excellent, 3 = Good, 2 = Fair, and 1 = Poor (or Poop). Other evaluation guides from Pathways are discussed on a web page at UmErYouKnow with a link to a list of them.     

-->