The QS evaluation rubric
based on discoursal analysis of what students said in their interviews. All of the students’ responses were transcribed and checked by another native speaker of English for authenticity.
The QS rubric is broken into various measurable elements and based on the four components to a rubric as put forward by Steven and Levi (2005). These are:
- A description of the task that is to be evaluated. Usually, this is taken from the course syllabus. In this case, it is the description of the QS in relation to the IRF structure, given to the students in the first classes.
- The task dimensions, which are the broad areas that are geared towards the learning outcomes. In the QS rubric, there are total of four, but the first and forth are divided into three each.
- The performance scale, used as a score between 5 (highest) to 1 (lowest). There is also a bonus of 1 that the student can achieve by four of the dimensions provided. The bonus of 1 is allocated to each of the four dimensions, so the total score that is possible is 25. The bonus also functions as a safety strategy, so that even if a student does not perform satisfactorily on the main four dimensions, the teacher could still give the student at least some score for attempting to communicate.
- The individual cells, which describe what is expected to receive the allocated score. In order to avoid confusion, the descriptions in each cell have been reduced to how they compare to the highest score.
The QS rubric is commonly referred to as an analytical rubric (Mertler, 2001). This means that there are clear functional and authentic goals that students aim to become proficient in, and finally put into practise once in their homestay. The rubric, therefore, is designed to mimic and position students into near-real homestay situational conversations, by using role-play scenarios that commonly occur in homestays. The fact that the conversations are, from the out-set, open-ended, means that conversations are more unscripted and spontaneous and therefore add further authenticity, which Bachman and Palmer (1999) emphasise as students demonstrating strategic competence. It is based on the five core principles of language assessment put forward by Brown and Abeywickrama (2010). These consist of validity, authenticity, reliability, practicality, and washback.
The elements used in this rubric are discussed below starting with the kind of questions that students ask their partners in the classroom.
The rubric is designed to evaluate speaking from questions that evoke students’ feelings and are at a level whereby both students can connect on common ground in order to push speaking output further. In the classroom, there is a common understanding that the goal of the class is for students to maximize output, therefore, even if students feel that some questions do not require much of a response they should still attempt to employ QS.
Examples of common questions asked in order to activate the QS rubric are based on common questions asked in homestay situations, such as: “What did you do, today?” or “What’s your favourite movie?” etc.
Once the initial question has been asked, the speaker can proceed through the rubric.
The rubric is broken into individual elements that can be evaluated on a scale of 5 (the highest score) to 1 (the lowest score). The conditions for achieving 5 are explained below. The formula is diagrammed in brief below.
A + S + H + H + H +S + H/Q
GENERAL DETAILS/FEELINGS CONCLUSION
A = Acknowledgement
S = Summary
H = Hesitation
Q = Question
- 1. Acknowledgment (A)
In the very first instance, the student offers an acknowledgement as to how they have gauged the question. An example could be a high pitched expression, such as “oh” signalling surprise, or interest, or a mono-tonal expression that might signal a neutral response.
- Summary (S)
Following the students’ acknowledgement, the student quickly summarises, or states the “General” (G) in quadratic structure as a response to the immediate question. For example, if the question is “What did you do, today?” then the summary could be something such as, “I had a good day,” or “I went shopping.” The summary quickly directs the listener to the context of the response.
- 3. Hesitation (H)
If the speaker has trouble interpreting the listener’s signals then they could employ hesitation strategies, such as pausing, or using sounds, such as “um,” or “ah.” Hesitations give the listener and speaker time to gauge whether the situation is conducive for the speaker to begin their reply. Since the speaker is under the expectation that the listener would allow them to continue, the hesitations should not be long. Hesitations are also used after the summary that function as requests from the speaker for the listener to signal if they should continue, pass their turn to the listener, or move to conclude. However, since the speaking situation has been set up for the speaker to progress through all of QS stages, the signals that the listener provides should favour the speaker to continue.
- Hesitation (H)
The summary could be followed by hesitation strategy by the speaker to signal if they should continue, pass the turn to the listener, or conclude. (In the classroom, the listener’s signal should be conducive for the speaker to proceed.) There are many possible responses, but the general pattern of the student attempting to interpret signals from the listener is a skill that occurs in naturally occurring situations and as such should be graded highly if the speaker can achieve this.
- Details (D/F)
When providing details and feelings, the student looks for signals from the listener, as to whether, or not to continue with their speaking, or wait to clarify a point. For example, if the speaker were giving details for going shopping and said “I went to Nagaokakyo,” then the speaker might choose to clarify the name place “Nagaokakyo” by recognising a particular facial, or vocal signal by the listener that they do not know where Nagaokakyo is situated. The speaker could then clarify by saying “Nagaokakyo is a city close to Kyoto city.” In order to do gauge is the listener has understood, the speaker could pause, or directly ask a question. Interpreting micro signals from the listener is assessed in the rubric.
After the student has made the initial acknowledgement and given details, while continually indicating acceptance of attending to micro signals of the listener they might not understand something then the speaker could also mix the details with their feelings. For example, the student could add in “I really like Nagaokakyo because it’s got big shopping malls.” The intention of the student to include their feelings could be interpreted as the speaker inviting the listener to express empathy, which, depending on the signal given back by the listener could give further options to the speaker, either to continue, pass their turn, or conclude their move. The number of details that the speaker should give is dependent on what they are talking about, but the minimum should be to cover the what, where, when, who, why, and how. Anything less than these details could cause confusion for the listener and would receive a lower mark of 4, or below. Since the details and feelings are mixed together there is opportunity for the speaker to segue into other topics that involve connections to past memories, or connect to odo more details than the traditional six.
The speaker might also ask questions to the listener, but since the expectation is for the speaker to continue the speaker should not pass the turn, so as to lose the speaking floor, but merely to acknowledge that there is common ground with the listener. (The QS rubric for evaluating pair conversations will be discussed in a later paper.)
The student also has to show that they can react to the signals that the listener provides throughout the response. Reactions by the speaker function to cement, or confirm common ground with the listener, such as laughing with the listener, repeating information to clarify, or emphasise, or even asking a question in order to confirm that the listener is connecting to what the speaker is saying.
In most naturally occurring conversations, the conclusion usually begins with a hesitation strategy that signals the speaker is about to conclude their turn. Following this hesitation strategy the student should summarise their whole experience, similar to what they might have said in the opening summary in the general section. After the summary, a hesitation will prompt the speaker to initiate something such as “yeah,” or “im.” These hesitations function as requests to the listener that the speaker is waiting for a signal to either relinquish their turn, continue with more information, or a question to the listener. Since the conversation is with the teacher, the teacher could signal that the conversation should come to an end by saying something such as, “okay, thank you” etc.
- Intention to connect (Macro)
This section involves the mechanics of the overall response. It includes fluency, the interest that the speaker evokes in their content, body language, and charisma. The use of this section is to give credit to students that do not do particularly well in the other sections, but can get some credit for applying certain gestures, or willingness to communicate.