Automatic Summarization for Student Reflective Responses

Type PhD Dissertation Defense
Start Date April 21, 2017 01:00 PM
End Date April 21, 2017 03:00 PM
Location 5317 Sennott Square
Organizer Name
Speaker Name Wencan Luo
Speaker Affiliation University of Pittsburgh Ph.D. Candidate
Abstract Educational research has demonstrated that asking students to respond to reflection prompts can increase interaction between instructors and students, which in turn can improve both teaching and learning. However, summarizing student responses to these prompts for large courses (e.g., introductory STEM, MOOCs) is an onerous task for humans and poses challenges for existing summarization methods.

From the input perspective, there are three challenges to summarize student responses. First, there is a lexical variety problem due to the fact that different students tend to use different expressions. Second, there is a length variety problem that the linguistic units of student inputs range from single words to multiple sentences. Third, there is a redundancy issue since some content among student responses are not useful.

From the output perspective, there are at least two additional challenges. First, the human summaries consist of a list of important phrases (phrase scale). Second, from an instructor's perspective, the quantitative number of students (quantity) who have a particular problem or are interested in a particular topic is valuable.

The goal of this research is to enhance student response summarization at multiple levels of granularity.

At the sentence level, we propose a novel summarization algorithm [Luo et al., NAACL 2016] by extending ILP-based framework with matrix imputation to address the challenge of lexical variety. The resulting system allows sentences authored by different students to share co-occurrence statistics. Experiments show that this approach produces better results on one student responses data set in terms of both automatic evaluation and human evaluation.

At the phrase level, we propose a phrase summarization framework [Luo and Litman, EMNLP 2015] by a combination of phrase extraction, phrase clustering and phrase ranking in order to address the phrase scale challenge and to meet the need of aggregating and displaying student responses in a mobile application [Luo et al, NAACL-demo 2015, Fan et al., CHI-WIP 2015]. To address the length variety and redundancy challenges, we extract phrases rather than sentences to form summaries. To address the lexical variety and quantity challenges, we adopt a metric clustering paradigm with a semantic distance to group extracted phrases. Experimental results show the effectiveness on multiple student response data sets.

Also at the phrase level, we propose a quantitative phrase summarization algorithm [Luo et al., COLING 2016] in order to estimate the number of students who semantically mention the phrases in a summary. We first introduce a new phrase-based highlighting scheme for automatic summarization. It highlights the phrases in the human summary and also the corresponding phrases in student responses. Enabled by the highlighting scheme, we improved the phrase-based summarization framework proposed by Luo and Litman [EMNLP 2015] by developing a supervised candidate phrase extraction, learning to estimate the phrase similarities, and experimenting with different clustering algorithms to group phrases into clusters.
We further introduced a new metric that offers a promising direction for making progress on developing automatic summarization evaluation metrics.
Experimental results show that our proposed methods not only yield better summarization performance evaluated using ROUGE, but also produce summaries that capture the pressing student needs.
Committee Diane Litman (advisor)
Rebecca Hwa
Jingtao Wang
Fei Liu