Multimodal Summary of Complex Sentences for People with Cognitive Disabilities

The following is a synopsis of work on creating multimodal summaries of complex sentences.  A poster of that work, performed by The Hajim School of Engineering and Applied Sciences at The University of Rochester, is the source of all the quoted information in this blog post. I plan to employ this approach on the future Clear Helper Web Site.

The Approach

We propose Multimodal summary of complex sentences. It gives readers the main idea of sentences using pictures and compressed text structured according to simplified text.

The general steps in the MMS approach are:

  • Identify both the main idea of the sentence and related entities and use them to create a compressed summary.
  • Extract pictures for the compressed summary.
  • Add structure to the pictures and text.

Example

Input sentence: In 1492, Genoese explorer Christopher Columbus, under contract to the Spanish crown, reached several Caribbean islands, making first contact with the indigenous people.

Identify event and related entities: In 1492, Genoese explorer Christopher Columbus, under
contract to the Spanish crown, reached several Caribbean islands, making first contact with the
indigenous people.

Extract picture and add structure:

Sentence text, parsed by event and entities, alongside representative pictures.

Naushad UzZaman, Jeffrey P. Bigham and James F. Allen. “Multimodal Summarization for People with Cognitive Disabilities in Reading, Linguistic and Verbal Comprehension” poster presented at “All Together Now: The Power of Partnerships In Cognitive Disability & Technology.” Tenth Annual Conference of The Coleman Institute for Cognitive Disabilities. Westminster, Colorado. 21 October 2010.

Note: No endorsement of The Hajim School of Engineering and Applied Sciences at The University of Rochester is intended or implied.

Tags:


%d bloggers like this: