Event representations: Mapping among linguistic, visual and programmatic ones

In this project, I'm currently investigating a way to learn programmatic representation of events from their linguistic and visual representations. I'm currently using a combination of planning with Bayesian network and sequential modeling to teach a simulated agent to perform activites after humans.

Ongoing work 2017

Event learning of Human-Object Interaction with Sequential Modeling

This project aimed to proposes a framework for learning complex human-object interaction (HOI) events, involving the recording, annotation and classification of event interactions. I classified event classes by Long-Short Term Memory (LSTM) sequential models with Conditional Randon Field (CRF) for constraints of outputs.

2016

Event capture annotation tool

This project aims to create a video annotation tool capable of extracting 3D positional and orientational information about objects annotated in video captured by Microsoft's Kinect hardware, named ECAT. It captures a level of detail not provided by most existing video annotation tools, provides links to existing linguistic infrastructures, and is well suited for building a corpus of event-annotated multimodal simulations for use in the study of spatial and motion semantics

2015

Verb semantics analysis through distributed representations

This project aimed to find a better linguistic event representation, which normally associates with syntactic structure of verbs. I proposed a modified model, called Skipgram Backward-Forward that utilizes thematic ordering of verb arguments to create distributed representation for verbs. This representation is then used to carry out a set of experiments, including analogy test, word disambiguation and antonyms-synonyms separation

2015

Qualia annotation, classification and extraction

This project aimed to investigate into computationality of qualia structure, a component in the larger framework of Generative Lexicon. In specifics, a corpus annotated with qualia relations between nouns and verbs was created, and an algorithm for qualia extraction from Wordnet synsets was suggested.

2014

Event narrative ordering by temporal relations

This project aimed to improve the performance of TARSQI toolkit, a pipeline software used to extract event and temporal expressions, classifying with factuality and tense, and sort them according to narrative order.

2014