The direct answering of open-ended questions is the next stage in conversational AI and search engines. In my Master's thesis, I took on this challenge and developed a question-answering chatbot using deep learning. Which I tackled by breaking the problem down into a sequence of simple retrieval problems.

What I Did

I formulated my approach as an extension of a classic retrieval problem; Given a set of documents and a query, find the set of relevant documents for a query. In my case, we need to extend this; Given a set of relevant documents retrieved from a search engine, find the subset of each document that contains relevant information to the query. From this subset of relevant information generate a response that includes it.

By formulating it this way, you can break down the solution into three chief components: Search engine to retrieve documents, answer candidate extraction, and Answer Generation. In my approach, the search engine was Google (Why make things more difficult for yourself?). To generate answer candidates the main body of the document's text is cleaned, tokenized, and ran through a DL model called QANet which I implemented from a paper. The answer generation was a specialised form of a pointer network that could point to both the query and one or more answer candidates. The implementation of a custom training procedure that randomly marked dates, names, etc. allowed the pointer network to be easily trained.

Results

I'm proud of the outcome of my thesis, despite having to spend many summer months indoors. It's my most impressive technical accomplishment to date. If given the chance, I would adjust my expectations to be more realistic, as I didn't fully comprehend the level of difficulty when I first started. It pushed me to my limits at times, but the Distinction I received for all my hard work certainly made it worthwhile.