Neural Machine Translation (NMT) systems, introduced only in 2013, have achieved state of the art results in many MT tasks. MetaMind’s submissions to WMT ’16 seek to push the state of the art in one such task, English→German newsdomain translation. We integrate promising recent developments in NMT, including subword splitting and back-translation for monolingual data […]
MetaMind Neural Machine Translation System for WMT 2016 Read More »
The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License. Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over 110
The WikiText Long Term Dependency Language Modeling Dataset Read More »
Imagine you were a young child and wanted to ask about something. Being so young (and assuming you are not exceedingly precocious), how would you describe a new object, the name of which you have yet to learn? The intuitive answer: point to it! Surprisingly, neural networks have the same issue. Neural Networks typically use
Teaching Neural Networks to Point to Improve Language Modeling and Translation Read More »
In deep learning, there are two very different ways to process input (like an image or a document). Typically, images are processed all at once, with the same kind of computation happening for every part of the image simultaneously. But researchers have usually assumed that you can’t do this for text data: that you need
New Neural Network Building Block Allows Faster and More Accurate Text Understanding Read More »
We introduce the Dynamic Coattention Network, a state of the art neural network designed to automatically answer questions about documents. Instead of producing a single, static representation of the document without context, our system is able to interpret the document differently depending on the question. That is, given the same document, the system constructs a
State of the art Deep Learning Model for Question Answering Read More »
Humans learn natural languages, such as English, starting from basic grammar to complex semantics in a single brain. How do we build such a single model to handle a variety of natural language processing (NLP) tasks in computers? As a first step towards this goal, we have developed a single deep neural network model which
Multiple Different Natural Language Processing Tasks in a Single Deep Model Read More »
LSTMs have become a basic building block for many deep NLP models. In recent years, many improvements and variations have been proposed for deep sequence models in general, and LSTMs in particular. We propose and analyze a series of augmentations and modifications to LSTM networks resulting in improved performance for text classification datasets. We observe
A way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs Read More »
Automatically generating captions for images has emerged as a prominent interdisciplinary research problem in both academia and industry. It can aid visually impaired users, and make it easy for users to organize and navigate through large amounts of typically unstructured visual data. In order to generate high quality captions, the model needs to incorporate fine-grained
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning Read More »
Deep reinforcement learning (deep RL) is a popular and successful family of methods for teaching computers tasks ranging from playing Go and Atari games to controlling industrial robots. But it is difficult to use a single neural network and conventional RL techniques to learn many different skills at once. Existing approaches usually treat the tasks
Thinking out Loud: Hierarchical and Interpretable Multi-task Reinforcement Learning Read More »
Speech recognition has been successfully depolyed on various smart devices, and is changing the way we interact with them. Traditional phonetic-based recognition approaches require training of separate components such as pronouciation, acoustic and language model. Since the models are all trained separately with different training objectives, improving one of the components does not necessarily lead
Improving end-to-end Speech Recognition Models Read More »
