Deep Learning for Computer Vision and Natural Language Processing

• Schedule
• Reading
• Resource
• Projects
• Prog. Problem

Google Group

This Google group is the place students can discuss their projects or questions related to this course.

Software

Toolkits used in this class

• Theano (required)
• Caffe: (suggested for large scale CV although not required)
• iPython notebook: (suggested as the Python editor in the team collaboration, although you are free to choose other IDEs like Eclipse, pycharm, or pyscripter, etc.)

Theano resources

• Official Theano tutorial
• Goodfellow's tutorial
• A more technical tutorial at HPCS2011
• Alec Radford's introduction to Deep Learning with Python and Theano

Computing resources

• CU Amazon $1000 funding for web-based startups

Deep Learning for computer vision

• Caffe: the most popular opensource project on cv+deep learning
• Torch7 (with a nice iTorch interface): widely used in Facebook and NYU
• CUDA-convnet2: excellent GPU programming with multi-GPU supports, less user-friendly
• CCV: a compact c package

Toolkits for speech and language

• Kaldi
• Word2vector

GPU programming and fast CNN

• CUDA Toolkit and SDK (with a lot examples)
• cuDNN paper and official website
• CNN with fbfft
• Sander Dieleman's blog on FFT based convolution in Theano
• more to be added...

Datasets

You may find a lot of challenges with training/testing datasets on Kaggle. But the following come from traditional academic researchers.

CoNLL shared tasks & datasets over the years (including a variety of NLP tasks):

• http://ifarm.nl/signll/conll/

Other NLP resources

• Parsing: http://universaldependencies.github.io/docs/
• Semantic relation extraction: http://googleresearch.blogspot.com/2013/04/50000-lessons-on-how-to-read-relation.html
• Information extraction: https://catalog.ldc.upenn.edu/LDC2011T08 (requires LDC membership)
• Information extraction: http://www.nist.gov/tac/data/index.html
• Paraphrase http://research.microsoft.com/en-us/downloads/607d14d9-20cd-47e3-85bc-a2f65cd28042/
• Paraphrase http://knowitall.cs.washington.edu/paralex/
• Question answering http://www-nlp.stanford.edu/software/sempre/
• Machine translation: http://www.statmt.org/wmt14/translation-task.html#download

Image with captions/text

• PASCAL sentences
• Microsoft Common Objects in Context
• MIRFlickr
• Yahoo Flickr Creative Commons 100M (you may use a subset)
• Wikipedia articles (with images)
• NUS Wide

Image classification

• Yann LeCun et al's MNIST and a copy at deeplearning.net
• PASCAL Object Recognition Database
• CIFAR 10 and CIFAR 100
• The Oxford-IIIT Pet Dataset

Live Image Classification Challenges

• LifeCLEF 2015 Fish counting
• LifeCLEF 2015 Plant recognition

Faces and OCRs

• Labeled Faces In the Wild
• Pub Fig
• MIT 10k US Adult Faces Database
• CASIA Online and Offline Chinese Handwriting Databases

Large scale image and video datasets (you may probably need powerful GPUs if you want to try them)

• ImageNet LSVRC
• 1 million YouTube videos
• Hollywood2 dataset
• UCF 101

Related Deep Learning courses and resource

• Hinton's course on Neural Networks for Machine Learning.
• Andrew Ng's deep learning class at Coursera.
• Yann LeCun's deep learning course at NYU.
• Fei-Fei Li and Andrej Karpathy have a course specialized for Convolutional Networks.
• Xiaogang Wang's class on Introduction to Deep Learning at CUHK.
• Both instructors learned from this deep learning tutorial associated with theano.
• Also more and more tutorials and talks form ML/CV/NLP conferences... (to be added)