How Do I Learn Neural Network (and Deep Learning)

First, I am nobody in this topic, please don’t hesitate to give better suggestion. This is actually just a “documentation” about my experience when I work on my undergraduate thesis in 2015. Once again, please don’t hesitate to discuss and give better suggestion about this topic, or.. correct my English if I made some mistakes 🙂

Background

Okay, before I tell you my story, I’ll tell you about my background when I started to learn neural network. I wrote this section only have one intention, to make you easy measure what you need, nothing else.

1. I have familiarity with some of mathematical symbols, like how to read sigma symbol, derivative symbol and many others (but not every symbols).
2. I have familiarity with basic math operation like derivative function, algebra, matrix and vector operation (but only “the basic”).
3. I’ve never coded in python, but I’ve been using C++ around 2-3 years and I used to code  Matlab style programming language in some project before (and thanks, this knowledge is very helpful)
4. I studied the Introduction of Artificial Intelligence at the fourth semester in college, but I won’t said I did it well 🙁

After a month googling randomly, I started to make a list what should I learn, the order of the list below it’s just the best order in my opinion. In my experience I jumped many times over the topic because at that time, I didn’t know what I didn’t know 🙁

If you want to learn about Deep Learning, lets begin with: What is Machine Learning?

• Easy start to get the big picture : http://www.r2d3.us/visual-intro-to-machine-learning-part-1/
• Start to learn the basic math of machine learning : https://www.coursera.org/learn/machine-learning (I didn’t watch all videos, only the first three weeks but I did the assignment too)

Then start to learn about Neural Network

• Easy start to get the big picture : https://www.youtube.com/playlist?list=PLiaHhY2iBX9hdHaRr6b7XevZtgZRa1PoU (I watched all videos even I didn’t understand at that time)
• After I knew the big picture of neural net I also read the simple history of neural net, and the “basic” of neural net, I think this gave me a few intuition. I read a book “Fundamentals of Neural Network” by Fausset, only the first two chapters
• But, I think you can get the “alternative version” what I got from Fausset’s book in the first chapter from http://neuralnetworksanddeeplearning.com/ (try to read the first three chapters)
• You can also follow the first modul from http://cs231n.github.io/ and some chapters from http://deeplearning.stanford.edu/tutorial/
• Other tutorial from stanford gave me introduction to neural network for deep learning http://cs.stanford.edu/~quocle/tutorial1.pdf

Let’s do some math, Backpropagation?

I suggest you to fully-understand this algorithm in order to give some intuitions to help you in model tuning or error handling when you start implement the neural network

• Easy start : I watched again the neural network demistifyed, and slowly got the intuition how backpropagation work, I followed and tried the math step-by-step
• Watch the 4th week on Coursera Introduction to Machine Learning
• Try the example of how backpropagation works : https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/
• Read carefully the second chapter from http://neuralnetworksanddeeplearning.com/
• And if you start thinking they have different formula / calculation / interpretation, start to prove that you are wrong. I started to compare all the calculations to understand more

Python for machine learning?

• Learn the basic of python. I forgot how I learn my first python syntax, I remember I use some interactive platforms like https://www.codecademy.com/learn/python then I practice and read some python code in http://codeforces.com‘s problemset
• Get some knowledge about Numpy https://docs.scipy.org/doc/numpy-dev/user/quickstart.html and because I already knew some matlab syntax I also read http://mathesaurus.sourceforge.net/matlab-numpy.html
• I started using python for machine learning by following kaggle tutorial https://www.kaggle.com/c/titanic/details/getting-started-with-python
• Then I started implement the neural network, first I followed this simple tutorial : http://iamtrask.github.io/2015/07/12/basic-python-network/ this tutorial will give you some insight about backpropagation too
• Because I want to use GPU for deep learning, I started to learn Theano from Theano’s site http://deeplearning.net/software/theano/ and this nice tutorial https://bigaidream.gitbooks.io/subsets_ml_cookbook/content/dl/theano/theano_tutorial.html

So what is Deep Learning?

• Easy start : read all chapter from http://neuralnetworksanddeeplearning.com/
• The second tutorial from stanford : http://cs.stanford.edu/~quocle/tutorial2.pdf
• Amazing book from Ian goodfellow and friends : http://www.deeplearningbook.org/
• start to understand the meaning of “deep learning”, you can try by read a lot of articles on the internet

What is Convolutional Neural Network (I use CNN in my undergraduate thesis)?

• The second module from http://cs231n.github.io/ give nice explanation
• And also try http://ufldl.stanford.edu/tutorial/supervised/ConvolutionalNeuralNetwork/
• And the great video that gives me a lot of new knowledge about CNN : https://www.youtube.com/watch?v=cDdpwAIsuD8

Backpropagation in CNN?

I don’t know it is necessary or not, but I learned it. I think if you understand the basic of backpropagation it’s enough. But if you are curious how backpropgation through max pooling or convolution layer, it does not hurt to learn it.

• Okay, this part is a lot of math, you can start from read carefully http://ufldl.stanford.edu/tutorial/supervised/ConvolutionalNeuralNetwork/
• Then try some cute math from http://andrew.gibiansky.com/blog/machine-learning/convolutional-neural-networks/ and http://timdettmers.com/2015/03/26/convolution-deep-learning/ and from http://cogprints.org/5869/1/cnn_tutorial.pdf

And…great, It’s done, That’s all my references that I used when I did my undergraduate thesis, I hope it will be useful for someone :/ or at least for me in the future 🙂