From 6910e5885f5d6ed3c71b759f86feece3fe32b5bc Mon Sep 17 00:00:00 2001 From: rasbt Date: Mon, 9 Nov 2015 01:40:29 -0500 Subject: [PATCH] origins of ml --- README.md | 1 + faq/README.md | 1 + faq/ml-origins.md | 11 +++++++++++ 3 files changed, 13 insertions(+) create mode 100644 faq/ml-origins.md diff --git a/README.md b/README.md index 045bb15..84c15f2 100644 --- a/README.md +++ b/README.md @@ -103,6 +103,7 @@ Excerpts from the [Foreword](./docs/foreword_ro.pdf) and [Preface](./docs/prefac - [What factors should I consider when choosing a predictive model technique?](./faq/choosing-technique.md) - [Does regularization in logistic regression always results in better fit and better generalization?](./faq/regularized-logistic-regression-performance.md) - [Why did it take so long for deep networks to be invented?](./faq/inventing-deeplearning.md) +- [What are the origins of machine learning?](./faq/ml-origins.md) - [How was classification, as a learning machine, developed?](./faq/classifier-history.md) - [What are some good books/papers for learning deep learning?](./faq/deep-learning-resources.md) - [What are the different dimensionality reduction methods in machine learning?](./faq/dimensionality-reduction.md) diff --git a/faq/README.md b/faq/README.md index a9969d9..6fdd1fb 100644 --- a/faq/README.md +++ b/faq/README.md @@ -46,6 +46,7 @@ Sebastian - [What factors should I consider when choosing a predictive model technique?](./choosing-technique.md) - [Why did it take so long for deep networks to be invented?](./inventing-deeplearning.md) - [Does regularization in logistic regression always results in better fit and better generalization?](./regularized-logistic-regression-performance.md) +- [What are the origins of machine learning?](./ml-origins.md) - [How was classification, as a learning machine, developed?](./classifier-history.md) - [What are some good books/papers for learning deep learning?](./deep-learning-resources.md) - [What are the different dimensionality reduction methods in machine learning?](./dimensionality-reduction.md) diff --git a/faq/ml-origins.md b/faq/ml-origins.md new file mode 100644 index 0000000..74c7924 --- /dev/null +++ b/faq/ml-origins.md @@ -0,0 +1,11 @@ +# What are the origins of machine learning? + +I think that it all started with the McCulloch-Pitt (MCP) Neuron, a first model of how a neuron in a mammal's brain could work: +W. S. McCulloch and W. Pitts. [A logical calculus of the ideas immanent in nervous activity](http://link.springer.com/article/10.1007/BF02459570). The bulletin of mathematical biophysics, 5(4):115–133, 1943. + +Note that other methods like linear regression were already invented (F. Galton. [Regression towards mediocrity in hereditary stature](http://www.jstor.org/stable/2841583). Journal of the Anthropological Institute of Great Britain and Ireland, pages 246–263, 1886.). Here, I would like to make the distinction between ML and statistics in terms of how ML evolved. I see ML as a field that emerged from artificial intelligence research, hence, the MCP neuron. +However, ML is deeply intertwined with statistics. For example, I would describe a linear regression analysis based on the closed-form solution (normal equation) primarily as a technique that came from the statistics field, and I would associate linear regression with stochastic gradient descent learning as an ML technique. I think the early goal in ML was how the algorithm can "learn" a function by itself rather than solving an equation mathematically. + +So, I would say that the first ML algorithm really was the perceptron (F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957.) What followed was the gradient descent algorithm used in adaptive linear neurons (B. Widrow. Adaptive ”Adaline” neuron using chemical ”memistors”. Number Technical Report 1553-2. Stanford Electron. Labs., Stanford, CA, October 1960.). Those single learning units were then connected to multi-layer architectures, and what followed was the multi-layer perceptron also around the first half of the 20th century. + +Note that I would put, for example, Fisher's Linear Discriminant Analysis (R. A. Fisher. [The use of multiple measurements in taxonomic problems](http://onlinelibrary.wiley.com/store/10.1111/j.1469-1809.1936.tb02137.x/asset/j.1469-1809.1936.tb02137.x.pdf;jsessionid=F0E7EFF219B2C2D94FF2B2981D3533E8.f04t03?v=1&t=igrkwkvk&s=631d3f737becda820356e6862bffc239e9b1f2d6). Annals of eugenics, 7(2):179–188, 1936) into developments in the statistics department -- note that this was before the term ML was coined.