Universal approximation

只有一層隱藏層且有足夠多(有限)的隱藏神經元的前饋神經網路(但activation function必須為squashing function)，可逼近 $\mathbb{R}^n$空間中值域為compact sets的任意連續函數。

參考文獻

• Wikipedia: Universal approximation theorem

• [Hornik89] Kurt Hornik, Maxwell Stinchcombe, and Halbert White, "Multilayer feedforward networks are universal approximators," Neural Networks, vol. 2, issue 5, pp. 359-366, 1989, ISSN 0893-6080, http://dx.doi.org/10.1016/0893-6080(89)90020-8.

• [Hornik91] Kurt Hornik, " Approximation capabilities of multilayer feedforward networks, Neural Networks," vol. 4, issue 2, pp. 251-257, 1991, ISSN 0893-6080, http://dx.doi.org/10.1016/0893-6080(91)90009-T.

• [Huang06] Guang-Bin Huang, Lei Chen, and Chee-Kheong Siew, "Universal approximation using incremental constructive feedforward networks with random hidden nodes," IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879-892, July 2006. doi: 10.1109/TNN.2006.875977