Extracting and composing robust features with denoising autoencoders | Proceedings of the 25th international conference on Machine learning
Published: 05 July 2008 Publication History
Extracting and composing robust features with denoising autoencoders
Pages 1096 - 1103
Abstract
Previous work has shown that the difficulties in learning deep generative or discriminative models can be overcome by an initial unsupervised learning step that maps inputs to useful intermediate representations. We introduce and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern. This approach can be used to train autoencoders, and these denoising autoencoders can be stacked to initialize deep architectures. The algorithm can be motivated from a manifold learning and information theoretic perspective or from a generative model perspective. Comparative experiments clearly show the surprising advantage of corrupting the input of autoencoders on a pattern classification benchmark suite.
References
[1]
Bengio, Y. (2007). Learning deep architectures for AI (Technical Report 1312). Université de Montréal, dept. IRO.
[2]
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems 19 (pp. 153--160). MIT Press.
[3]
Bengio, Y., & Le Cun, Y. (2007). Scaling learning algorithms towards AI. In L. Bottou, O. Chapelle, D. DeCoste and J. Weston (Eds.), Large scale kernel machines. MIT Press.
[4]
Bishop, C. M. (1995). Training with noise is equivalent to tikhonov regularization. Neural Computation, 7, 108--116.
[5]
Doi, E., Balcan, D. C., & Lewicki, M. S. (2006). A theoretical analysis of robust coding over noisy overcomplete channels. In Y. Weiss, B. Schöölkopf and J. Platt (Eds.), Advances in neural information processing systems 18, 307--314. Cambridge, MA: MIT Press.
[6]
Doi, E., & Lewicki, M. S. (2007). A theory of retinal population coding. NIPS (pp. 353--360). MIT Press.
[7]
Elad, M., & Aharon, M. (2006). Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 15, 3736--3745.
[8]
Gallinari, P., LeCun, Y., Thiria, S., & Fogelman-Soulie, F. (1987). Memoires associatives distribuees. Proceedings of COGNITIVA 87. Paris, La Villette.
[9]
Hammond, D., & Simoncelli, E. (2007). A machine learning framework for adaptive combination of signal denoising methods. 2007 International Conference on Image Processing (pp. VI: 29--32).
[10]
Hinton, G. (1989). Connectionist learning procedures. Artificial Intelligence, 40, 185--234.
[11]
Hinton, G., & Salakhutdinov, R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504--507.
[12]
Hinton, G. E., Osindero, S., & Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527--1554.
[13]
Hopfield, J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, USA, 79.
[14]
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., & Bengio, Y. (2007). An empirical evaluation of deep architectures on problems with many factors of variation. Proceedings of the 24th International Conference on Machine Learning (ICML'2007) (pp. 473--480).
[15]
LeCun, Y. (1987). Modèles connexionistes de l'apprentissage. Doctoral dissertation, Université de Paris VI.
[16]
Lee, H., Ekanadham, C., & Ng, A. (2008). Sparse deep belief net model for visual area V2. In J. Platt, D. Koller, Y. Singer and S. Roweis (Eds.), Advances in neural information processing systems 20. Cambridge, MA: MIT Press.
[17]
McClelland, J., Rumelhart, D., & the PDP Research Group (1986). Parallel distributed processing: Explorations in the microstructure of cognition, vol. 2. Cambridge: MIT Press.
[18]
Memisevic, R. (2007). Non-linear latent factor models for revealing structure in high-dimensional data. Doctoral dissertation, Departement of Computer Science, University of Toronto, Toronto, Ontario, Canada.
[19]
Ranzato, M., Boureau, Y.-L., & LeCun, Y. (2008). Sparse feature learning for deep belief networks. In J. Platt, D. Koller, Y. Singer and S. Roweis (Eds.), Advances in neural information processing systems 20. Cambridge, MA: MIT Press.
[20]
Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007). Efficient learning of sparse representations with an energy-based model. Advances in Neural Information Processing Systems (NIPS 2006). MIT Press.
[21]
Roth, S., & Black, M. (2005). Fields of experts: a framework for learning image priors. IEEE Conference on Computer Vision and Pattern Recognition (pp. 860--867).
[22]
Utgoff, P., & Stracuzzi, D. (2002). Many-layered learning. Neural Computation, 14, 2497--2539.
[23]
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders (Technical Report 1316). Université de Montréal, dept. IRO.
Information & Contributors
Information
Published In
ICML '08: Proceedings of the 25th international conference on Machine learning
July 2008
1310 pages
Copyright © 2008 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 July 2008
Permissions
Request permissions for this article.
Check for updates
Qualifiers
- Research-article
Conference
Acceptance Rates
Overall Acceptance Rate 140 of 548 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
Downloads (Last 12 months)1,589
Downloads (Last 6 weeks)180
Reflects downloads up to 19 Apr 2025
Other Metrics
Citations
- Yu HWang QZhou X(2025)Adaptive-weighted federated graph convolutional networks with multi-sensor data fusion for drug response predictionInformation Fusion10.1016/j.inffus.2025.103147122(103147)Online publication date: Oct-2025
- Liu HChen YHuang YSong TShen WTian YYou Y(2025)Clustering investigation of scramjet combustion processes based on contrastive learningActa Astronautica10.1016/j.actaastro.2025.03.036233(55-65)Online publication date: Aug-2025
- Song QYang LJiang XZhu Z(2025)Self-supervised progressive learning for fault diagnosis under limited labeled data and varying conditionsNeurocomputing10.1016/j.neucom.2025.130126637(130126)Online publication date: Jul-2025
- Du ZZhang YLiu ZWang GMa ZXie NYang Y(2025)SGCDiff: Sketch-Guided Cross-modal Diffusion Model for 3D shape completionNeurocomputing10.1016/j.neucom.2025.130025636(130025)Online publication date: Jul-2025
- Li ZZhang KZheng QDing GHao WZhang HZhang W(2025)Unsupervised fault detection with multi-source anomaly sensitivity enhancing convolutional autoencoder for high-speed train bogie bearingsExpert Systems with Applications10.1016/j.eswa.2025.127570281(127570)Online publication date: Jul-2025
- Malik FGorini DRicles JRahnemoonfar M(2025)Multi-physics framework for seismic Real-time Hybrid Simulation of soil–foundation–structural systemsEngineering Structures10.1016/j.engstruct.2025.120247334(120247)Online publication date: Jul-2025
- Dai WChen GPeng WChen CFu XLiu LLiu LYu N(2025)Domain alignment method based on masked variational autoencoder for predicting patient anticancer drug responseMethods10.1016/j.ymeth.2025.03.012238(61-73)Online publication date: Jun-2025
- Zhang JGuo YDong XWang TWang JMa XWang H(2025)Opportunities and challenges of noise interference suppression algorithms for dynamic ECG signals in wearable devices: A reviewMeasurement10.1016/j.measurement.2025.117067250(117067)Online publication date: Jun-2025
- Yasutomi STanaka T(2025)Style Feature Extraction Using Contrastive Conditioned Variational Autoencoders With Mutual Information ConstraintsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2025.354338337:5(3001-3014)Online publication date: May-2025
- Ullah HHeyat MBiswas TNeha NRaihan MLai D(2025)An end-to-end motion artifacts reduction method with 2D convolutional de-noising auto-encoders on ECG signals of wearable flexible biosensorsDigital Signal Processing10.1016/j.dsp.2025.105053160:COnline publication date: 1-May-2025
- Show More Cited By
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Full Access
View options
View or Download as a PDF file.
eReader
View online with eReader.