Database Reference
In-Depth Information
much interested in theoretical work that sets the foundation for how to think
about a problem, but I'm not impressed by vacuous theory, even if it's techni-
cally impressive.
Gutierrez:
What's the biggest thing you've changed your mind about?
LeCun:
Unsupervised learning. I've actually changed my mind two or three
times about this, and I'm probably not done. Back in the old days, when
I started working with neural nets in my undergrad and early grad student
days, I worked on unsupervised learning and never published anything about
it. I mean, not really. I did, but it wasn't that great. At that point, I thought that
unsupervised learning was really ill-defined. You can measure performance,
but there was really no objective way of saying, “Here is an objective function
to minimize. And I know if I minimize it, my system will work well.” And so,
because I thought that it was so badly formulated, I decided it was useless.
So I started working on and became a big believer in discriminative, purely
supervised learning. Convolutional nets are a result of this thinking. Basically,
there were similar kinds of architectures that were used before. Fukushima's
neocognitron is one, for example, which had a similar architecture to convo-
lutional nets. It's not the same, but it's really similar. They were trying to use
unsupervised learning mostly. I thought that that particular approach was insuf-
ficient, so convolutional nets are basically a supervised simplified version of
Fukushima's neocognitron. At the same time, my friend Vladimir Vapnik came
up with the theoretical argument that you should never try to solve a more
complex problem than you have to. In this case, unsupervised learning, to some
extent, is a more complex problem than, say, classification in the sense that it's
like learning a density in a high-dimensional space. That's like the hardest thing
you can imagine. So I was against unsupervised learning in some ways.
At the time, however, Geoff Hinton was actually a big advocate and believer of
unsupervised learning. It's funny because we've been sort of out of phase in our
beliefs about supervised learning versus unsupervised learning. So at the time
he was trying to convince me that I should work on unsupervised learning, all
the while I was telling him that the stuff that really works is actually supervised
learning. Then came the early 2000s and my opinion changed completely. I real-
ized that to really solve the deep learning problem with very deep networks,
perhaps you would need unsupervised learning to do pre-training. Geoff came
up with some idea on how to do this, and that was very inspiring.
So I started to get really interested in this work and unsupervised learn-
ing. I worked on a technique called “sparse auto-encoders,” which are now
relatively widely used, although not really for industrial applications. We got
a bunch of interesting work done with this technique and saw some improve-
ment on some data sets which were really, really interesting. Unsupervised
learning is more biologically plausible, as it's clear that the brain is trained
more in an unsupervised manner than a supervised manner. So this work had
