Jürgen Schmidhuber

Schmidhuber completed his undergraduate (1987) and PhD (1991) studies at the Technical University of Munich in Munich, Germany.[1] His PhD advisors were Wilfried Brauer and Klaus Schulten.[5] He taught there from 2004 until 2009. From 2009,[6] until 2021, he was a professor of artificial intelligence at the Università della Svizzera Italiana in Lugano, Switzerland.[1]

He has served as the director of Dalle Molle Institute for Artificial Intelligence Research (IDSIA), a Swiss AI lab, since 1995.[1]

In 2014, Schmidhuber formed a company, Nnaisense, to work on commercial applications of artificial intelligence in fields such as finance, heavy industry and self-driving cars. Sepp Hochreiter, Jaan Tallinn, and Marcus Hutter are advisers to the company.[2] Sales were under US$11 million in 2016; however, Schmidhuber states that the current emphasis is on research and not revenue. Nnaisense raised its first round of capital funding in January 2017. Schmidhuber's overall goal is to create an all-purpose AI by training a single AI in sequence on a variety of narrow tasks.[7]

In the 1980s, backpropagation did not work well for deep learning with long credit assignment paths in artificial neural networks. To overcome this problem, Schmidhuber (1991) proposed a hierarchy of recurrent neural networks (RNNs) pre-trained one level at a time by self-supervised learning.[8] It uses predictive coding to learn internal representations at multiple self-organizing time scales. This can substantially facilitate downstream deep learning. The RNN hierarchy can be collapsed into a single RNN, by distilling a higher level chunker network into a lower level automatizer network.[8][9] In 1993, a chunker solved a deep learning task whose depth exceeded 1000.[10]

In 1991, Schmidhuber published adversarial neural networks that contest with each other in the form of a zero-sum game, where one network's gain is the other network's loss.[11][12][13] The first network is a generative model that models a probability distribution over output patterns. The second network learns by gradient descent to predict the reactions of the environment to these patterns. This was called "artificial curiosity." In 2014, this principle was used in a generative adversarial network where the environmental reaction is 1 or 0 depending on whether the first network's output is in a given set. This can be used to create realistic deepfakes.

Schmidhuber may be best known for his early research on the long short-term memory (LSTM), a type of recurrent neural network. The LSTM was developed by his student Sepp Hochreiter and initially reported on in his 1991 diploma thesis which analyzed and overcame the famous vanishing gradient problem.[14] The name LSTM was introduced in a tech report (1995) leading to the most cited LSTM publication (1997), co-authored by Hochreiter and Schmidhuber.[15] Schmidhuber, Hochreiter, and Schmidhuber's other students, including Felix Gers, Fred Cummins, and Alex Graves, and others, Schmidhuber published increasingly sophisticated versions of the LSTM.

The standard LSTM architecture which is used in almost all current applications was introduced in 2000.[16] Today's "vanilla LSTM" using backpropagation through time was published in 2005,[17][18] and its connectionist temporal classification (CTC) training algorithm[19] in 2006. CTC enabled end-to-end speech recognition with LSTM. By the 2010s, the LSTM became the dominant technique for a variety of natural language processing tasks including speech recognition and machine translation, and was widely implemented in commercial technologies such as Google Translate and Siri.[20] LSTM has become the most cited neural network of the 20th century.[9]

In 2015, Rupesh Kumar Srivastava, Klaus Greff, and Schmidhuber used LSTM principles to create the Highway network, a feedforward neural network with hundreds of layers, much deeper than previous networks.[21][22] 7 months later, the ImageNet 2015 competition was won with an open-gated or gateless Highway network variant called Residual neural network.[23] This has become the most cited neural network of the 21st century.[9]

Since 2018, transformers have overtaken the LSTM as the dominant neural network architecture in natural language processing[24] through large language models such as ChatGPT. As early as 1992, Schmidhuber published an alternative to recurrent neural networks[25] which is now called a Transformer with linearized self-attention[26][27][9] (save for a normalization operator). It learns internal spotlights of attention:[28] a slow feedforward neural network learns by gradient descent to control the fast weights of another neural network through outer products of self-generated activation patterns FROM and TO (which are now called key and value for self-attention).[26] This fast weight attention mapping is applied to a query pattern.

In 2011, Schmidhuber's team at IDSIA with his postdoc Dan Ciresan also achieved dramatic speedups of convolutional neural networks (CNNs) on fast parallel computers called GPUs. An earlier CNN on GPU by Chellapilla et al. (2006) was 4 times faster than an equivalent implementation on CPU.[29] The deep CNN of Dan Ciresan et al. (2011) at IDSIA was already 60 times faster[30] and achieved the first superhuman performance in a computer vision contest in August 2011.[31] Between 15 May 2011 and 10 September 2012, their fast and deep CNNs won no fewer than four image competitions.[32][33] They also significantly improved on the best performance in the literature for multiple image databases.[34] The approach has become central to the field of computer vision.[33] It is based on CNN designs introduced much earlier by Yann LeCun et al. (1989)[35] who applied the backpropagation algorithm to a variant of Kunihiko Fukushima's original CNN architecture called neocognitron,[36] later modified by J. Weng's method called max-pooling.[37][33]

Schmidhuber has controversially argued that he and other researchers have been denied adequate recognition for their contribution to the field of deep learning, in favour of Geoffrey Hinton, Yoshua Bengio and Yann LeCun, who shared the 2018 Turing Award for their work in deep learning.[2][20][38] He wrote a "scathing" 2015 article arguing that Hinton, Bengio and Lecun "heavily cite each other" but "fail to credit the pioneers of the field".[38] In a statement to the New York Times, Yann LeCun wrote that "Jürgen is manically obsessed with recognition and keeps claiming credit he doesn't deserve for many, many things... It causes him to systematically stand up at the end of every talk and claim credit for what was just presented, generally not in a justified manner."[2] Schmidhuber replied that LeCun did this "without any justification, without providing a single example,"[39] and published details of numerous priority disputes with Hinton, Bengio and LeCun.[40] Some have suggested that Schmidhuber's accomplishments have been downplayed because of his personality.[20]

Schmidhuber received the Helmholtz Award of the International Neural Network Society in 2013,[41] and the Neural Networks Pioneer Award of the IEEE Computational Intelligence Society in 2016[42] for "pioneering contributions to deep learning and neural networks."[1] He is a member of the European Academy of Sciences and Arts.[43][6]