Listen: https://serve.buildtall.systems/680a2db8184b0ab9e3c3283ee31d959a42dd65e5e70c3742d331d73f13b100fd.mp3
Source: https://serve.buildtall.systems/e971d466392c52f0a5212fcacc022c1cffab16fcf120ff25106bd71ed685557d.pdf
This episode covers Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, a preprint by Jürgen Schmidhuber of TU Munich and IDSIA.
The paper argues that data becomes interesting to a learner for a limited time, once the learner finds a better way to predict or compress it. In the abstract's words, "data becomes temporarily interesting by itself" to a self-improving but computationally limited observer. The author calls the drive behind this compression progress and argues that it can account for beauty, novelty, surprise, curiosity, creativity, art, science, music and jokes. This is a conceptual and theoretical paper. It reports no new experiment and no new dataset. Its support is the reasoning it lays out, informal examples such as faces and drawings, and a review of earlier implementations from 1990 to 2008 that the author says approximate the framework. The formal details are deferred to an appendix, and parts of that appendix describe optimal learners that the paper itself says cannot be computed in finite time. The account is clear and internally consistent, and its link between curiosity and learning is easy to follow. Its weaker points are that the claims about art, music and humour are argued by example rather than tested, and the paper itself says that tests in psychology and neuroscience remain to be done. The text available here contains no funding or data statement, so this episode does not describe one. The document is an arXiv preprint in the cs.AI category, first published in December 2008 and revised in April 2009. The paper refers to a short and a long version of the work and does not name a journal. OpenAlex has no record of this document in its index, so this episode makes no claim about how often it has been cited.
The paper opens with a thought experiment. It asks what would follow if the history of the entire universe were computable, and says there is no evidence against that possibility. In that case, the authors write, "its simplest explanation would be the shortest program that computes it." The difficulty is that finding such a program is not straightforward. In the paper's words, "there is no general way of finding the shortest program computing any given data." Physicists have therefore worked incrementally. They study one small aspect of the world at a time and look for a simple law that describes their limited observations better than the best law known so far. The paper summarizes this as "trying to find a program that compresses the observed data better than the best previously known program."
To compress data here means to find a shorter description that still reproduces it. Newton's law of gravity is the paper's example. It can be written as a short piece of code that substantially compresses many observation sequences about falling apples and other objects. The law has clear limits, since it does not explain quantum fluctuations of apple atoms. It still reduces the number of bits needed to encode the data stream, "by assigning short codes to events that are predictable with high probability" under the assumption that the law holds. Einstein's general relativity adds further compression progress, because it compactly explains many deviations from Newton's predictions that had gone unexplained. The authors note that most physicists believe there is still room for further advances.
The paper then argues that this drive is not limited to physicists. Short, simple explanations of the past usually reflect a repeating regularity, and that regularity helps predict the future too. From this the authors draw their central premise: "every intelligent system interested in achieving future goals should be motivated to compress the history of raw sensory inputs in response to its actions, simply to improve its ability to plan ahead."
The paper also points to earlier thinking on the same idea. The psychologist Piaget explained how children learn by exploring, using two concepts. Assimilation means new inputs are fitted into existing mental schemas, which the paper reads as a form of compression. Accommodation means an old schema is adjusted to fit a new input, which the paper reads as an improvement in compression. The authors credit Piaget's ideas but add a limitation: "his informal ideas did not provide enough formal details to permit computer implementations of his concepts."
The open question is how to build a compression progress drive into an artificial system. The setting is an active agent that interacts with a world it initially knows nothing about. The authors use their reinforcement learning framework of artificial curiosity, developed between 1990 and 2008, to make the agent seek out data that allows further compression progress and better predictability. Reinforcement learning is a method in which an agent learns by acting and receiving rewards. The framework steers the agent toward understanding its world through exploration, even when outside rewards are rare or absent. It does this through what the paper calls "intrinsic reward or curiosity reward for actions leading to discoveries of previously unknown regularities in the action-dependent incoming data stream." In other words, the agent is rewarded from within for finding patterns it had not yet found.
The paper previews the framework it builds on, which has three parts. The first is a predictor or compressor that keeps improving, described as "a continually improving predictor or compressor of the continually growing data history." The second is a computable measure of how much that compressor is improving, which yields the intrinsic rewards. The third is a reward optimizer, or reinforcement learner, that turns those rewards into action sequences expected to maximize future reward. The authors say the formal details and discrete-time implementations are deferred to an appendix. They also say that a later part of the paper will discuss what they mean by external reward, where external means originating outside the brain that controls the actions of its body. The preview then promises that many features of intelligence and cognition, including novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music and jokes, follow from this framework. It states that the authors reject the traditional Boltzmann and Shannon notion of surprise. It also says the authors will argue that both science and art are by-products of the desire to create or discover more data that is compressible in previously unknown ways. Finally, the preview lists the paper's planned later contents: a survey of earlier implementations that approximate the framework, applications to images designed for human viewers, and suggestions for improving those implementations and for testing the theory in psychology and neuroscience.
The paper then lays out its algorithmic framework, a set of simple principles that distill ideas from earlier publications on the topic. It says the formal details are left to an appendix, and that the principles at least qualitatively explain many aspects of intelligent agents such as humans. That is the authors' reason for wanting to implement and test them in cognitive robots and other artificial systems.
The first principle is to "store everything." The paper writes that "the data is holy as it is the only basis of all that can be known about the world," so an agent should keep the entire raw history of its actions, sensory observations and reward signals. To show this is not unrealistic, the authors make a rough estimate. A human lifetime lasts about 3 billion seconds, and the brain has roughly 10 billion neurons, each with around 10,000 synapses. Assuming only half of that capacity is used for raw data, and that each synapse stores at most 6 bits, they conclude the brain has enough room to encode the lifelong sensory stream at roughly 100,000 bits per second. They compare that rate to the demands of a movie at reasonable resolution, note that affordable technical systems will soon exceed it, and close with the line
If you can store the data, do not throw it away!
The second principle is to improve what the paper calls subjective compressibility. Any regularity in the stored history can be used to compress it, and the compressed version is treated as the data's simplifying explanation. So the agent should spend some of its computation on an adaptive compression method that tries to predict or postdict parts of its history from other parts, reducing the number of bits needed to encode the whole. Postdicting means predicting an earlier item from other data, including data that came later. The paper offers an adaptive neural network as one example of such a compressor.
The third principle ties curiosity to that process. The agent monitors how its compressor is improving. Whenever the compressor learns to encode the history in fewer bits, the agent generates an intrinsic reward, meaning a reward produced inside the agent rather than supplied from outside. Its size is proportional to the learning progress, or compression progress, measured as the number of bits saved.
The fourth principle hands this curiosity reward to a general reinforcement learning algorithm, a method that learns which actions lead to the most reward over time. That algorithm can observe the state of the compressor, and it maximizes expected reward including the intrinsic part. The paper expects the result to be that the agent chooses actions which focus its attention and learning on aspects of the world where new, previously unknown but learnable regularities can be found or created. In the paper's words, "it will try to maximize the steepness of the compressor's learning curve." The authors call this active unsupervised learning, and they argue it can help an agent figure out how the world works.
The paper is explicit about what the framework does and does not fix. It "essentially specifies the objectives of a curious or creative system," but not the way of reaching them, which depends on the choice of a particular adaptive compressor or predictor and a particular reinforcement learning algorithm. Several such choices produce special cases of the framework, including earlier concrete implementations, which the paper says it returns to later.
Next the paper turns to how curiosity relates to the external problems an agent is actually given. Many cognitive systems, it notes, do not aim only to satisfy their curiosity. Their real goal is "to solve externally given problems." The paper states that any formalizable problem can be phrased as a reinforcement learning problem, in which an agent in a possibly unknown environment tries to maximize the external reward it expects to collect until the end of its possibly finite lifetime. Reinforcement learning, or RL, is learning by trial and reward: the agent acts, the world responds, and some responses carry reward.
The paper acknowledges that recent theory has produced a few extremely general RL algorithms, sometimes called universal problem solvers or universal artificial intelligences. These are optimal in various theoretical senses, though not necessarily in practical ones. Its argument is that if learning progress, compression progress and curiosity help an agent, these universal methods would find and exploit such concepts on their own. That raises the question of why an explicit framework for curiosity-based experimentation is needed at all.
The paper's answer is that the universal approaches sweep certain problem-independent constant slowdowns under the carpet, burying them in asymptotic notation. Asymptotic notation is the shorthand of computer science that describes how running time grows for large inputs and sets aside fixed factors. The paper then poses what it calls an essential remaining question. If an agent can execute only a fixed number of computational instructions per unit of time, say 10 trillion elementary operations per second, what is the best way to use them to get as close as possible to the theoretical limits of universal AIs? The paper stresses that this matters "especially when external rewards are very rare, as is the case in many realistic environments".
The premise is that the curiosity drive is a general and broadly useful concept for resource-limited reinforcement learning when rewards are rare. For that reason, the paper argues, it should be prewired, meaning built in, rather than learned from scratch, which would spend computation that may still be huge even when its cost is constant. The paper states an inherent assumption: in realistic worlds, a better explanation of the past can only help predict the future and speed up the search for solutions to external tasks. It sets aside the possibility that curiosity may actually be harmful, and that it could "kill the cat."
The paper turns to what it calls the consequences of the compression drive, and asks how many core ingredients of intelligence and cognition might arise as side effects of that drive. The first it takes up is internal representations, or symbols. The authors write that "the compressor (say, a predictive neural network) will automatically create internal representations or symbols (for example, patterns across certain neural feature detectors) for things that frequently repeat themselves." In plain terms, when a system tries to squeeze its history of observations into less space, it gives recurring things their own internal labels, much as a shorthand reuses a brief code for a long phrase.
The paper adds that compression does not require perfect predictability to pay off.
Even when there is limited predictability, efficient compression can still be achieved by assigning short codes to events that are predictable with high probability.
Its example is a simple one: "the sun goes up every day." Because that regularity recurs, it is cheaper to store a short internal symbol such as daylight, which stands for the repeating pattern, than to keep a raw record of every sunrise.
The paper then notes that this is not only a theoretical possibility. Existing systems show it.
In fact, predictive neural networks are often observed to create such internal (and hierarchical) codes as a by-product of minimizing their prediction error on the training data.
Hierarchical here means the codes are layered, with simple internal labels combined into labels for larger patterns. The argument is that symbol formation follows from the drive to predict and compress well, so no separate mechanism for building concepts is needed.
The paper then turns to consciousness, which it treats as a by-product of the same compression process. Its starting point is a simple observation: there is one thing involved in all of the agent's actions and sensory inputs, and that is the agent itself. The authors reason that to encode its entire data history efficiently, the agent will profit from building some internal symbol or code, such as a pattern of neural activity, that represents the agent itself. Whenever that representation is actively used, for example when new sensory input activates the corresponding neurons or when it is called up some other way, the agent could be called self-aware or conscious.
The paper says this account "apparently does not abandon any essential aspects of our intuitive concept of consciousness", yet it is far simpler than several other recent theories of the subject. The authors say they do not attach any particular mystical meaning to consciousness. In their view, it is just a natural by-product of the agent's ongoing process of problem solving and world modeling through data compression. Consciousness, they add, will not play a prominent role in the rest of the paper.
The paper's account of beauty starts from what it calls a lazy brain theory. Its central claim is that the subjective beauty of a new observation depends on how cheaply the observer can encode it. The paper says the subjective beauty of a new observation D is "proportional to the number of bits required to encode D, given the observer’s limited previous knowledge embodied by the current state of its adaptive compressor." Put simply, the more an observer's existing knowledge already lets it compress something, the more beautiful that thing feels.
The paper illustrates this with faces. A compressor, such as a neural network, can learn a prototype face that captures what the observer has already seen. Then, in the paper's words,
To encode a new face, it must only encode the deviations from the prototype.
A new face that stays close to that prototype needs few extra bits, so the paper argues it will look more beautiful than faces that deviate a lot. The same logic covers faces with symmetries or simple proportions, because a compressor can exploit any regularity that reduces the number of bits needed to store the data.
From this the paper draws a general rule. Among patterns an observer treats as comparable, the most beautiful one is "the one with the simplest (shortest) description, given the observer’s current particular method for encoding and memorizing it." Mathematicians, for example, find beauty in a simple proof with a short description in the formal language they use, and many people like geometrically simple, low-complexity drawings of everyday objects.
The paper also draws a consequence about personal taste. What a person sees every day shapes their internal prototype face, so the paper says this "immediately explains why many human observers prefer faces similar to their own." The explanation is coding efficiency, not anything mysterious.
The second idea is interestingness, which the paper keeps separate from beauty. A beautiful thing holds our attention only for a while. The paper puts it this way:
A beautiful thing is interesting only as long as it is new, that is, as long as the algorithmic regularity that makes it simple has not yet been fully assimilated by the adaptive observer who is still learning to compress the data better.
To capture this, the paper defines interestingness as the rate at which beauty changes over time, which it calls
the first derivative of subjective beauty.
Imagine an observer whose compressor keeps improving. Data that first looked random becomes regular, and it takes fewer and fewer bits to encode. As that happens, the data becomes more beautiful to the observer, and its interest depends on how fast that improvement is happening. In the paper's terms, interestingness tracks the steepness of the learning curve. Data is most engaging while the observer is still climbing that curve, and it loses its pull once the regularity has been absorbed. The paper states the consequence directly:
As long as this process is not over the data remains interesting and rewarding.
The paper says the details of discrete-time versions of this idea come later, in its appendix and in its survey of earlier implementations.
The authors first clarify that the notions of beauty and interestingness they have been building are limited and pristine. By pristine they mean that these notions are not tied in advance to pleasure from external rewards. They give an example of what they set aside. Someone might say a hot bath on a cold day feels beautiful because it satisfies prewired target values of external temperature sensors, where external means outside the brain that controls the body. A song might likewise be called beautiful for emotional reasons, because a listener links it to memories of external pleasure, such as a first kiss. The authors state that
Obviously this is not what we have in mind here.
Their focus is "solely on rewards of the intrinsic type based on learning progress," meaning rewards that come from the agent getting better at predicting and compressing its own experience.
The text then turns to novelty and surprise and sets them against the traditional view from information theory. Compression progress means finding ever shorter descriptions of what has already been experienced. The authors begin with two extreme cases of dull data. The first is a vision-based agent that always stays in the dark. Its history is an unchanging stream of visual input, which becomes extremely compressible and soon totally predictable. The second is a screen full of white noise. Under the traditional notion of surprise, which the authors tie to "the traditional sense of Boltzmann and Shannon," that screen carries a great deal of information and so counts as novel and surprising. Yet each new pattern is highly unpredictable and cannot be compressed at all. The authors call both cases boring, because neither allows any further compression progress. For this reason they "reject the traditional notion of surprise."
In their place they offer a conclusion:
Neither the arbitrary nor the fully predictable is truly novel or surprising\u2014only data with still unknown algorithmic regularities are.
In plain terms, random data has no pattern to find, and fully predictable data has nothing left to learn. Only data that hides regularities not yet discovered, patterns a short program could describe but the agent has not yet found, counts as truly novel or surprising.
Here the paper turns to what it calls attention, curiosity and active experimentation. When there is no external reward to chase, or no known way to raise the reward already on offer, the authors say their controller "essentially tries to maximize true novelty or interestingness, the first derivative of subjective beauty or compressibility, the steepness of the learning curve." In plain terms, that first derivative is a rate of change: how quickly the agent's ability to compress its own experience is improving. The controller therefore chooses action sequences that it expects will produce observations yielding the most compression progress in the future, and the paper stresses that this is bounded by "the limitations of both the compressor and the compressor improvement algorithm." The result, the paper argues, is that attention settles on things that are still incompressible but expected to become compressible or predictable through further learning. Things that are already compressible lose their pull:
It will get bored by things that already are subjectively compressible.
The controller also loses interest in things that seem likely to stay incompressible given the experience so far, and in things where the cost of making them compressible exceeds the cost of making other things compressible. The paper points to a long run of earlier work on attention and curiosity to support these claims.
The paper gives a name to especially large jumps in compression: "An unusually large compression breakthrough deserves the name discovery." To make this concrete, the author points to the law of gravity. That law can be written as a very short piece of code, yet it lets an observer compress a great many earlier observations, from falling apples to other falling objects, into one brief description. In this framing, a discovery is a rule that shrinks a whole history of data far more than anything known before.
The paper then turns to how this idea relates to unsupervised learning, the branch of machine learning that looks for structure in data without being given labels or answers. The author argues that most traditional unsupervised learning is already a form of data compression. Grouping data into clusters, for example, lets each data point be stored as its cluster center plus relatively few bits for how it deviates from that center. Where data repeats itself, a non-redundant encoding does better than the raw data, and where some parts of the data can be predicted from others, those predictable parts can be given short codes.
The paper's objection is that this kind of learning is passive. In its words, traditional unsupervised learning "just analyzes and encodes the data but does not choose it." The learner takes whatever data it is given. The author proposes extending it with active action selection, so that the learner also chooses the actions that shape the data it sees. The comparisons are to a scientist choosing experiments, a baby choosing toys, an artist choosing colors, a dancer choosing moves, or any attentive system choosing its next sensory input. The paper says its reinforcement-learning framework for curiosity and creativity is built to do exactly this.
The final stretch makes a broader claim about art and music. The paper grants that works of art and music may have important purposes beyond their social aspects, despite those who regard art as superfluous. It then offers its own account. Good art that depends on the observer, the paper says, "deepens the observer's insights about this world or possible worlds." Such art unveils regularities in data that could be compressed, and it connects previously disconnected patterns in a way that first surprises and then makes those patterns easier to compress together. The paper calls this art as an eye-opener. Once the connection is absorbed, the work becomes known and less interesting.
The author states the central proposal directly: "the active creation and attentive perception of all kinds of artwork are just by-products of our principle of interestingness and curiosity yielding reward for compressor improvements." On this view, making and enjoying art is not a separate faculty. It follows from the same drive that rewards finding better compression.
The paper spells out what this means for the observer. Art is perceived sequentially and usually actively, through shifting attention such as eye movements scanning a sculpture, or a listener's attention filtering a pianist's sounds out of background noise. Observers differ in their senses and in how they compress new input, so different observers will prefer different sequences. For that reason, the paper argues, "any objective theory of what is good art must take the subjective observer as a parameter." The question it asks is which sequence of actions and shifts of attention an observer should follow to maximize his pleasure. Its answer is that the observer should pick the sequence that maximizes "the quickly learnable compressibility that is new, relative to his current knowledge."
Music is the paper's next example, and it uses the choice of a song to show how the principle works. Picture a listener deciding what to play next. He should not pick the song he just heard ten times in a row, because it "became too predictable in the process." Nor should he pick a brand-new piece with an unfamiliar rhythm and tonality, which "seems too irregular and contain too much arbitrariness and subjective noise." The better choice sits between the two: a song "unfamiliar enough to contain somewhat unexpected harmonies or melodies or beats etc., but familiar enough to allow for quickly recognizing the presence of a new learnable regularity or compressibility in the sound stream." The author adds a note of realism:
Sure, this song will get boring over time, but not yet.
The paper says that taste depends on the observer. As evidence it points to Schönberg's twelve-tone music, which it says is less popular than certain pop tunes, presumably because its algorithmic structure is less obvious to many listeners. Twelve-tone music often relies on frequency ratios between successive notes that cannot be written as fractions of very small whole numbers, so the underlying order is hard for an untrained ear to hear. Listeners who have learned the basic concepts, objectives and constraints of twelve-tone music tend to appreciate it more than those who have not. In the paper's account, the same piece can be interesting to one listener and dull to another, depending on what each has already learned.
The paper argues that all of this fits its principle. Put simply, the learning algorithm inside an observer's compressor, which is a learned model that finds shorter descriptions of experience, tries to compress the history of sounds and other inputs better where it can. The action selector looks for choices that influence that history and help the compressor perform better on what it has already seen. The paper draws the conclusion directly:
The interesting musical and other subsequences are those with previously unknown yet learnable types of regularities, because they lead to compressor improvements.
By contrast,
The boring patterns are those that seem arbitrary or random, or whose structure seems too hard to understand.
The paper extends its account beyond music to other art forms. It says that similar statements "not only hold for other dynamic art including film and dance," and it adds a caveat for those forms: the compressibility of the controller actions has to be taken into account. Painting and sculpture are static, but the paper argues they are also "which cause dynamic pattern sequences due to attention-shifting actions" of the observer. As the viewer's attention moves across a canvas or a statue, the work becomes a changing stream of patterns, and so it can be a source of the same kind of compression progress that a moving work provides.
The paper then sets its approach against an older tradition. Earlier information-theoretic attempts to explain aesthetic experience "emphasized the idea of an "ideal" ratio between expected and unexpected information" carried by an aesthetic object, which these accounts describe as a balance between its "order" and its "complexity". On that view, a pleasing object strikes a particular mix of predictability and surprise. The paper says its own approach "does not have to postulate an objective ideal ratio of this kind." Instead, its dynamic measure of interestingness "reflects the change in the number of bits required to encode an object". The measure is relative to the observer, since it "explicitly takes into account the subjective observer's prior knowledge as well as the limitations of its compression improvement algorithm." In other words, there is no fixed sweet spot that every person should find beautiful. What matters is how much the observer's own compression of the object improves over time.
Artists, the paper suggests, are rewarded in much the same way as observers. Just as observers get intrinsic rewards for sequentially focusing attention on artwork that exhibits new, previously unknown regularities, the creative artists get reward for making it. The author gives a personal example: "I found it extremely rewarding to discover (after hundreds of frustrating failed attempts) the simple geometric regularities that permitted the construction of the drawings" he made. The reward came from finding the hidden rule behind the image, not from the image alone.
The paper says that the line between artist and observer is blurred.
The distinction between artists and observers is blurred though.
Both carry out action sequences that reveal new kinds of compressibility, and the authors argue that
The intrinsic motivations of both are fully compatible with our simple principle.
Some artists also want external rewards such as praise or money, the paper notes, in addition to the intrinsic reward of making a truly novel work. The principle, the paper argues, "conceptually separates these two reward types."
The paper then compares scientists with artists. "From our perspective, scientists are very much like artists." Both select experiments or works in search of simple but new regularities that compress the record of what they have observed. The paper claims that the creativity of painters, dancers, musicians, pure mathematicians and physicists "can be viewed as a mere by-product of our curiosity framework based on the compression progress drive." Each group tries to make "new but non-random, non-arbitrary data with surprising, previously unknown regularities." Physicists may build experiments that produce data governed by laws nobody has yet found, so that the data can be compressed further. Many artists combine familiar objects in a fresh way, so that the observer's description of the result is shorter than the descriptions of the parts added together, because of a regularity shared by those parts that nobody had noticed before.
The paper locates the difference between science and art in how the progress is recorded.
The essence of science is to formally nail down the nature of compression progress achieved through the discovery of a new regularity.
As an example, the law of gravity can be written in just a few symbols. In the fine arts, the compression progress from an artwork that joins previously unconnected things may be subconscious. Such progress "may be subconscious and not at all formally describable by the observer," who feels it as intrinsic reward without being able to say which of their memories became more compressible.
Finally, the paper notes that its framework is formal enough to be implemented on computers. The artificial observers that result would differ in the power of their history compressors and learning algorithms, and so
This will influence what is good art / science to them, and what they find interesting.
The paper turns next to jokes and other sources of fun. Comedians, like other entertainers and artists, often combine well-known concepts in a new way. The paper's claim is that the result makes the observer's description shorter, so that, in its framing, the observer's subjective description of the result is shorter than the sum of the lengths of the descriptions of the parts, because of some previously unnoticed regularity the parts share. On this account, a joke gets its laugh because it lets the listener compress familiar material more efficiently than before.
The author links this to learning new skills. Past the age of 25, the author learnt to juggle three balls. The progress was not sudden. At first the juggling lasted about one second before the balls fell, then two seconds, then four, and so on until it was done properly. Watching in a mirror, as juggling teachers recommend, the author noticed "an idiotic grin across my face" whenever progress was made. The author's young daughter grinned in the same way when she first stood on her own feet.
The paper explains these grins within its algorithmic framework. Such grins are presumably triggered by an intrinsic reward for generating a stream of data with previously unknown regularities. Watching oneself juggle produces a sensory stream that differs from the familiar experience of watching someone else juggle, so it is truly novel and intrinsically rewarding, for as long as the adaptive predictor or compressor is still getting used to it.
The paper then turns to earlier systems. It recalls that predictors and compressors are closely related. As the paper puts it,
Any type of partial predictability of the incoming sensory data stream can be exploited to improve the compressibility of the whole.
For that reason, the systems described in the first publications on artificial curiosity can already be viewed as implementations of a compression progress drive.
The paper's account starts in 1990 with an early version of the idea. Its authors describe a predictor built on a recurrent neural network, a type of network whose connections loop back so that it can carry information forward from everything it has already seen. The paper calls this, in principle, "a rather powerful computational device, even by today's machine learning standards." The predictor tried to forecast the agent's sensory inputs, including reward signals, from the entire history of earlier inputs and actions. In this design the curiosity reward was set in proportion to the predictor's errors, so the bigger the miss, the bigger the reward. The paper notes that this rested on an assumption: it "was implicitly and optimistically assumed that the predictor will indeed improve whenever its error is high."
Follow-up work in 1991 pointed out the problem with that assumption. In the paper's words, the approach "may be inappropriate, especially in probabilistic environments." The remedy it proposed is stated directly:
one should not focus on the errors of the predictor, but on its improvements.
The reason is that in a world with randomness, a predictor can keep producing large errors on noise that never goes away. A system rewarded for errors would keep returning to those spots forever. Noise, or the limited computing power of the predictor, would block any real gain in how briefly the data can be described, which is what the paper means by subjective compressibility.
The 1991 implementation used a neural predictor that the paper says was computationally weaker than the 1990 one. What it added was an explicit adaptive model of the predictor's improvements. This second network learned to predict how the predictor's own settings would change. The paper gives noise as an example. Noise produces wildly varying target signals, but over the long run those signals barely shift the predictor's parameters, and the model of changes was able to learn that pattern. A standard reinforcement learning algorithm, a general method by which an agent learns which actions lead to rewards, was then fed curiosity rewards proportional to the expected long-term changes in the predictor. The system, the paper says, "tried to maximize information gain" within the limits it had.
The paper sums up the goal in mathematical terms. It says that "the system tried to maximize an approximation of the (discounted) sum of the expected first derivatives of the data's subjective predictability." Put plainly, the system aimed for the largest expected gains in how predictable its experience becomes, with the rate of change counted over time. The word "discounted" means that gains further in the future count for less than gains nearby. The authors add that this also amounts to maximizing an approximation of the expected changes in the data's subjective compressibility, so the two ways of describing the drive come to the same thing.
The paper then describes a variant of the same idea for worlds that cannot be fully predicted. In 1995, the curiosity reward was again tied to the predictor's surprise or information gain, but this time it was measured as the Kullback-Leibler distance, a standard way of comparing two probability distributions. The distance was taken between the learning predictor's subjective probability distributions before and after each new observation. The paper puts it as "the relative entropy between its prior and posterior." In plain terms, the agent is rewarded by how much a new observation changes what it believes.
The paper notes that the ideas behind Huffman coding, a classic method that gives frequent symbols short codes, and this prior-to-posterior measure both translate into a measure of learning progress. That progress is the number of saved bits, which the paper calls "a measure of improved data compression."
According to the paper, Baldi and Itti in 2005 called this approach "Bayesian surprise." They tested it experimentally and found that it explains certain patterns of human visual attention better than certain earlier approaches.
The paper then warns of a limit. It states that the naive probabilistic approach to data compression is "unable to discover more general types of algorithmic compressibility." Its example is the decimal expansion of pi, which "looks random and incompressible but isn't." A very short algorithm computes all of pi. Yet any finite sequence of digits appears in pi's expansion as often as it would if pi were truly random. So a simple statistical learner that predicts the next digit from a limited window of earlier digits does no better than random guessing. The paper turns next to more general program search as the way past this limit.
The paper then turns to a more recent line of work from 1997, which it says greatly increased the computational power of the controller and the predictor. In that design, the two are built as co-evolving, symmetric, opposing modules. Each is made of self-modifying probabilistic programs, which are small programs that include a random element and can rewrite parts of themselves, written in a universal programming language that allows loops, recursion and hierarchical structures. The internal storage where these programs keep temporary results is treated as part of the changing environment. Each module can propose an experiment, in the form of a probabilistic algorithm to be run, and can make a confident prediction about its outcome by betting on it. The betting money plays the role of the intrinsic reward. The opposing module can accept or reject the bet, which amounts to making a contrary prediction, so the two sides form a zero-sum game in which one side's gain is the other's loss. If the bet is accepted, the experiment is actually executed and its outcome checked to decide the winner, and the money passes from the surprised loser to the confirmed winner. Both modules try to maximize their money using a general reinforcement learning algorithm designed for complex stochastic policies, which is a rule for choosing actions when outcomes are uncertain. The paper notes that other reinforcement learning algorithms could be plugged in as well. The result, the authors argue, is that both modules are pushed to find truly novel algorithmic regularity, meaning compressibility that the other side does not already know. The baseline for novelty is set by what the opponent already knows about the world's repeating patterns. In the paper's words,
the method can be viewed as system identification through co-evolution of computable models and tests.
The paper also points to a 2005 study by Bongard and Lipson that followed a similar co-evolutionary approach, though with less general models and tests.
The paper next asks whether this kind of reward helps in practice. It says that its earlier cited experiments showed the effect directly: "the presence of intrinsic reward or curiosity reward actually can speed up the collection of external reward." In plain terms, an agent that is paid for learning something new, on top of the rewards the task itself gives, can gather task rewards faster than one that receives only the task rewards. Intrinsic reward is the paper's term for the internal payoff an agent gives itself, while external reward is what the environment provides.
The paper then turns to other researchers who built versions of the same idea. It says several groups implemented "variants or approximations of the curiosity framework," and it singles out Singh and Barto and their coworkers. They worked within the option framework of reinforcement learning, a method for grouping sequences of actions into reusable skills, and they used prediction errors directly as curiosity rewards. The paper credits them with a naming contribution: "they actually were the ones who coined the expressions intrinsic reward and intrinsically motivated RL." Here RL means reinforcement learning, the general approach in which an agent learns by acting and receiving rewards. The paper adds that further implementations were presented at the 2005 AAAI Spring Symposium on Developmental Robotics, and it points readers to a special issue of the journal Connection Science.
The paper then moves from agents to images. It recalls that its probabilistic version of the theory, published in 1995, could explain certain shifts of human visual attention reported in 2005. The new aim is the reverse problem: building images that contain regularities a viewer can learn quickly. The paper argues again that there is no basic difference between the motivation of a creative artist and that of a passive observer of art. In its account, both create or select sequences of actions that yield interesting inputs, and interestingness is measured as learning progress. Learning progress is the gain a learner makes as its model of the data improves. The paper names one measure, the relative entropy between a learner's prior and posterior beliefs, which captures how much those beliefs shift on seeing new data. It names another, the number of bits saved when encoding the data, and allows for similar measures.
The paper says it offers examples of subjective beauty tailored to human observers, meant to show the learning process that moves a viewer from less beauty to more. Because the medium is written, it uses visual examples where sound or touch would otherwise be the natural choice. The text here introduces those examples and does not describe them in detail, so the listener should take the visual claims as the paper's stated intent.
The paper's central hypothesis is put in its own words: "unsupervised attention and the creativity of artists, dancers, musicians, pure mathematicians are just by-products of their compression progress drives." Unsupervised attention means attending to things without a teacher saying what matters, so the learner's own drive to compress its experience is treated as the common root of curiosity and artistic creation.
The paper turns to a concrete example: a drawing of a female face that some human observers consider beautiful. The authors argue that the face's essential features follow "a very simple geometrical pattern", one that can be specified with very few bits of information. Put another way, the stream of data an observer collects while looking at the picture, perhaps through a sequence of eye movements, is more compressible than it would be if no such regularities existed. Compressible means it can be described more briefly, and the regularity is what makes that possible. The paper admits that few people can see how the drawing was made without the grid overlay that explains its construction. Most viewers, however, notice that the facial features somehow fit together and show some kind of order. Under the paper's postulate, "the observer compressibility" is the key: "the observer compressibility" is the source of the reward. Stated in full, the paper says the observer s reward "is generated by the conscious or subconscious discovery of this compressibility." The face stays interesting only as long as looking at it keeps revealing previously unknown regularities. Once there is nothing new left to find, it becomes boring, even to those who still regard it as beautiful. That is why the paper insists that beauty and interestingness are "two different things."
The paper follows its face example with a second drawing, a butterfly and a vase holding a flower. The authors argue that this drawing also needs very few bits of information to describe, because it can be built by a very simple procedure based on fractal patterns made of circles. People who understand that procedure tend to appreciate the drawing more than people who do not, because they see how simple it is. The paper stresses that this recognition is not an immediate, all-or-nothing process. Since the human visual system has a lot of experience with circles, most people quickly notice that the curves somehow fit together in a regular way. Few, however, can immediately state the precise geometric principles underlying the drawing. The authors say this pattern can still be learned from the illustration of the construction. They then tie the example back to their central idea. The discovery process that moves from a longer description of the data to a shorter one, from less compression to more, or from less to more subjectively perceived beauty, yields reward. That reward, the paper says, depends on the first derivative of subjective beauty, that is,
the steepness of the learning curve.
The authors gather their argument into three ingredients. The first is "a continually improving predictor or compressor of the continually growing data history", meaning a system that gets better at describing its own experience in a shorter form. The second is a computable measure of how much that compressor is improving, which produces the intrinsic rewards. The third is a reward optimizer, or reinforcement learner, that turns those rewards into sequences of actions expected to collect the most reward in future. In the authors' words, the whole idea rests on "a surprisingly simple algorithmic principle based on the notions of data compression and data compression progress", which they argue informally accounts for attention, novelty, surprise, interestingness, curiosity, creativity, subjective beauty, jokes, and science and art in general.
Looking ahead, the authors describe three ways to improve their earlier versions of these pieces. First, they want better adaptive compressors, pointing to recent, novel recurrent neural networks, which are learning systems that carry memory across a sequence, along with other general but practically feasible methods for making predictions. Second, they want to find out under which conditions learning progress can be measured both accurately and efficiently, without repeatedly and expensively re-evaluating the compressor on the entire history so far. Third, they plan to study recent improved reinforcement learning techniques, including policy gradient methods, which adjust a system's action choices directly according to reward, and artificial evolution, which searches for good solutions through variation and selection.
The authors also see psychology as a way to test the theory. They say it should be easy to devise controlled experiments in which people must anticipate initially unknown but causally connected sequences of events that contain learnable patterns, and then rate their own intrinsic rewards as their predictions improve. They pose two questions for such studies:
Is the reward indeed strongest when the predictions are improving most rapidly?
and
Does the intrinsic reward indeed vanish as the predictions become perfect or do not improve any more?
Finally, they turn to neuroscience, noting that
Currently we hardly understand the human neural machinery.
They point out that certain neurons seem to predict others, and that brain scans show certain areas lighting up in response to reward. They suggest pairing the psychological experiments with neurophysiological studies to locate where intrinsic rewards originate, possibly by linking them to improvements in neural predictors. They end by saying that
Success in this endeavor would provide additional motivation to implement our principle on robots.
The appendix opens with the idea that frames everything after it:
The world can be explained to a degree by compressing it.
The authors add that "Discoveries correspond to large data compression improvements," found by an improvement algorithm that depends on the application. In other words, a discovery is a moment when the agent's own description of its experience becomes markedly shorter. The appendix says it draws in part on the author's earlier work.
The question it then poses is this:
How to build an adaptive agent that not only tries to achieve externally given rewards but also to discover, in an unsupervised and experiment-based fashion, explainable and compressible data?
Unsupervised here means that nobody labels the right answers. The agent learns by trying things and seeing what happens. The authors also suggest that explanations gained through exploratory behavior "may eventually help to solve teacher-given tasks," so curiosity is treated as a way of getting better at ordinary goals too.
To make this precise, the appendix describes a learning agent that lives one life, split into discrete time steps numbered from 1 to T. The total length of that life may or may not be known in advance. At each step the agent receives a real-valued input from its environment and produces a real-valued action, and that action can affect what it sees later. The agent's goal at any moment is to maximize what the text calls "future success or utility," which is the expected sum of future rewards, given everything it has experienced so far. The expectation is taken over an unknown probability distribution drawn from a set of possibilities. The authors note that this set could include all computable distributions, meaning any environment that a computer program could simulate.
The framework is deliberately general. There is just one life, so no repeatable trials are needed. The environment is not assumed to be Markovian, meaning the next situation need not depend only on the current one. The utility also accounts for the agent's expected remaining lifespan, so an agent can in principle lengthen its own life through its actions.
The authors point to recent work on learning machines that are universal and optimal in several very general senses. Such machines could in principle work out for themselves whether curiosity and building a model of the world pay off in a given environment. This appendix does not leave that to chance. It assumes from the start that compressing and explaining one's history is good and should be done. It sets aside the worry that curiosity could be harmful and "kill the cat."
The reward signal is then split into two parts, combined by a function g, which could be as simple as addition. One part is the traditional external reward from the environment, such as a penalty for bumping into a wall or a payoff for reaching a goal set by a teacher. The other part, which the appendix says it is especially interested in, is the internal, intrinsic, or curiosity reward. This is given whenever the agent's data compressor or internal world model improves.
The basic principle is one the authors have published before in several variants. They state it this way: "Generate curiosity reward for the controller in response to improvements of the predictor or history compressor." The idea is to separate the goal from the means. The goal is to explain, or compress, the history of what the agent has observed. The authors say they "conceptually separate the goal ... from the means of achieving the goal." The means is a reinforcement learning controller, a learning system that works out from rewards which actions to take. Once the goal is written as an algorithm that computes curiosity rewards, the controller is left to find which sequences of actions let the compression algorithm discover and exploit previously unknown kinds of regularity. For now, the paper focuses on the case where no external reward is given at all.
The paper then explains why predicting and compressing are closely linked. A predictor that correctly guesses many events from their history can be used to store that history compactly. If it gets most events right, one only needs to record the events it got wrong, together with the time steps at which they happened, to rebuild the full history. A predictor that estimates the probability of each possible next event can do something similar, encoding likely events with few bits and unlikely ones with many. The authors describe the predictor as the essential part of a short program that regenerates the history. If that program is short compared with the raw data, the history is regular rather than random, and the authors write that such regularity is "presumably reflecting essential environmental laws." A program of this kind may also help predict events that have not yet occurred.
The paper also compares two approaches to this kind of prediction. The Minimum Description Length principle, which favors the shortest description of the data, works from the compressor side. The authors note that this approach "does not necessarily converge to the correct predictions as quickly as" Solomonoff's universal inductive inference, a theoretical method that weighs possible explanations of data by their simplicity. Both approaches still converge to correct predictions in the limit under general conditions.
On the question of which predictor or history compressor to use, the paper says the cost of evaluating a compressor depends both on the compressor itself and on the measure used to score its performance. The simplest option is a linear mapping that predicts the next observation from the current one and one other signal. More complex options are adaptive recurrent neural networks, which use nonlinear mappings and can draw on the entire history. The authors note that the first work on artificial curiosity focused on online learning with recurrent networks of this type. A theoretically optimal predictor would be Solomonoff's universal induction scheme, which the paper mentions as the ideal reference point.
The paper next spells out how a compressor's quality is scored. A compressor is a program that describes the agent's history of observations more briefly, and the paper's first measure simply counts the length of that program in bits. The authors write that the shorter the program, "the more algorithmic regularity and compressibility and predictability and lawfulness in the observations so far." The best possible value of this measure would be a variant of Kolmogorov complexity, the length of the shortest program that can produce the observed history. Kolmogorov complexity is a theoretical ideal that cannot be computed in practice, so it serves as a benchmark rather than a tool.
A second measure adds running time to program length. Here the paper counts the logarithm of the time the program needs to compute the history, so that saving one bit of description counts for as much as cutting the runtime by a factor of two. The paper says this is one of the best ways to trade off storage against computation time, based on earlier work on optimal universal search.
The paper then makes its central point. A compressor's score alone is not what drives curiosity. What matters is improvement. The authors write that
The important thing are the improvements of the compressor, not its compression performance per se.
The reward at each step is therefore a function of two scores: the old compressor and the improved compressor, both tested on the same history. The simplest choice is to subtract the new score from the old one, which the paper describes as a discrete-time version of maximizing the first derivative of how compressible the data is.
The framework runs in two loops at once. The controller chooses actions, observes the result, and updates its behavior with a reinforcement learning method, which is a way of learning from reward signals over time. Separately, a compressor loop repeatedly takes the current program, measures it on the history so far, tries to find a better program with some learning algorithm such as a neural network predictor, measures the new one, and emits a curiosity reward when the score has improved. Because measuring and improving can take many time steps, and may even happen during simulated sleep, the reward can arrive long after the action that earned it. The paper notes that this delay makes credit assignment harder for the controller, but says that reinforcement learning methods exist that are theoretically optimal in several senses, which the paper goes on to discuss.
The paper turns to what an agent should do when no external rewards are available and curiosity is the only drive. It first defines optimal pure curiosity behavior relative to the computational limits of the compressor. At each moment, the agent picks the action that maximizes the expected sum of its intrinsic rewards over a span of future time steps, given everything it has seen so far. The paper notes a catch: the true distribution that generates the world is unknown, so the reinforcement learning task of working out the best actions "may be a formidable one." As the agent revisits parts of its environment that it could not compress before, those parts become more compressible, so the curiosity rewards they yield shrink over time. A good learning algorithm therefore has to detect this decline and predict it. The paper observes that traditional reinforcement learning methods "do not provide any theoretical guarantee of optimality for such situations." It adds that this does not mean suboptimal methods cannot succeed in practice, and that experimental studies might still yield useful insights.
To make the comparison precise, the paper makes a natural assumption: the compressor is not super-complex in the way Kolmogorov complexity is, a measure that cannot be computed at all. Its outputs and its intrinsic rewards can therefore be computed for every time step. Under that assumption, the question is whether a best possible reinforcement learning algorithm exists, one that comes as close as any other to maximizing the objective. The paper answers yes, with a caveat: the algorithm "is not computable in finite time." It is therefore a theoretical reference point, a yardstick for what could at best be achieved, rather than a method anyone can run.
The paper then describes such an optimal action selector. It draws on Solomonoff's universal predictors, which are idealized forecasting machines built on Bayesian learning, the method of updating beliefs as evidence arrives. The only assumption needed is that the environment's reactions are sampled from some unknown probability distribution within the set of all distributions that a computer program could describe, which the paper calls enumerable. Given the observations so far, the method uses Bayes' formula to predict the probability of the next observation. Because the true distribution is unknown, the method predicts with a mixture: a weighted sum of all candidate distributions, with positive weights that add up to at most one. The paper says that in a very general sense,
This is indeed the best one can possibly do.
The drawback is that the set of candidates is infinite, so the scheme cannot be computed.
The paper points to two ways of changing this. The candidate set could be enlarged with certain non-enumerable distributions that can still be approximated in a limited sense. Or it could be restricted so that it becomes computable, and the second route takes the form of assuming the world is computed by some unknown but deterministic computer program.
Building on a predictor derived from the Speed Prior, which the paper describes as one that "assigns low probability to environments that are hard to compute by any method," the authors move from prediction to action. Once an optimal predictor exists, they argue it can be extended by formally including the effects of the actions an agent takes. The result is an optimal action selector that maximizes expected future reward.
The reference point is Marcus Hutter's AIXI, which the paper calls a "theoretically optimal (yet uncomputable)" reinforcement learning algorithm. Reinforcement learning means an agent learning by trial and reward from its environment. At each moment, AIXI uses an extended version of Solomonoff's prediction scheme, a method for predicting the next observation from all possible explanations of the data so far. It uses this to pick the sequence of actions that promises the highest reward up to a fixed horizon. It then carries out only the first action of that sequence and repeats the process at the next step.
The paper explains why AIXI counts as optimal in the sense that matters. The policy built on its mixture of possible environments is self-optimizing. Its average utility converges over time to the value that a policy knowing the true environment in advance would achieve, for every environment in the class considered. The paper states that this holds exactly when the class admits self-optimizing policies. The policy is also Pareto-optimal, meaning that there is "no other policy yielding higher or equal value in all environments" and "a strictly higher value in at least one".
The catch is cost. AIXI "needs unlimited computation time." Its computable variant, AIXI(t,l), has asymptotically optimal runtime but, in the paper's words, "may suffer from a huge constant slowdown." To account for computation time in a general and optimal way, the paper turns to Gödel machines. It describes them as "the first class of mathematically rigorous, fully self-referential, self-improving, general, optimally efficient problem solvers." They are also applicable to the same utility objective the paper has just defined.
A Gödel machine starts with a software store the paper calls S. S holds an initial problem solver, which may be a typically sub-optimal method. It also holds an asymptotically optimal initial proof searcher, based on an online variant of Levin's Universal Search, a general way of searching candidate programs efficiently. This searcher runs and tests proof techniques, which are programs written in a universal language. In principle, these techniques can compute proofs about the machine's own future performance, relative to an axiomatic system encoded in S. That system describes the formal utility function, the hardware, the axioms of arithmetic and probability theory, data manipulation, and S itself. The paper says that including S is possible without introducing circularity.
The self-improvement mechanism takes its inspiration from Kurt Gödel's self-referential formulas of 1931. The machine rewrites any part of its own code, including the proof searcher, once the search has found a proof that the rewrite is useful under the objective. The paper cites a Global Optimality Theorem, under which such a self-rewrite is "globally optimal." The paper stresses that no local maxima are possible, because the self-referential code first had to prove that it is not useful to keep searching for alternative self-rewrites. The paper puts the consequence this way:
If there is no provably useful optimal way of rewriting S at all, then humans will not find one either. But if there is one, then S itself can find and exploit it.
The paper sets this against earlier approaches built on hardwired proof searchers, which it describes as non-self-referential.
The appendix closes its discussion of self-improving machines by noting that such machines not only have an optimal order of complexity but can, through their own changes, optimally reduce slowdowns hidden by the big-O notation, provided the usefulness of those speed-ups can be proved.
The appendix then turns to reinforcement learning, or RL, the family of methods in which an agent learns to act by trying actions and receiving reward. It says that "Recently there has been substantial progress in RL algorithms that are not quite as universal as those above, but nevertheless capable of learning very general, program-like behavior." These methods are less general than the provably optimal ones just described, but still broad in what they can learn.
One group of these methods is evolutionary. The appendix explains that evolutionary search can train recurrent neural networks, which it calls general computers, and that many ways of evolving such networks have been proposed. One particularly effective approach uses cooperative coevolution. Instead of evolving whole networks, it searches over the components of a network, such as individual neurons or synapses, combines them into networks, and selects for reproduction those components that took part in the best performing networks.
Other recent RL techniques, the appendix notes, are based on policy gradients, which adjust an agent's behavior rule directly in the direction that raises its expected reward. The appendix closes this point by stating that
It will be of interest to evaluate variants of such control learning algorithms within the curiosity reward framework.
The paper's central claim is that the pull of interesting material comes from learning progress, and that art, science and play are by-products of that drive. Its own stated limits and open questions are several. It says the framework "essentially specifies the objectives of a curious or creative system" but not the way of reaching them, which depends on the choice of compressor and learning algorithm. It sets aside the worry that curiosity could be harmful, saying it could "kill the cat", and it does not settle that question. The optimal action selectors it describes cannot be computed, so practical systems would have to approximate them. For psychology, the paper leaves two questions open: is the reward strongest when predictions are improving most rapidly, and does the intrinsic reward vanish as predictions become perfect.

