^{1}

^{*}

^{2}

^{3}

^{4}

^{5}

^{1}

^{2}

^{3}

^{4}

^{5}

Edited by: Víctor M. Eguíluz, Instituto de Física Interdisciplinar y Sistemas Complejos (IFISC), Spain

Reviewed by: Kunal Bhattacharya, Aalto University, Finland; Matjaž Perc, University of Maribor, Slovenia

*Correspondence: Satoshi Uchida

This article was submitted to Interdisciplinary Physics, a section of the journal Frontiers in Physics

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Indirect reciprocity is one of the basic mechanisms to sustain mutual cooperation, by which beneficial acts are returned, not by the recipient, but by third parties. This mechanism relies on the ability of individuals to know the past actions of others, and to assess those actions. There are many different systems of assessing others, which can be interpreted as rudimentary social norms (i.e., views on what is “good” or “bad”). In this paper, impacts of different adaptive architectures, i.e., ways for individuals to adapt to environments, on indirect reciprocity are investigated. We examine two representative architectures: one based on replicator dynamics and the other on genetic algorithm. Different from the replicator dynamics, the genetic algorithm requires describing the mixture of all possible norms in the norm space under consideration. Therefore, we also propose an analytic method to study norm ecosystems in which all possible second order social norms potentially exist and compete. The analysis reveals that the different adaptive architectures show different paths to the evolution of cooperation. Especially we find that so called Stern-Judging, one of the best studied norms in the literature, exhibits distinct behaviors in both architectures. On one hand, in the replicator dynamics, Stern-Judging remains alive and gets a majority steadily when the population reaches a cooperative state. On the other hand, in the genetic algorithm, it gets a majority only temporarily and becomes extinct in the end.

Cooperative relationships such as I-help-you-because-you-help-me relations can often be found in both biological systems and human societies. Cooperative behaviors are obviously essential to make societies effective and smooth. However, evolutionary biologists and social scientists have long been puzzled about the origin of cooperation. Recently, scientists from a variety of fields such as economics, mathematics and physics have been tackling the puzzle using tools developed in each discipline.

According to a thorough review published from statistical physics viewpoints recently [

Following the context of the physics literature, in this paper, we deal with interactions of “social norms.” Social norms are interpreted as views on what is “good” or “bad” and play an essential role in indirect reciprocity based on reputation systems. Indirect reciprocity is known as one of the main mechanisms for the emergence of cooperation. It has a long history and has been amply documented in human populations [

As mentioned in Nowak and Sigmund [

Vast studies on indirect reciprocity in the framework of evolutionary game theory have discovered various types of norms or assessment rules that enhance the evolution of cooperation in the modern society with highly mobile interactions. Theoretically, assuming that the same norm is adopted by all members of a population, Ohtsuki and Iwasa have shown that only eight out of 4,096 resulting possible norms lead to a stable regime of mutual cooperation. These are said to be the “leading eight” [

Many theoretical studies also considered another stability criterion. Those studies focus on whether the corresponding population cannot be invaded by or can invade into unconditional strategies such as perfect cooperators and perfect defectors [

If one wants to analyze the evolution of even the simplest system of morals, one has to consider the interaction of several assessment rules in a population. Some studies meet the theme. For example, comparing Simple-Standing with Stern-Judging, both members of the leading eight, is an important task to explore a champion of the assessment rules using second-order information. Uchida and Sigmund [

Despite the theoretical developments of Uchida and Sigmund [

Therefore the main focus of the present paper is in developing a systematic analytical methodology with which entanglements of all sixteen norms using second-order information can be formulated in an equation system. Extending the methodology proposed by Uchida and Sigmund [

The authors' development is useful not only for rigorous analysis of norm ecosystems, but also helps compare different “adaptive architectures.” Here an adaptive architecture means a way for individuals to adapt to their environments. In this paper, we take up the two representative architectures, replicator dynamics and genetic algorithm. Although these architectures are popular in the literature, they are studied independently in different domains and their comparison in the framework of evolutionary game theory has not yet been done because there has been no technical method developed to capture all strategies in a norm space at once as the study of genetic algorithm requires. Our approach offers a first opportunity to theoretically analyze a comparison of replicator dynamics and genetic algorithm in evolutionary game theory.

The analysis reveals that the two representative adaptive architectures show different paths to the evolution of cooperation. We find that Stern-Judging, one of the best studied norms in the literature, plays important but different roles in both cases [

In the next section, we describe the model ecosystem, derive the equation to analyze it and introduce the adaptive architectures. Then we present the results and discuss them.

An infinitely large, well-mixed population of individuals (or players) is considered. From time to time, one potential donor and one potential recipient are chosen at random from the population and they engage in a donation game: the donor decides whether to help the recipient at a personal cost

Individuals in the population have the ability to observe and assess others following their assessment rules (or social norms). Here “assess” means that players label other players “good” or “bad” according to their actions as a donor in their last interactions. The images of players are also denoted by “1” (for “good”) and “0” (for “bad”). The assessment is done privately but the information needed for the assessment is so easily accessed that all individuals have the same information (on private information see for example [

A donor determines whether or not to help the recipient, depending on the current image of the donor (i.e., whether the recipient is labeled as 1 or 0). If the recipient is viewed as 1 in the eyes of the potential donor, the recipient will be given help, otherwise the recipient will not be offered a help. Note that we do not assume any kind of error in the model because this is a first attempt to describe competitions of all norms in the focal norm space (for the role of errors, see [

The social norms in this present research are at most of second order, i.e., they take the image of the recipient as well as the action of the donor into consideration. Denoting the action of the donor by α ∈ {0, 1} and the image of the recipient by β ∈ {0, 1}, the new image of the donor after the game from the view point of some norm is a binary function of α and β: β^{new} = ^{3} + ^{2} + ^{1} + ^{0} + 1. The 16 norms include some well-studied norms in the literature: the 9th norm (1000) is known as Shunning (SH), the 10th norm (1001) is called Stern-Judging (SJ), the 13th norm (1100) Image-Scoring (IS) (which is of first order) and the 14th norm (1101) Simple-Standing (SS). The first norm (0000) and the last one (1111) are unconditional norms and called AD and AC, respectively.

We denote by _{i} the frequency of individuals that follow social norm

As individuals play the game, the images of the individuals gradually change. At the equilibrium of images, the average payoff of individuals with norm _{i} at the equilibrium of images is in fact given by

where _{ij} is the probability that a random player with norm _{j} fixed. The outline of the calculation for the image matrix is shown in the Results section (The full information on the calculation is found in the

The players adaptively switch their assessment rules, aiming at more payoffs. We examine two different switching processes: adaptive changes due to social learning by imitation described by the replicator dynamics and those changes of norms modeled by the genetic algorithm.

In case of replicator dynamics, an individual occasionally has a chance to change its norm by imitating another individual (i.e., adopting its norm as a model). The probability that an individual (with norm _{i} and that model's fitness _{i} = _{i}. Here, _{i} in simulations.).

With some probability, an individual selects a norm totally at random and adopts that norm. This occurs due to mutation. The resulting dynamics is given by the replicator-mutation equation

In case of genetic algorithm, an individual decomposes a norm into a collection of bits and changes its norm “bit-wise” by imitating the norms of two randomly selected individuals (called parents) [_{i} and the square of the fitness of norm

After parents have been chosen [now, the norms of the parents are

From rules 1 and 2, we can derive the probability that any norm _{i}). However, due to mutation, a bit of the generated norm can be flipped with probability μ. Note that μ in the RD and that in the GA have different meanings. We assume that at most one bit can be inverted because of the small mutation probability. Thus the probability that none of 4 bits is flipped is 1 − 4μ. Therefore the probability that norm _{i} = (1 − 4μ)_{i} + μ

The frequency of norm _{i}(_{i}(_{i}, where

In addition to the ordinary adaptive architectures well-studied in the literature mentioned above, we consider two other adaptive architectures that are modified versions of the conventional replicator dynamics and the genetic algorithm, respectively. The first one is replicator dynamics with multiple models.

In this adaptive architecture, an individual learns each bit of its norm independently from probably different models. The probability that an individual having norm _{1} is the set of norms whose first bit is 1 − _{V} is the normalized fitness of _{ij}) and out-flow from _{ji}). Then the increase rate of norm

As in ordinary replicator-mutation dynamics, we also include a mutation term in addition to the switching process described above. But here, we assume that mutation occurs “bit-wise” as assumed in genetic algorithm. That is, by μ, we denote the probability that each bit is flipped by mutation. Then the in-flow to _{j}.

The resulting dynamics is given by

The other one is genetic algorithm with a single parent. In this architecture, only one individual is chosen as the unique parent of an individual. Then the child copies the norm of the parent. That is, the child adopts the entire norm of the single parent. Mutation effects and the probability that an individual is chosen as a parent are calculated in the same way as in the ordinary genetic algorithm mentioned above.

All four corresponding evolution equations depend on expected payoffs. We assume that images are always at equilibrium at each time step of the evolution equations. Under this assumption in the next section, we derive the equation system to specify image matrices (thus expected payoffs of norms) and show time evolutions of norms based on the above mentioned adaptive architectures.

Images of individuals change in time as well as frequencies of norms. But we assume that the time scale of the changes of images is much faster than that of norm changes. As a result, images are always at equilibrium and norm frequencies are treated as constant in estimating image matrices, as is assumed in the literature (See [

To calculate image matrix _{ij}, we introduce “image profile” _{1} ∈ {0, 1} from the first norm and _{2} from the second norm, …, and _{16} from the 16^{th} norm. Note that, since the first norm is unconditional AD, the probability that _{1} = 1 is zero. Thus

The image profile is a joint probability distribution and contains the finest probabilistic information about the system. For example, the image matrix is interpreted as the marginal distribution:

with ẽ_{i} = 1.

Now we define the joint distribution in the whole population by

which gives the proportion of those individuals in the whole population, who are labeled image _{1} from the viewpoint of the first norm and _{2} from the second norm, _{3} from the third norm and so on. Since _{f1f2…f16} is a probability distribution, there is a constraint on _{f1f2…f16}:

According to our analysis, it is possible to derive the equation system that yields the values of all image profiles _{f1f2…f16}. Then inserting those relations between _{f1f2…f16}into Equations (3) and (4), we can have an inhomogeneous linear equation system for _{f1f2…f16}. Solving this equation system yields the values of _{f1f2…f16}. See the supplementary material for the details of the derivation.

We remark that the equation system with respect to _{f1f2…f16} includes 2^{16} − 1 unknowns in principle, but the fact that the equation system contains some trivial variables such as _{f1f2…f16} = 0 with _{1} = 1 or _{16} = 0 reduces the dimension of the equation system.

Moreover, the case where _{13} = _{(1,1,0,0)} = 1 indicates that action C has been taken. In this case, the following conditions must be satisfied: _{(1,1,0,1)} = _{(1,1,1,0)} = 1 and _{(0,0,1,1)} = _{(0,0,0,1)} = _{(0,0,1,0)} = 0. The situations in which the above conditions are broken never happen. For those situations, _{f1f2…f16} = 0.

Similarly if _{13} = _{(1,1,0,0)} = 0, which implies that action D has been chosen, then _{(0,1,0,0)} = _{(1,0,0,0)} = 0 and _{(0,0,1,1)} = _{(0,1,1,1)} = _{(1,0,1,1)} = 1. Therefore _{f1f2…f16} = 0 for the situations where the above condition is not satisfied.

As a result the dimension of the equation system reduces to 2^{9} − 1, which can computationally be handled.

Note that the solution depends on the frequencies of norms in the population. In Figure _{i} = 1/16. We see that the simulation and the analytic method generate parallel results.

Image matrix _{ij} (_{ij}. The value of _{ij} is shown in gray scale, in which white corresponds to “1,” and black to “0”.

Frequencies of norms in a population change in time, based on its adaptive architecture. The equations describing such changes depend on payoffs. Therefore the calculating image matrices by the above mentioned method makes it possible to investigate the evolution of multiple norms caused by both switching processes, replicator dynamics and genetic algorithm.

In Figure

A typical pattern of time evolution of norms' frequencies and the cooperation rate in the population generated by replicator dynamics

A typical pattern of time evolution of norms' frequencies and the cooperation rate in the population generated by genetic algorithm

As Figures

However the long-term behavior of Stern-Judging differs in both architectures. In replicator dynamics, Stern-Judging gets a majority after defective norms have disappeared and cooperation has been realized. This trend after the transition between non-cooperative states and cooperative states is preserved stably (Figure

Generally, from Figure

But this is not true for replicator dynamics. In replicator dynamics, Stern-Judging (red dashed line) is the best, Simple-Standing (gray dashed line) is the second best and Image Scoring (yellow dashed line) is the third. All these norms are well-known in the literature. Note that in both architectures, Image-Scoring survives in the long run. This is a significant finding since, in literature, Image-Scoring is known as an unstable strategy [

Figure

A typical pattern of time evolution of norms' frequencies and the cooperation rate in the population generated by replicator dynamics with multiple models

A typical pattern of time evolution of norms' frequencies and the cooperation rate in the population generated by genetic algorithm with a single parent

In the last section we found that the norm ecosystems based on different architectures show similarity and dissimilarity. Although the norm ecosystems investigated here are complex systems, their analyses enable us to gain deep understanding of a simple single norm. For instance, an unstable norm, Image-Scoring, evolves and survives in the melting pot of competing norms regardless of architectures individuals are based on. This insight cannot be obtained if we solely analyze the single norm.

The main difference of the two representative architectures (ordinary replicator dynamics and genetic algorithm) appears in the roles of Stern-Judging, whose local stability is well-studied in the literature. The analysis revealed that Stern-Judging wins the competition against other norms and stays alive in ordinary replicator dynamics even after cooperation is achieved. That is, Stern-Judging is not only locally stable but can evolve from a mixture of diverse norms and gets a majority in the end as far as ordinary replicator dynamics is assumed. In this sense, we say that Stern-Judging plays the role of a “leading” norm in the framework of replicator dynamics.

This norm also plays a vital role in genetic algorithm since it gets a majority just before the cooperation rate starts rising. This occurs because Stern-Judging can defeat defective norms such as AD or Shunning and can increase its frequency in defective states. In other words, Stern-Judging kick-starts the evolution toward cooperation. In Yamamoto et al. [

But why do these architectures show such different results? What is the essential difference between the two? In genetic algorithm, individuals divide norms into smaller parts (bits) and learn the parts more or less independently (from its mother and father). So we can call the learning process “analytic.” For individuals with genetic algorithm, the first bit of a norm represents pro-sociality of the norm, the second bit tolerance, the third anti-sociality and the fourth intolerance (i.e., punitive nature) and they imitate each aspect of their parents, respectively.

On the other hand, individuals based on (ordinary) replicator dynamics do not analyze norms into parts but treat norms as a whole. The learning process based on replicator dynamics can therefore be called “synthetic.” And whether or not the adaptive architecture is analytic or synthetic has a large impact on the results.

In fact, we modified genetic algorithm so that an individual learns how to assess others from only one parent (i.e., the norm is not divided into parts), and we obtained similar results as ordinary replicator dynamics. Moreover we extended replicator dynamics so that an individual decomposes norms into four bits and imitates each part of different models. As a result, we found similar results as ordinary genetic algorithm (with two parents). From these results, we can conclude that whether Stern-Judging can survive in a long run in cases where cooperation is achieved does not depend on switching processes (i.e., whether replicator dynamics is assumed or genetic algorithm is used). But it relies on whether norms are treated as a whole or “bit-wise” in the corresponding switching processes.

In spite of the findings mentioned so far, we have to remark that much remains to be studied. The model studied in this present research especially has many limitations, which offers some tasks for future research from physics perspectives. First of all, we omitted implementation errors in the model to simplify the analysis. Whether and how errors change the results is interesting and necessary research yet to be done.

Moreover we assumed well-mixed populations in the analysis and ignored the effects of structured populations and group formations on cooperative behaviors of individuals. Recently interactions between heterogeneity of populations and reciprocal behaviors are investigated from physics viewpoints. For example, Nax et al. [

Another factor that is out of scope in this research is the imperfectness of information. From the players' viewpoint, although the same interaction can be interpreted differently by players with distinct norms, different individuals that share the same norm always have the same opinion since all individuals are based on the same information in the model. In the literature, the imperfectness of information has been studied in several ways [

All authors conceived and designed the project. SU built and analyzed the model and wrote the manuscript. All authors discussed the results, helped draft and revise the manuscript, and approved the submission.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

SU wishes to thank Voltaire Cang for his useful comments.

The Supplementary Material for this article can be found online at: