Impact Analysis of Order of Presentation on Champion Book Selection in Bibliobattle

There are numerous approaches to create communication games and revitalize it by using a voting process to select winners and losers. Bibliobattle is a good example of such a communication game. Bibliobattle is a popular social book-review game, especially in Japan, which encourages sharing of interesting books [1].

The important element of Bibliobattle is deciding a Champion Book by voting [2]. McGonigal defined four elements of the game, i.e., goals, rules, feedback system, and voluntary participation [3]. The goal of Bibliobattle is to be chosen from numerous participants and to win a Champion Book award. This mechanism encourages people to present books that participants are interested in at the Bibliobattle. Therefore, the process of deciding the Champion Book must be fair and appropriate.

However, the response order effect has been recognized in cognitive science [4]. The response order effect is the effect wherein the response result changes corresponding to the difference in the order of presentation of a subject to be measured in the experiment or investigation. There are two types of effects: primacy effects (in which response choices presented early were most likely to be selected) and recency effects (in which response choices presented last were more likely to be selected). In Bibliobattle, each presentation is delivered individually. Consequently, the response order effect might have a certain influence on the decision.

However, no studies have provided any evidence on the response order effect in Bibliobattle. Therefore, in this study we investigated the response order effect in Bibliobattle by gathering information about results of approximately 800 Bibliobattle games and presented a statistical analysis of the response order effect in the selection process of Bibliobattle.

Bibliobattle at a university

Full size image

2.1 Bibliobattle

The Bibliobattle is a communication field for knowledge sharing through book reviews. The Bibliobattle game was initially proposed as a social interaction field design facilitated by book reviews [1]. Subsequently, this game became popular with not only related researchers but also with people in various fields. Figure 1 shows Bibliobattle conducted in a class at a university.

According to a survey conducted by the Committee of Bibliobattle, this game has already been organized in 247 universities and 284 public libraries in Japan. Bibliobattle has become popular all over the world including American bookstores, Korean television programs, reading education programs in the Solomon Islands^{Footnote 1}.

The Committee of Bibliobattle has defined the official rules for Bibliobattle^{Footnote 2}. Figure 2 illustrates the official rules.

Bibliobattle: official rules

Full size image

Empirical research on Bibliobattle has been conducted on areas such as the influence of the Bibliobattle field on introduction, the influence on the setting of the time limit, and the evaluation when considering it as a recommendation system [1, 5, 6].

2.2 Response Order Effect

The response order effect is an effect that changes the measurement/response result marginally or significantly depending on the difference in the order of presenting the subject to be measured in the experiment or investigation [7]. It is primarily known as knowledge in the field of research and cognitive science [8]. Here, we will mainly describe the recency effect and the primary effect which are assumed to have influenced the voting of Bibliobattle.

First, there are two types of option lists. The first list is the case where individual choices constitute an independent meaning. They can arbitrarily rearrange the options. The second is a choice list whose order is meaningful. For example, “1. I like it 2. I like it to a certain extent 3. I dislike it to a certain extent 4. I dislike it.” The order in which the choices are presented is either in descending or ascending. The selection of books available for voting in the Bibliobattle is an example of the first list and the response order effect that will be described subsequently also occurs mainly when using the first list.

The case where an item is selected because it was placed at the beginning of the list is called the primary effect, and the case where an item is selected because it was presented at the end of the list is referred to as a recency effect. The primary effects are likely to occur in the case of surveys that utilize visual media such as postal mail surveillance reporting and interview surveys using option cards that are referenced by the respondents. Further, in the case of an interview survey without selection cards or a survey based only on auditory evaluations such as telephone surveys, the recency effect tends to occur.

The process of occurrence of such a tendency is explained primarily by two factors, memory limitation and cognitive elaboration [9, 10]. The factor of memory limitation is related to the cases where a long choice list is used, especially in an auditory investigation. For respondents, choices presented from the beginning of the list to the middle stage may be difficult to remember when they are finally queried for answers. Therefore, the options presented at the end are easier to choose. This is one of the causes of the recency effect.

However, even when short and concise choice lists are used in an auditory survey, the nearest effect occurs. This effect cannot be explained by only the factor of memory limitation. Further, in a visual investigation, a primary effect is demonstrated. This also cannot be explained by the factor of memory limitation. This is because as the options are constantly being presented in a visual investigation, the respondents are not required to memorize a considerable quantity of data.

The second factor is cognitive elaboration. When choices with equal probability are compared with each other, it becomes easier to select the option examined in detail by the respondents. In the visual survey, we assume that the respondents will consider a choice based on the order of presentation. Then, for the choices presented later, the analysis is more likely to be influenced by the information from the options presented earlier. Therefore, if we compare the options exhibiting the same degree of probability, the option presented earlier becomes easier to choose. Conversely, in the auditory survey, the examination time for each option is limited by the pace at which the investigator reads. Moreover, the investigator usually takes time to wait for a response only after reading all the options. Therefore, the review time of the last option tends to be longer than other options. Owing to this and the memory limitation factors mentioned above, it is observed that the auditory survey is likely to be influenced by the recency effect.

2.3 Communication-Field Mechanism Design

The communication-field mechanism design is a new approach to technically understand the communication field between humans. Bibliobattle is one good example of a communication-field mechanism. The definition of communication-field is presented as follows; as a result of participants acting to maximize their utility, building a mechanism to fulfill the purpose by making fruitful communication.

Taniguchi classifies the approach of revitalizing community and organizational communication and creating intellectual creation into three categories: space design, facilitation, and communication-field mechanism design.

The design of space is an approach to create a space to communicate, such as creating a resting room inside a company or setting up an in-house social network service. However, if we just create a space, we cannot induce the behavior and communication of the people participating in it; therefore, sometimes such spaces will be quiet.

Facilitation focuses on the involvement and remarks of participants in the communication field and the method to control the whole communication field. The facilitator does not participate in the discussion and only supports activities from a neutral position. However, this approach is based on the existence of a facilitator, and the quality of the conference depends significantly on the ability of the facilitator.

The mechanism of the communication-field refers to the design of the institution that controls the field. Specifically, in many cases, it refers to rules when communication is a segment of the games. Communication-field mechanism design tries to promote intellectual activities by creating a liberal environment for communication. Even though the facilitator is not skillful, the designed mechanism is aimed to be available to everyone. In Bibliobattle, for example, a moderator does not need special skills. This contributes significantly to the popularity of Bibliobattle.

We gathered data on Bibliobattle under the following conditions from the Internet.

1. 1.

   We obtained information about Bibliobattle from Google Alert, Twitter, and the Committee of Bibliobattle group page on Facebook. We searched for the term “Bibliobattle” in Japanese and English on these websites.

2. 2.

   We gathered data that included the number of presenters and the order of presentations in Bibliobattle from January 2011 to September 2016.

3. 3.

   We excluded the data when there were multiple Champion Books or the number of presenters was less than or equal to two.

4. 4.

   This survey was conducted around October 2016.

There were 797 data elements satisfying these conditions. Table 1 lists the number of acquisitions of Champion Book award per list containing the presentation order.

This survey is significant importance. Generally, it is challenging to conduct experiments for examining the mechanism design of communication in various studies because they require a lot of time and labor. However, Bibliobattle is a communication field that is already popular throughout Japan. Many people in various places use the same rules to communicate. As these games are not conducted as experiments, the specific situation in every game varies. However, it is considerably useful to collect these data to analyze the tendency of a communication mechanism. This is because significant time and labor required for the data collection can be reduced. We believe that this research method also provides knowledge for the research on other designs of communication mechanism having similar properties.

4.1 Test Methods

We used the data of Bibliobattle with 4, 5, and 6 presenters because they contained sufficient number of data (664 data) for analysis. We conducted a chi-square test (significance level: 5%) considering each of the following conditions.

1. 1.

   The Champion Book acquisitions by the first presenter and the presenter other than the first presenter.

2. 2.

   The Champion Book acquisitions by the final presenter and the presenter other than the final presenter.

3. 3.

   The Champion Book acquisitions by the first and second presenters and the presenters other than the first and second presenters.

4. 4.

   The Champion Book acquisitions by the last and last but one presenters and the presenters other than the last and last but one presenters.

4.2 Test Results

Table 2 lists the test results.

Considering conditions 1 and 2, there was no significant difference in the number of Champion Book acquisitions (p = 0.109, 0.231).

Under condition 3, the Champion Book acquisitions by the first and second presenters were significantly less than those of presenters other than the first and second presenters (\(\text {p}= 4.94\times {10^{-3}}\)).

Under conditions 4, the Champion Book acquisitions by the second and last presenters from the end were significantly more than those of presenters other than the last and last but one presenters (\(\text {p}=3.63\times {10^{-4}}\)).

5.1 Interpretation of Results

The above results suggest the following while considering the Bibliobattle with 4, 5, and 6 presenters:

1. 1.

   The possibility of acquiring a Champion Book for the first and last presenters is neither advantageous nor disadvantageous compared to other orders.

2. 2.

   The possibility of acquiring a Champion Book for the first and second presenters has a disadvantageous tendency compared to other orders.

3. 3.

   The possibility of acquiring a Champion Book for the last and last but one presenters is an advantageous tendency compared to the other orders.

These trends indicate that the voting in Bibliobattle is similar to an auditory investigation with a long choice list, indicating that there is a possibility that the recency effect may have occurred. However, in Bibliobattle, unlike auditory investigation, the options (book titles) are often presented after publication, or participants present it in memos. Therefore, effect of the memory limitation factor is weakened, and it is considered that the option that was presented at the end was not necessarily advantageous as compared to other presentation orders. Further, as the options are presented visually, the time to review the books presented at the beginning is longer. Therefore, we consider that the effect of the cognitive elaboration mentioned earlier has a primacy effect, and the result is that the first introduced book is not necessarily in a disadvantageous position when compared to the other presentation orders.

Based on this interpretation, we considered two points. Firstly, we only revealed the trends mentioned above, and the cause of these trends is not necessarily the response order effect. This research collected and analyzed the results of Bibliobattle simultaneously performed in various parts of Japan. Detailed conditions such as presence/absence of ice-breaker sessions, voting system, method for deciding the order, were not uniform across the venues where Bibliobattle was conducted. To understand the influence of the response order effect on the possibility of acquiring Champion Books according to the presentation order in Bibliobattle, it is necessary to conduct experiments uniformly across all venues.

Secondly, the players accustomed to Bibliobattle, generally choose the order of presentation that is closest to the end. This implies a trend of the last and last but one in the order being advantageous. Thus, it is impossible to exclude the possibility that presenters at Bibliobattle feel that it is advantageous to be closer to the end of the order of presentation and expect that the second half will be likely to be chosen. However, Bibliobattle is a game in which victory/defeat significantly depends on the “participant’s preference of books”, which implies that the factors cannot be predicted by players. Thus, having a lot of experience as a presenter does not imply that it is easy to acquire Champion Books. Therefore, even if experienced players tend to choose the latter half of the order for presentation, we think that the influence is small.

5.2 Consideration on Methods to Reduce the Influence of Response Order Effect

Although its strength is unknown, it is considered that the response order effect influences the decision of awarding the Champion Book of Bibliobattle. Here, we consider two ways to weaken its effect in Bibliobattle.

The first suggestion is to decide presenting options after placing the books in a position that can be seen by all the participants. Alternately, the titles of the book can be displayed in a visible position. Consequently, the influence of the recency effect caused by the memory limitation factor can be reduced. However, as this method has already been adopted empirically by many organizers, it can be observed that this method does not exert a big influence on the tendency discussed in this paper.

Secondly, we considered changing the procedure of voting. According to the official rule of Bibliobattle, the procedure of voting is decided by the organizer. Therefore, there are various voting methods, such as raising hands, pointing fingers, and ballot sheets. However, voting by hands is frequently used in many Bibliobattle. In this method, the chief speaker reads out the title name in the order of presentation, and the participants raise their hands to vote for the book that they want to read the most. We considered the possibility of reducing the response order effect of the presentation order by voting in the reverse order of presentations. To verify the second method, we conducted experiments on Bibliobattle with 4, 5, and 6 presenters. The number of games conducted was 26, and the total number of presenters was 119. Table 3 lists the number of acquisitions of Champion Books per presentation order when voting was conducted in the reverse order.

As with the previous test, we performed a chi-square test (significance level 5%) considering the following four conditions.

1. 1.

   The Champion Book acquisitions by the first presenter and the presenter other than the first presenter.

2. 2.

   The Champion Book acquisitions by the final presenter and the presenter other than the final presenter.

3. 3.

   The Champion Book acquisitions by the first and second presenters and the presenters other than the first and second presenters.

4. 4.

   The Champion Book acquisitions by the last and second-last presenters and the presenters other than the last and second-last presenters.

Table 4 shows the results of the test.

In this experiment, there was no significant difference in any of the conditions. Therefore, it is possible to reduce the influence of the recency effect by voting in the reverse order of the presentations at the time of voting. However, as there is a possibility that there is no significant difference owing to the small sample size, we will maintain this as reference material.

In this paper, we discussed the possibility that the response order effect influences the Bibliobattle and analyzed methods to reduce this influence. The response order effect is also a challenge for other communication mechanisms where winning or losing happens by voting. The impact analysis of the order of presentation on Champion Book selection in a Bibliobattle was considered beneficial for the study of other mechanisms.

In addition, in this paper, we investigated the characteristics by gathering information on widely popular mechanisms. This is an era where anyone can transmit information on the Internet, and we think that there is a high possibility that research on the communication mechanism, which was impossible owing to time and manual problems earlier, can now be conducted more easily.