Why inconsistent results in experiment

Here, I describe ten common mistakes listed in no particular order that occur in laboratory research settings. The ever-present possibility of mistakes in experimental research should always be taken seriously, but should not be discouraging; all of these mistakes are preventable when researchers are properly trained and remain vigilant.

By reducing mistakes we not only create a safer, more productive environment, but we do better science as a result. His research has focused on the study of the role of immunoglobulin D IgD in lymphocyte regulation. This article is also published on the OUP Blog. With a flexible approach to fit around a busy schedule, our interactive, online training programmes support researchers throughout all stages of the research life-cycle, integrating key issues and developments in research practice, building vital knowledge and skills, and encouraging personal reflection and development.

Explore our full range of our research-focused training programmes. We use cookies to enhance your experience on our website.

You can change your cookie settings at any time. Find out more Continue. Mistakes can occur at many levels, and sometimes they turn out to be due to innocent reliance on common or specialized methods — even published protocols — that are less than optimal. All experiments should begin with a well-planned protocol. If the protocol is investigator-initiated, permutations of steps should be built into the protocol to determine the optimal methodological approach.

Practice makes perfect, but you also need to consider the possibility that a published protocol may have failed to provide important caveats e. That can cause experiments to fail or yield inconsistent results. Failing to troubleshoot by at least attempting to get advice from others who have successfully used a method is also a mistake. You can reduce the risk of failed protocols by reaching out the investigator who published the protocol. Row three shows that the reported relationship between juvenile period and brain size [ 40 ] reverses from significant to non-significant when adding more data to predictors by pooling.

Row four shows that the relationship reported by Lindenfors et al. Lastly on row five, the significant relationship between group size and neocortex [ 19 ] had its slope significantly changed when adding more data by pooling and controlling for phylogeny.

Note that when predicting with PGLS here, the model does not account for the phylogenetic position of the observation to be predicted.

All reanalyzes reported here use phylogeny to correct for non-independence. Further, data on group size and body weight was pooled from [ 35 , 67 ]. As is custom in phylogenetic comparative analysis, phylogenetic information is used to estimate the covariance of the residuals [ 80 ]. This process can lead to an R 2 value different from model fit with non-phylogenetic least squares. Collinearity is member of a family of problems with model fitting referred to as weakly-identifiable parameters or sometimes non-identifiable [ 81 ].

If the predictors co-vary a lot, i. Looking at the correlation plot it can be shown that some variables correlate substantially. We still believe that multicollinearity cannot be ruled out because when we calculated posterior distributions for all parameters in a Bayesian framework, for the full model containing all six predictors and plotted the correlation matrix excluding varying intercepts, see supporting information S1 Fig it is obvious that some parameters correlate substantially which would explain the varying results exposed in this study [ 81 — 83 ].

However, the Markov chains sampled poorly and the analyses should not be fully trusted. Our analyses indicate that the field of primate brain evolution is best characterized as an array of contradicting results [ 1 , 2 ] and our results reveal one reason why this is so. Within the PGLS framework, choice of what variables to include, and what observations for those variables to include, fundamentally changes the conclusions as to what drives primate brain evolution.

In this study we conducted analyses on new data [ 5 ] combined with Stephan et al. If so inclined, we could have presented support for any hypothesis of our choice, but also refuted pretty much any study we would have liked. The AIC tests in turn had six explanatory variables to combine.

The predictors and the AIC test itself were chosen according to our best effort to follow the established method within the field of primate brain evolution. In other words, we chose variables that according to the literature are plausible determinants in brain evolution, and used established methods to choose among combinations of predictors.

As has happened throughout research on primate brain evolution, some researchers have concluded that one factor has been the main determinant, whereas other researchers have made different models and concluded something contradictory.

Thus, using p-values to evaluate the importance of hypotheses that affect primate brain size leaves us ambivalent. AIC was developed to select among models and thus to save us from such ambivalence, but AIC can only evaluate the models given to it, which is why the results still are dependent on pre-test variable choice.

However, it is our opinion that this result also should be judged with due caution, for several reasons. When we included new brain data and updated variables on previously reported results, we found the same patterns. As shown in Table 7 , previous results [ 35 ] indicated that brain size was best predicted by diet and not by sociality measured as group size. When we added more observations to the explanatory variables, both diet and sociality turned out to be non-significant.

When we kept their original predictors, but used pooled brain data [ 3 , 5 ], sociality became significant but not diet see supporting information S6 Table. Further, Lindenfors et al. They found male group size to be a significant predictor for all these structures except cerebellum. However, when we did a reanalysis with updated variables we found no significant relationships for any of these predictors.

AIC is a method for choosing the model with the lowest out-of-sample deviance and as such a method concerned with prediction, not p-values. Clearly, as shown in this paper, the best predicting model may include several variables that have non-significant p-values. In the context of AIC, it is easy to illustrate that the most predictive models sometimes do not reveal the true relationship between individual predictors and the outcome, as for example in the case of concomitant variable bias [ 84 ] or collinearity.

Yet inference about individual predictors is mostly what concerns scientists of primate brain evolution, not mere prediction. Our suggestion for future studies is to run Bayesian analysis for all regression parameters.

As we have demonstrated in this study, the probability for many of the slopes given a null hypotheses lie in the region of 0. Even if a Bayesian result will not give us the decisive answer we seek it will definitely provide the distributions of likelihood for each slope and, as we have initiated here, expose multicollinearity by calculating posterior distribution correlations.

Further caveats on the current practices in comparative studies of primate brain evolution have been raised by other researchers, such as problems with measuring and comparing intelligence [ 15 , 85 ], the idea of adaptive specializations of cognitive mechanisms [ 18 , 86 ], validity of observational data versus experiment [ 18 ], choice of brain measure [ 6 , 52 ], measuring and defining sociality [ 70 , 87 , 88 ] and p-hacking: given that the same sample on brain volume [ 3 ] has been modeled against many variables, it is to be expected that Type 1-errors will emerge [ 89 ].

Also, there is some evidence that different data samples are qualitatively different from each other [ 70 , 90 , 91 ]. It has for example been shown that data on body size often are averaged, inaccurate and from unspecified sources [ 90 , 92 ]. This does not leave much to be explained by the competing adaptive hypotheses. The fact that these adaptive hypotheses explain very little variation, taken together with the unstable nature of results, suggest that it is easy to overstate the importance of sociality, diet, problem solving, or life-history for understanding brain evolution.

Further, other measures not included here may be more important for our understanding of brain evolution. Indeed, other combinations of predictors have been used in previous studies, however, we believe that adding more predictors would reveal similar inconsistencies in the results and that the six predictors used in this study suffice to illustrate this.

That variation in sensory and perceptual systems give rise to variation in brain size is not controversial [ 93 , 94 ]. A primate with very large eyes will have brain areas that correspond to sensory and perceptual needs. In addition, animals that are motor flexible, have many different kinds of muscles, and large behavior repertoires need brain areas that control muscles.

Therefore, larger brains are needed to drive more motor flexible bodies [ 95 ]. To put this in Tinbergian terminology: a mechanistic link between brain size and body functions is straight forward and non-controversial, while a functional link between brain size and mental capacities is harder to define to non-controversial precision.

We conclude that, given the instability of results and the PGLS approach, there is no empirical justification to highlight any particular hypothesis of those adaptive hypotheses we have examined here, as the main determinant of primate brain evolution.

We are grateful for input from six anonymous reviewers that greatly improved the manuscript. National Center for Biotechnology Information , U. PLoS One. Published online Jul Denis Horvath, Editor. Author information Article notes Copyright and License information Disclaimer.

Competing Interests: The authors have declared that no competing interests exist. Received Oct 26; Accepted Jun 6. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

This article has been cited by other articles in PMC. Associated Data Supplementary Materials S1 Fig: Correlation matrix of posterior distributions for all predictors used in this study calculated using a bayesian multilevel model. S1 Table: Main data set. S3 Table: Correlation matrix all variable. S4 Table: Partial R2. S5 Table: Partial R2. S6 Table: Reevaluating DeCasien [ 35 ] et al. S7 Table: Data used to reevaluate DeCasien et al.

S8 Table: Reevaluating Joffe [ 40 ] results of a significant relationships between juvenile period and the ratio of non-visual cortex to the rest of the brain. S9 Table: Data used to reevaluate Joffe [ 40 ]. S10 Table: Reevaluating Lindenfors [ 27 ]. S11 Table: Data used to reevaluate Lindenfors [ 27 ]. S13 Table: Data used to reevaluate Dunbar [ 19 ]. Abstract Primate brains differ in size and architecture. Introduction The field of primate brain evolution can be characterized as an array of contradicting results [ 1 , 2 ].

Allometric relationships Brains are similar to other organs in that they scale allometrically with body size. General cognitive abilities Larger relative brain size or brain component size has evolved to meet higher cognitive demands [ 8 , 13 — 18 ]. Sexual selection Demands of sociality are different between males and females.

Life history Variation in juvenile period and life span is hypothesized to affect brain size evolution [ 38 , 39 ]. Method All data used in this study were collected from published literature and are presented in supporting information S1 Table. Results We analyzed the effect of six predictor variables on two outcome variables: total brain size and neocortex size.

Table 1 The following model was selected with AIC for total brain size as the dependent variable. Predictor b se t p Female weight 0. Open in a separate window. Table 2 The following model was selected with AIC for neocortex size as the dependent variable. Table 5 Changes in p-value for each predictor when altering concomitant predictors using total brain as dependent variable.

Focal predictor Min p-value Concomitant predictors Max p-value Concomitant predictors Male group size 0. Table 6 The change in p-value for each predictor when altering concomitant predictors using neocortex size as dependent variable. Table 7 Overview of changes in the relation between brain size and predictors as different data is used.

N is identical to the original studies in all re-analyses. Discussion Our analyses indicate that the field of primate brain evolution is best characterized as an array of contradicting results [ 1 , 2 ] and our results reveal one reason why this is so. Supporting information S1 Fig Correlation matrix of posterior distributions for all predictors used in this study calculated using a bayesian multilevel model.

DOCX Click here for additional data file. S1 Table Main data set. S3 Table Correlation matrix all variable. S4 Table Partial R2. S5 Table Partial R2. S6 Table Reevaluating DeCasien [ 35 ] et al. S7 Table Data used to reevaluate DeCasien et al. S8 Table Reevaluating Joffe [ 40 ] results of a significant relationships between juvenile period and the ratio of non-visual cortex to the rest of the brain.

S9 Table Data used to reevaluate Joffe [ 40 ]. S10 Table Reevaluating Lindenfors [ 27 ]. S11 Table Data used to reevaluate Lindenfors [ 27 ]. S13 Table Data used to reevaluate Dunbar [ 19 ]. Acknowledgments We are grateful for input from six anonymous reviewers that greatly improved the manuscript. Data Availability All relevant data are within the manuscript and its Supporting Information files.

References 1. Re-evaluating the link between brain size and behavioural ecology in primates. Healy SD, Rowe C. A critique of comparative studies of brain size. New and revised data on volumes of brain structures in insectivores and primates.

Folia primatologica. Endocranial volumes of primate species: scaling analyses using a comprehensive and reliable data set. Journal of Human Evolution. Primate brain anatomy: New volumetric MRI measurements for neuroanatomical studies. Brain, behavior and evolution. Linked regularities in the development and evolution of mammalian brains.

Developmental structure in brain evolution. Behavioral and Brain Sciences. Jerison HJ. Evolution of the brain and intelligence. Current Anthropology. Willemet R. Reconsidering the evolution of brain, cognition, and behavior in birds and mammals. Frontiers in psychology. Comparative analyses of evolutionary rates reveal different pathways to encephalization in bats, carnivorans, and primates.

Proceedings of the National Academy of Sciences. Herculano-Houzel S. When we kept their original predictors, but used pooled brain data [ 3 , 5 ], sociality became significant but not diet see supporting information S6 Table. Further, Lindenfors et al. They found male group size to be a significant predictor for all these structures except cerebellum.

However, when we did a reanalysis with updated variables we found no significant relationships for any of these predictors. AIC is a method for choosing the model with the lowest out-of-sample deviance and as such a method concerned with prediction, not p-values.

Clearly, as shown in this paper, the best predicting model may include several variables that have non-significant p-values. In the context of AIC, it is easy to illustrate that the most predictive models sometimes do not reveal the true relationship between individual predictors and the outcome, as for example in the case of concomitant variable bias [ 84 ] or collinearity.

Yet inference about individual predictors is mostly what concerns scientists of primate brain evolution, not mere prediction. Our suggestion for future studies is to run Bayesian analysis for all regression parameters. As we have demonstrated in this study, the probability for many of the slopes given a null hypotheses lie in the region of 0. Even if a Bayesian result will not give us the decisive answer we seek it will definitely provide the distributions of likelihood for each slope and, as we have initiated here, expose multicollinearity by calculating posterior distribution correlations.

Further caveats on the current practices in comparative studies of primate brain evolution have been raised by other researchers, such as problems with measuring and comparing intelligence [ 15 , 85 ], the idea of adaptive specializations of cognitive mechanisms [ 18 , 86 ], validity of observational data versus experiment [ 18 ], choice of brain measure [ 6 , 52 ], measuring and defining sociality [ 70 , 87 , 88 ] and p-hacking: given that the same sample on brain volume [ 3 ] has been modeled against many variables, it is to be expected that Type 1-errors will emerge [ 89 ].

Also, there is some evidence that different data samples are qualitatively different from each other [ 70 , 90 , 91 ]. It has for example been shown that data on body size often are averaged, inaccurate and from unspecified sources [ 90 , 92 ]. This does not leave much to be explained by the competing adaptive hypotheses. The fact that these adaptive hypotheses explain very little variation, taken together with the unstable nature of results, suggest that it is easy to overstate the importance of sociality, diet, problem solving, or life-history for understanding brain evolution.

Further, other measures not included here may be more important for our understanding of brain evolution. Indeed, other combinations of predictors have been used in previous studies, however, we believe that adding more predictors would reveal similar inconsistencies in the results and that the six predictors used in this study suffice to illustrate this.

That variation in sensory and perceptual systems give rise to variation in brain size is not controversial [ 93 , 94 ]. A primate with very large eyes will have brain areas that correspond to sensory and perceptual needs. In addition, animals that are motor flexible, have many different kinds of muscles, and large behavior repertoires need brain areas that control muscles.

Therefore, larger brains are needed to drive more motor flexible bodies [ 95 ]. To put this in Tinbergian terminology: a mechanistic link between brain size and body functions is straight forward and non-controversial, while a functional link between brain size and mental capacities is harder to define to non-controversial precision. We conclude that, given the instability of results and the PGLS approach, there is no empirical justification to highlight any particular hypothesis of those adaptive hypotheses we have examined here, as the main determinant of primate brain evolution.

Browse Subject Areas? Click through the PLOS taxonomy to find articles in your field. Abstract Primate brains differ in size and architecture. Introduction The field of primate brain evolution can be characterized as an array of contradicting results [ 1 , 2 ]. Allometric relationships Brains are similar to other organs in that they scale allometrically with body size.

General cognitive abilities Larger relative brain size or brain component size has evolved to meet higher cognitive demands [ 8 , 13 — 18 ]. Sexual selection Demands of sociality are different between males and females. Life history Variation in juvenile period and life span is hypothesized to affect brain size evolution [ 38 , 39 ].

Method All data used in this study were collected from published literature and are presented in supporting information S1 Table. Model selection was carried out utilizing the Akaike information criteria AIC.

Results We analyzed the effect of six predictor variables on two outcome variables: total brain size and neocortex size. Download: PPT. Table 1. The following model was selected with AIC for total brain size as the dependent variable.

Table 2. The following model was selected with AIC for neocortex size as the dependent variable. Table 3. Table 4. Table 5. Changes in p-value for each predictor when altering concomitant predictors using total brain as dependent variable.

Table 6. The change in p-value for each predictor when altering concomitant predictors using neocortex size as dependent variable. Table 7. Overview of changes in the relation between brain size and predictors as different data is used. Discussion Our analyses indicate that the field of primate brain evolution is best characterized as an array of contradicting results [ 1 , 2 ] and our results reveal one reason why this is so.

Supporting information. S1 Fig. Correlation matrix of posterior distributions for all predictors used in this study calculated using a bayesian multilevel model. S1 Table. Main data set. S2 Table. S3 Table. Correlation matrix all variable. S4 Table. Partial R2. S5 Table. S6 Table. Reevaluating DeCasien [ 35 ] et al. S7 Table. Data used to reevaluate DeCasien et al. S8 Table. Reevaluating Joffe [ 40 ] results of a significant relationships between juvenile period and the ratio of non-visual cortex to the rest of the brain.

S9 Table. Data used to reevaluate Joffe [ 40 ]. S10 Table. Reevaluating Lindenfors [ 27 ]. S11 Table. Data used to reevaluate Lindenfors [ 27 ]. S12 Table. S13 Table. Data used to reevaluate Dunbar [ 19 ]. Acknowledgments We are grateful for input from six anonymous reviewers that greatly improved the manuscript. References 1. Re-evaluating the link between brain size and behavioural ecology in primates. Healy SD, Rowe C. A critique of comparative studies of brain size.

View Article Google Scholar 3. New and revised data on volumes of brain structures in insectivores and primates. Folia primatologica. View Article Google Scholar 4. Endocranial volumes of primate species: scaling analyses using a comprehensive and reliable data set. Journal of Human Evolution. Primate brain anatomy: New volumetric MRI measurements for neuroanatomical studies. Brain, behavior and evolution. View Article Google Scholar 6. Linked regularities in the development and evolution of mammalian brains.

View Article Google Scholar 7. Developmental structure in brain evolution. Behavioral and Brain Sciences. Jerison HJ. Evolution of the brain and intelligence. Current Anthropology. View Article Google Scholar 9. Willemet R. Reconsidering the evolution of brain, cognition, and behavior in birds and mammals.

Frontiers in psychology. Comparative analyses of evolutionary rates reveal different pathways to encephalization in bats, carnivorans, and primates. Proceedings of the National Academy of Sciences. View Article Google Scholar Herculano-Houzel S. Brains matter, bodies maybe not: the case for examining neuron numbers irrespective of body size.

Annals of the New York Academy of Sciences. Rilling JK. Human and nonhuman primate brains: are they allometrically scaled versions of the same design? Overall brain size, and not encephalization quotient, best predicts cognitive ability across non-human primates. Social intelligence, innovation, and enhanced brain size in primates. Do some taxa have better domain-general cognition than others?

A meta-analysis of nonhuman primate studies. Evolutionary Psychology. Bayesian analysis of rank data with application to primate intelligence experiments. Journal of the American Statistical Association. Comparative investigation of the relationship between cerebral indices and learning abilities. Brain, Behavior and Evolution. The evolution of self-control. Dunbar RI. Neocortex size as a constraint on group size in primates.

Journal of human evolution. Shultz S, Dunbar RI. The evolution of the social brain: anthropoid primates contrast with other vertebrates. Dunbar R, Shultz S. Why are there so many explanations for primate brain evolution?

Whiten A, Byrne RW. Tactical deception in primates. Behavioral and brain sciences.