There is a movement afoot to encourage a more systematic and replicable form of literature review within science. On the one hand, I do recognize and approve the notion that we need to be broad in our reviews and not ‘hunt and peck’ among the materials to support our own pre-existing views. Such an approach is clearly undesirable from the standpoints of enhancing human understanding (my own interest) or predicting human behaviour (as some seek). However, the approach of sticking only to a pre-specified set of key words and dates to do searches on the internet—the most widely encouraged suggestion—has serious failings.
The one that first came to my own attention is the small number of books that emerge using this approach. A book author is normally quite unable to specify 5 keywords (a common number allowed) to reflect the contents of several hundred pages. Since social science material particularly is often published in book form, whole disciplines become disadvantaged, in the sense that their materials are unlikely to emerge in the review process. My own concern is with anthropological and ethnographic analyses, which particularly rely on such materials—and which I consider important to incorporate into many practical fields, for instance—though the problem is broader than this field alone.
Secondly, and related, although some topics are amenable to brief explanation, any approach that looks at human behaviour holistically will have difficulty explaining what is happening in a short form—leading both to the production of books (again) and to the inability to summarize the topics covered in a small number of keywords. Portrayals of human (and biophysical) complexity represent treasure troves of material we should be seeking, not avoiding.
Thirdly, there is the question of terminology. Different disciplines use the same term for different phenomena: Culture means a human lifeway to an anthropologist, a blob in a petri dish to a biologist, cultivation to an agronomist. Although one can distinguish these different meanings, this introduces an element of subjectivity that proponents of systematic reviews are trying so hard to avoid.
Additionally, authors may simply neglect to mention key features—irrelevant to their own analyses—about which they have written material, which is extremely pertinent for a new topic. In a recent review of gender issues occurring within dry forests, for instance, I found that many social scientists working in dry forests, having no particular interest themselves in that feature, failed to mention the habitat in which the people lived, though their material was very pertinent to my search. Such analyses also fail to emerge in simple key word searches.
The use of different terms for the same thing (or very similar things) in different fields represents another more serious constraint if we limit our reviews to a few key words. Anthropologists may refer to swiddens, agricultural scientists to slash and burn; foresters to shifting cultivation. Extension agents write about farmers, where gender specialists differentiate men and women—meaning that sticking to a pre-specified set of terms is likely to rule out a significant amount of very relevant material. One can of course constantly revise one’s original estimation of what to include before finally settling on THE chosen set—which moves us toward improvement in the review system. But, knowing human nature, many will deem such an iterative process more trouble than it’s worth—particularly given our equally human inclination to work within our own silos of knowledge.
In a recent search using the key word ‘equity’ (or more precisely, ‘equit’)—in an effort to satisfy my suspicion that humid forest literature was more likely than dry forest literature to feature/stress equitable social systems— I had to distinguish between articles that talked about ‘not equit’ or ‘inequit’ exclusively or predominantly from those that talked about truly equitable systems. Determination of what to include and what not to include again required human judgment.
Finally, neglecting old literature and the knowledge and guidance of experts in the field strikes me as even more problematic. Not everything is available on the internet. To do a proper review, to understand the topic we are seeking to summarize/synthesize/reflect in a review, it seems to be a travesty to ignore what has come before.
I recognize and applaud the concern to cover a topic fairly and as objectively as is humanly possible. However, the suggestions I’ve seen so far for doing this basically ‘throw the baby out with the bath.’