In an earlier report, OpIndia.com has comprehensively debunked the BBC research on ‘Fake News’. The BBC has tried to brush the criticisms under the garb of ‘Qualitative Research’ and ‘Ethnographic Research’, therefore, it is imperative that we take a moment to understand what these terms actually mean and what such research entails.

For those unaware, Qualitative Research focuses on subjective experience and it’s objective is not to determine objective truth which can be generalized across all sections of the population. It employs techniques such as interviews and on-field observations to develop a greater understanding of a particular social phenomenon. It seeks to answer the questions of ‘why’ and ‘how’ to understand a particular social phenomenon. One of the ontological principles of qualitative research is that there are multiple realities which can be accessed through subjective experiences.

Limitations of Qualitative Research

One of the fundamental positions of Qualitative research is that there are multiple realities and people perceive reality in their own unique way. Thus, the focus is not on discovering an objective reality that holds true across all situations but to explore relationships within a unique setting and based upon the purpose of research and its results, develop hypotheses which could then be tested using Quantitative methods. Qualitative methods cannot be generalized to any significant extent, much less, an entire country.

Even an editor at the BBC claimed that since it isn’t possible to interview 1.2 billion people, ‘there will have to be a generalization’. Anyone who knows anything about Qualitative research knows that generalization is not the objective of such research and the results obtained from it cannot be generalized across all situations to any significant extent. It appears that the BBC isn’t even aware of the purpose and objective of the Qualitative research.

Unless we interview all 1.2bn people in India, there will always have to be a generalisation though. We are at least laying open our methodology and want to encourage genuine conversation and debate about this research. #BeyondFakeNews

— Trushar Barot (@Trushar) November 15, 2018

There are some crippling limitations of Qualitative research. It depends a lot on the personal skills and biases of the researcher. Since such research employs methods such as interviews and case studies, a research may choose to focus on one aspect due to his or her own individual biases even though another detail in the study is more significant.

Under such circumstances, it was imperative that researchers in their team included people of diverse political affiliations if they wanted their research to be more authentic. However, the research makes no mention of the political affiliations of the researchers themselves.

Flaws in Research Methodology

Flaws in the using the ‘localised version of left-right orientation questionnaire

The research states that it used a “localised version of the 10 point left-right orientation questionnaire developed by John Curtice and Caroline Bryson” to determine the political affiliation of their subjects to ensure an equal representation. But the reliability and validity of the localized version is not mentioned. And it’s unclear to what extent such a test would be effective in a country like India considering the ‘Left-Right’ paradigm is not applicable in our country. For instance, if the test indicated that a person was likely to vote for Shiv Sena, would he be classified as pro-BJP or anti-BJP?

The researchers state that using the localized version of the test “allowed us to recruit for political affiliation without asking questions about people’s voting behaviours or support for particular political entities.” The researchers need to justify their reasoning here as India has a multi-party political system where the left-right paradigm is not applicable by giving more details about the localized version of the test that was used. Binaries can work for countries like the USA where they have to choose between the Republicans or the Democrats, but in India, that prism gives a skewed result and thus, the validity of methodology employed has to be provided with respect to India.

Flawed definition of ‘Fake News’

The problems with the research begin with the operational definition of ‘Fake News’ itself. The research defines Fake News as “information, however, conveyed, in whichever format, on whatever platform, which is not fully supported by factual evidence”. The researchers appear to be unaware that people across the political spectrum differ on what constitutes as factual evidence and that naturally pervades even the academia whose standards of factual evidence they claim to have followed.

However, if ‘information not fully supported by factual evidence’ is the core definition, then will every message declaring the greatness of Gods be dubbed ‘Fake News’? More realistically, will messages that proudly proclaim “Diversity is our strength” be treated as ‘Fake News’ because there’s evidence which suggests that diversity is definitely not our strength and homogeneous societies tend to function a lot better in almost every respect.

If the aforementioned message is regarded as ‘Fake News’, then every single liberal is guilty of being the purveyor of Fake News. Why haven’t they been named in the list of people who spread Fake News? The fact that OpIndia.com, Swati Goel and R. Jagannathan were mentioned as purveyors of ‘Fake News’, it only goes to show that their definition of ‘Fake News’ and ‘factual evidence’ aren’t reliable at all. And what about that news which was believed to be true at one point of time but was proven as false after a couple of weeks, or maybe even a couple of days, with the revelation of new data? Were those clubbed as Fake News as well considering the study was completed within 3 months?

Flaws in selection and nature of interviews, sample selection and generalisation

The follies in the research do not stop there. There are multiple mistakes to show that the researchers themselves were not very serious about their research. For one, The research was limited to major cities alone. Smaller towns and cities did not have any representation, let alone the rural population. Secondly, 40 subjects were distributed across 10 cities, that is 4 subjects on an average in each city. With such an infinitesimally small sample size, that the researchers hoped to learn anything at all from their study only shows that the entire exercise was merely to provide them with a convenient excuse to peddle their agenda.

Under the best of circumstances, the results of Qualitative research can be generalized only across immediate homogeneous settings and here they have generalized the opinions of an average of four people across 10 cities to the entire nation which just magically happens to be one of the most heterogeneous countries in the entire world.

Moreover, the researchers state that they screened their subjects only for political affiliation and economic background. Any student of social science is aware that the educational qualifications of people play an important role in determining their behaviour in a social setting.

In the ‘Data and Methodology Appendix’ section, the researchers do not mention anywhere that they took the educational qualifications of an individual into account. If the BJP supporters who were part of the study were not very highly educated and the anti-BJP subjects were, then it is natural that their study will come to very dubious conclusions. Moreover, the researchers do not state whether they took into account the mental health of the subjects or any personal history of mental illness or in their family which could be a very important factor as mentally unhealthy people may be more inclined to believe and share ‘Fake News’.

Even the economic background of individuals isn’t mentioned as a criterion in the ‘Data and Methodology Appendix’ section although it is said in Methodology section that subjects came from a ‘mix’ of economic backgrounds. If they indeed came from a mix, then what are the different categories of economic backgrounds the subjects were categorized under? It is not mentioned. Was it ensured that the four groups of 10 subjects each had an equal representation of supporters from various political parties from the various categories of economic background? It was not mentioned.

The researchers haven’t even mentioned the sampling technique they have used. In Qualitative Research, random sampling is not employed. Usually, the purposive sampling technique is used to ensure that researchers have the desired subjects necessary to fulfil the objective of their study. The researchers state that they ensured there was equal representation of various political affiliations among their subjects. But with the multitude of political parties that we have, how did they ensure such a thing and how many supporters of each political party was part of the research exactly?

It should also be borne in mind that it’s true that ethnographic studies generally study a very small population and the BBC is not wrong when they say it explains the small number of people they studied. But they compromise on their ability to generalize across different situations with their small sample. And when the sample is so small, researchers need to have an extra robust sampling technique if they wish to generalize their findings at all. Therefore, what measures were taken to ensure that researcher bias did not creep into sampling? Were any measures taken at all? Unfortunately, the research provides no answer to these questions.

Call for Quantitative research

The conclusion segment of the research states, “Given that central to our observations is the idea that people’s socio-political identities seem to influence their consumption and sharing of fake news quite a bit, and that national pride and nationalistic feelings seem to affect their critical thinking, fake news will need to be handled as a social problem. We would urge future researchers to quantitatively assess – perhaps through regression analysis techniques – just how important socio-political identities are in fake news consumption relative to other factors.”

Thus, it is clear from the conclusion that researchers believe certain aspects of people identities ‘seem’ to felicitate sharing of fake news and call for quantitative research into the matter. Journalists at the BBC, on the other hand, take the leap to “Nationalism a driving force behind fake news in India, research shows” without mentioning that the researchers themselves call for more Quantitative research to determine the extent to which socio-political identities play a part.

Ethnographic or Semi-ethnographic?

The research employs words like ‘ethnographic’ and ‘semi-ethnographic’ liberally. On page number 18, the research states that they used ‘semi-ethnographic approaches’ to describe their interviews while in the ‘Data and Methodology Appendix’, they mention ‘ethnographic in-depth interviews’. Which one was it? Ethnographic or semi-ethnographic? An editor at the BBC also claimed that it was ethnographic research but how could it be if it adopted semi-ethnographic approaches? To the laymen, it may appear trivial but you cannot have that ambiguity in your research if you want it to be taken seriously.

The most egregious assault on academic rigour was the fact that the researchers took only 7 days to conclude their field immersion. In Ethnographic research, the researcher attempts to study people in their own natural setting. This requires the researchers to get to know more about the day to day life of their subjects and the subjects, too, need to get comfortable around the researchers to behave as they would naturally so that the researchers can observe their natural behaviour. Is 7 days enough for such an endeavor? Ethnographic research is unique for the long duration it takes to complete the collection of data. Generally, it takes months if not a whole year and on certain occasions, it takes years to complete the collection of data.

Of course, due to practicality, the time frame is being decreased due to various constraints but 7 days although not unheard of is much too less if you are to make such fantastical claims. One could call it semi-ethnographic (as they did call it in the methodology section) perhaps but it certainly cannot be said that the findings will offer results that could be generalized on such a high scale. It could still be excusable if they had mentioned that field immersion for the ‘ethnographic’ research was completed in 7 days, as research methods are subject to various constraints, but they should have mentioned and highlighted that fact if they were honest about their research.

One of the most fundamental criticisms of ethnographic research is that researcher bias plays too much of a significant role in the conclusions that are drawn. The bias of the research was amply revealed when they used portals like AltNews as their source for Fake News. Therefore, whatever findings the BBC observed from their study can be said to hold true only for those set of 40 people at its worst and maybe, just may, for other groups which bear a close resemblance to that particular group of 40 people. The Qualitative study does not reveal a single thing about the phenomenon of Fake News beyond that which could be generalized for other groups.

The crisis in social science

BBC’s research also points towards the crisis within social sciences. There have been numerous revelations regarding how corruptible social science research is. There is a widespread replication crisis where researchers have failed to reproduce the findings of previous studies and experiments.

On plenty of occasions, especially in high profiled ones, researchers have admitted to falsifying data to produce desired results. The BBC’s research may well be one of them. The million other problems with the research, including how their algorithm worked, has already been elaborated by my editor and therefore, I shall not elaborate on them here. BBC also mentioned that their algorithm selected for portals which shared even a single fake news. If that was indeed the case, why don’t all mainstream media channels including the BBC feature in the list? In conclusion, the BBC research is an assault on academic rigour and Ethnographic research.