Collaborative Ignorance in Software Engineering
Why We Believe What We Think Our Group Believes
There’s a problem in our software industry. Whenever we form a group to accomplish a task, we often engage in collaborative ignorance.
What is this, and how does it happen? It happens because, as individuals, we may believe that our group believes something that we personally doubt or don’t understand. Yet, we accommodate the belief, so we don’t risk being cast out of the group.
The good news is that this dynamic isn’t unique to just our industry. The bad news is that it’s a byproduct of most groups [1], meaning we can’t easily escape it.
Definition
Formally, social psychologists give the name Pluralistic Ignorance to a group dynamic where individuals in the group either privately disagree with the group’s belief or fail to understand it, yet in both cases, are reluctant to speak out and voice their concerns [1].
This process usually arises because the individual believes that all the other group members believe something to be true. So the individual decides to believe it as well, even if they harbor doubts. Collectively, it often occurs that many people in the group hold the same mistaken belief that everyone else believes something when many of them actually don’t.
Yet no one speaks up regarding their doubts or concerns. This silence usually happens because it’s safest for an individual to simply play along with everyone else rather than risk ridicule for “not being smart enough to understand” or to be ostracized as someone who “isn’t a team player.”
The result is an actual group belief, and a perceived group belief that each individual thinks is true. The actual and the perceived beliefs are often not aligned.
An even more troubling dynamic is that an individual in the group will often punish a non-believer, even if that individual has doubts, to prove to the group that they are a member in good standing and can be trusted to uphold the group’s beliefs [2].
Even though Pluralistic Ignorance is the formal name, it can be argued that this process is more than just a plurality of ignorance. Instead, when we collectively fail to voice our doubts or lack of understanding, we choose to collaborate on being ignorant, even if we don’t consciously realize it. For that reason, it’s perhaps more aptly described as Collaborative Ignorance, the name we will use in this paper.
Examples
Let’s look at some examples of Collaborative Ignorance in the software industry and see if they help us understand the phenomenon.
Estimates
A telling example is estimates. Individuals often harbor private doubts about their usefulness, but the group makes a collective decision to be ignorant about them despite abundant evidence of their abysmal track record [3].
If you poll individual workers about their private beliefs regarding the value of estimates and how they’re derived, you might hear statements like, “I just make something up,” or “I just always say two weeks,” or “It’s all nonsense. I just tell them what I think they want to hear so they can put it in their plan.”
Suppose you reach high up into the organization, well above those who create the estimates and into the range of executives who use them. In that case, you will likely hear statements like, “An estimate is a carefully-crafted assessment of how long a project will take. We use them to plan and allocate resources according to our budget.” Occasionally, you might hear something like, “A number that we never seem to hit, but our investors require them, so we do them.”
Notice the disconnect between what is believed as a group and what is professed as an individual. Each individual chooses to ignore their inner doubts and believe what the group believes.
Further, notice what happens when someone from the group speaks up about their inner doubts regarding estimates. I once did this in a meeting with a senior executive and was quickly admonished that I “didn’t understand how business works.” The lesson was clear: thou shalt not disrupt the group’s beliefs. These are classic signs of Collaborative Ignorance.
Project Schedules
Closely tied to estimates are project schedules. Even with the advent of Agile methods, it’s still common to see complex Gantt charts that plan how a project will unfold over an extended time. But most software work involves creating new products from scratch, which by nature requires invention, and invention is inescapably non-deterministic [4] and immune to long-range plans. Worse, the plans often turn unreliable estimates into commitments to deliver on a fixed schedule. So we effectively compound our ignorance by piling one form of delusion on top of another.
Yet we pay people to develop complex plans, then commit the business to deliver on them. As a group, the unshakeable consensus belief is that we will achieve what we plan. However, if we were to ask group members about their private beliefs, we would find severe misgivings about the plan’s viability. If we’re lucky enough to hit the schedule, it’s usually because of heroic overtime, cutting corners on quality, and reducing features. But we rarely question our approach and choose instead to be collaboratively ignorant about the problems by continuing to do the same thing.
The Dress Code
This particular example is less typical and more oblique but is illustrative. Some years ago, I worked in a company that believed a dress code was needed. I never understood the reasons, but they seemed to revolve around the belief that dressing in shorts and flip-flops was “unprofessional” and reflected poorly on our commitment to the work. Hence, a dress code would display our seriousness and, I suppose, improve our work quality.
My executive supervisor assigned groups of us managers to develop the dress code. As it happened, our particular group was tasked with deciding what the women in our company would be allowed to wear. Inexplicably, our group consisted of all men. Truly. And this was not the far, distant past when such things were distressingly common. This was frighteningly recent when everyone should long have known this was inexcusable.
When I raised the issue to our group leader that perhaps it wasn’t a good idea that a small group of men should dictate what the women would be allowed to wear, the leader refused to consider the issue. He replied, “I don’t care. My boss gave me a job to do, and I’m going to do it.”
Unfortunately, this is the “just following orders” defense, a tactic with a long and deeply disturbing legacy. In our group’s case, I excused myself by feigning that I had a customer emergency to address rather than suffer the maelstrom that was sure to follow. The group soldiered on in my absence by doing as they were instructed, choosing to be collaboratively ignorant even though I suspect many in the group privately thought our mandate was absurd.
Going Along to Get Along
Let’s consider the “just following orders” philosophy. With this approach, we effectively surrender our agency and independent thought and choose to be collaboratively ignorant. It’s common in closed, hierarchical organizations that reject outside information and are intolerant of those who question the system while promoting those who keep the wheels of the bureaucracy well-greased. We must go along to get along and keep our jobs.
Contrast closed systems to open systems where diverse viewpoints are welcomed, and the pursuit of ever more effective methods rewards those who question the system’s beliefs. In this kind of system, questioners aren’t shunned or treated as apostates but are instead expected to raise doubts as part of their professional obligations. To do otherwise would be to shirk our duties.
Which kind of organization do you work for? A closed or open one? Ask yourself this to find out: what happens when the CEO announces a new initiative at an all-hands meeting (which in itself is a warning of a closed system), then asks, “Any questions?” Does everyone fall silent? Or does someone feel free to say something like, “I don’t think that’s such a good idea, and here’s why.”
A closed system where everyone chooses to be collaboratively ignorant is one where the CEO’s new initiative immediately becomes the company’s highest priority, and its merits are never debated. An open system, where individual agency and independent thought are rewarded, is one where the new initiative isn’t handed down by royal decree from on high but is instead open for everyone to review and modify.
Reducing the Problem
It would be reassuring to close with some simple strategies that would allow us to avoid the problem of Collaborative Ignorance. Alas, that isn’t possible because Collaborative Ignorance is endemic to human nature [2]. As long as groups of people work together, the problem will exist.
However, some things will help offset the problem. The first is to have the courage to listen to our inner voice that speaks to us by saying, “Wait, I don’t understand” or “Hang on a second, are we sure we want to do this?” If we squelch that inner voice out of fear of reprisals, it’s a warning sign that we might be working in a closed, collaboratively-ignorant system, and we’re better off elsewhere.
The second thing to do is to inspect our beliefs. If we need clarification about what we believe to be the group’s belief system, we should poll the group members privately and ask what they think. If they know that it’s a private conversation and thus safe to express their true beliefs, then it’s possible to discover whether individual beliefs correspond to the group’s beliefs. From there, we can assess if the group is being collaboratively ignorant and decide what action to take.
The third thing we can do is share the problem with the group. Suppose everyone in the group knows the problem exists and shares the philosophy that they must work together to solve it. In that case, individuals will feel empowered to safely state, “I don’t understand. Please explain it to me,” or “I’m not sure this is a good idea, and I want us to avoid Collaborative Ignorance. Who else feels this way?”
Finally, just being aware that the problem exists can allow us to take steps to offset it. If we’re ignorant of our own Collaborative Ignorance, then it’s unlikely that we’ll ever address it. If we’re cognizant of the problem, we can bring our ignorance into a spotlight to inspect it and minimize its effect.
References
[1] “Pluralistic Ignorance,” Reed College, https://www.reed.edu/psychology/pluralisticignorance/
[2] Miller, D. T., and McFarland, C. (1991). “When social comparison goes awry: the case of pluralistic ignorance,” in Social Comparison: Contemporary theory and research. https://psycnet.apa.org/record/1991-97036-011
[3] “Accepting Uncertainty: The Problem of Predictions in Software Engineering,” J. Meadows, Medium, May 2020.
https://jmlascala71.medium.com/accepting-uncertainty-the-problem-of-predictions-in-software-engineering-26dbcd120b90
[4] “Paying Someone to do Our Homework: The Risk of Mediocrity with Agile Frameworks,” J. Meadows, Medium, October 2022.
https://jmlascala71.medium.com/paying-someone-to-do-our-homework-the-risk-of-mediocrity-with-agile-frameworks-2fbdea1d3a39
