Designing Proof and Evidence with Minorities and Outlier Data

This co-design activity addresses the challenge of working with small group and highly diverse data for evidence-based decisions. As government policy decisions are increasingly based on evidence that neglects the needs of diverse and marginalized groups, we are co-designing acceptable forms of evidence, proof and data for these groups, as well as strategies for collecting and providing evidence. Fall 2021

In a fall 2021 co-design, stakeholders address the challenge of working with small group and highly diverse data for evidence-based decisions.

Government policy decisions, and adjudicative rulings that interpret and reinforce them, are increasingly based on the collection and analysis of “evidence” through processes that rely on large data sets and statistical proof. These processes neglect the needs of diverse and marginalized groups, such as people with disabilities, since their experiences are dismissed as anecdotal or as producing outlier data. And yet it is often members of marginalized groups who are most impacted by policy decisions, and to whom these policy changes do the most harm.

In this design challenge we will co-design:

  1. Acceptable forms of evidence, proof and data that could be produced by a highly diverse group of small minorities and outliers who are marginalized, and
  2. Alternative strategies and tools for minorities to collect and provide this evidence to policy makers.

The recent policy changes to Medical Assistance in Dying (Bill C7) provide one example of this challenge. In this case, the Vulnerable Persons Secretariat was called upon to provide evidence to support their argument that the bill would negatively impact the lives of people with disabilities. Although much evidence was collected in the form of individual narrative accounts, anecdotes, and research from other jurisdictions around the world, it was not possible to meet the requirements for statistical proof. This topic is the focus for our co-design, with the aim that approaches designed for this scenario can be applied to other small minority data challenges.

To learn more about this case study, please see Catherine Frazee’s testimony to the Standing Committee on Justice and Human Rights, November 10, 2020.

The Challenge of Proof

In preparation for our Minority and Outlier Data Co-design, we’re encouraging everyone to think about the challenge of proof for a small group, even a group that is just one person: you.

Think of something you experience in your life that is difficult to demonstrate. Perhaps it’s an emotion you feel, a value you hold, or a pain you experience. How might you go about proving your experience to someone else? How might you document or demonstrate it? What kind of proof could you produce to convince someone about the validity of your experience?

You can use the comment form below to share your ideas of how to approach this challenge with the We Count community.

Comments

*

Indicates required field

*

Required
*

Required
Thank you for submitting your comment. It will be posted on the page once it is approved

Rosalie | October 19, 2021

As a policy analyst in government, I have observed that our community often employs policy research and agenda setting processes that have very much been designed to create evidence and address specific policy issues in a way that usually meets the needs of the statistical majority, the “average person”, or, often in the case of health policy, to medically address difference. Typically, I work with a public policy process that is a series of sequential stages that build upon each other. These are (1) Issue/problem identification, (2) Agenda setting, (3) Policy option development, (4) decision-making, (5) implementation, and (6) evaluation. Though current GBA+ guidance provided to Policy analysts warrants that policy research examine gender in combination with select identity factors, I feel that policy analysts are given much flexibility to ensure that their agenda setting and policy options account for the full range of human diversity with respect to the intersections of gender with ability, language, culture, age and other forms of human difference that they believe apply. What I like about Inclusive Design philosophy is focused on the idea of designing with/for the full range of human diversity, and this requires that we redefine the types of evidence that should be considered valid.

Laura J Mac | October 19, 2021

The "challenge of proof" is nightmare fuel. The oppression built into this is glaring to those of us without power and (usually) completely invisible to those in power. Not as a commentary on motives/intentions, it's simply a fact that white supremacy centers our whiteness, the patriarchy centers men (and toxic masculinity), our society centers (and reveres) ableds. It's also a fact that people much smarter than me have studied this and provided ample evidence that our society defaults in these ways to the point that the barriers are structural. Human Rights Codes are made up from these truths, not just occasionally but as a rule. The way I've been thinking about this question over the weekend is through the human rights lens, asking myself why would we have any need to demand proof of racism or ableism or classism etc. occurring on an individual level - we know this happens in society to the point that it's systemic: so it seems a fully flawed premise to ask for proof that systemic barriers/violence are having an impact on any one individual/s. It seems an abuse of power if we're being very honest about it. Its further violence to frame it as outlier data that can be brushed off, might as well just call the rule book "It didn't happen to me so I don't care"  Hunger is an example of an experience that captures this well. I can't see your hunger, but if I'm committed to simply, and always, centering you as a human being, I don't need "proof" even as far as an explanation of  "I haven't eaten all day", or witnesses to confirm, or step on the scale to show me with hard math (offset by the weight of the day's bowel movements) to make really sure you are indeed hungry when we sit for dinner. The science says an average human needs 2000 calories per day. What this should mean in practical terms is: Nobody ever needs to explain that all humans require food and water to survive, every single day - yes, even disabled people.  But this is effectively what our welfare/disability support system demands of each of us as vulnerable individuals in survival need - requiring that we bring doctors in to verify on an individual level the basic fact that ALL humans need daily calories and water, without exception. And we have normalized this business of dehumanizing and marginalizing vulnerable people by requiring professional strangers to validate basic facts that are true of all humans. Then by replicating this oppressive formula, all of these systems are built well to exclude us, and then shrug it off as our personal failing (*outlier!!) and the system as the righteous arbiter of all that is good and true.  As an example of how this Less Than Human status is clearly cemented into policy: ODSP (Ontario Disability Support Program) is 58% of CERB, and it takes years to get from OW (Ontario Works, general welfare) at 36% of CERB, to ODSP... If "Canadians" needed a fair financial bandaid during a health crisis and that value was determined to be $2000, why do I need to provide "proof" that $1169 (ODSP income) or $733 (OW income) is inadequate for a disabled human's survival? Are disabled people not "Canadians in a survival crisis". Better yet, why do each of us (half a million people in Ontario) have to present this case to the human rights tribunal to have this confirmed for the systemic violence that it is. Makework is violence. No, I won't provide proof of these facts either. So anyway, we have these very reliable social determinants of health and wellness, but then after marginalizing people as a reaction to our vulnerability, these oppressive-by-default systems continue to put the burden of proof on vulnerable people who are marginalized by these very systems. So, we have a huge smash of people whom we know have had our capacity all but stripped by chronic survival crisis, and then without a hint of irony demand proof it's happening. And the very ironic thing about these demands for proof, in any of these systems; if I showed up in my genuine state, which is feral and hissing, or in a state where I'm only able to communicate in emojis with a face that can't stop crying, this very real evidence of the systemic violence I'm experiencing would be thrown out, and likely taken as contempt of court for being in a format that's uncomfortable for people who usually only intellectualize this stuff. So I can't help but to think on all the people I've talked to about all of this in acute and broad terms, all the people who don't know fancy language and social justice theory, or they're not sober, or they just have crappy old clothes so no matter what they say it's dismissed because of their torn sneakers, and I become (even more) furious at the commitment to dismiss human rights as problems of an individual. It all comes back to human rights, and I'm left wondering why "highly educated" people who don't even understand these very basic concepts are tasked with developing and managing them when the drunk lady with torn tennis shoes who can't stop swearing and smoking seems to have a better handle on the facts of the matter, but she's too busy trying to stay alive outside to be able to be here, fighting at the Human Rights Tribunals, etc etc... she doesn't need to submit any proof that maslow's first tier of survival needs applies to her situation.

Jane | October 18, 2021

Being part of a group that is both underserved and understudied means that many stories and experiences are left out of the equation (i.e. what counts as statistical proof). For instance, many racialized im/migrants' experiences with being autistic and disabled are not accounted for because some of us may not have time to participate in studies, may not see ourselves as disabled, may not be diagnosed, and may not trust those conducting studies due to histories of exploitative, colonial dynamics of university sociological/psychological research. For someone who may not know they're autistic and may not see themselves that way, there is little opportunity for the right questions to be asked in the first place and if asked, those questions would only account for certain intersections, and not all. For this reason, the process of gathering evidence is contrived. Evidence is thus only gathered to feature what a researcher wants answered: how does one count those on a statistical model those who are not seen in the first place? If the research methodology itself is flawed and rooted in unjust power relations (for instance, a deficit model of autism), then evidence becomes further contrived. For evidence then to not leave out aspects of people's lives as well as to be fully understood, life narratives and personal accounts taken from longitudinal studies or focus groups should all be taken into account. Many researchers are doing this work and using life narratives and first-person accounts as the basis of evidence for their research, and thus these life narratives should be seen as sufficient evidence. While I do not think all life narratives should be taken at face value (especially if, within political arguments, they can be manipulated for a specific purpose), I do think research based in genuine relationships to a community, cultural respect and reciprocity, and informed consent that take personal accounts as evidence should be uplifted. In this context, outliers are not so much exceptions to the rule but rather becomes another possible study that hasn't yet been done. An important part of research then becomes creating an environment where evidence can be equitably gathered. For communities who are asked to share traumatic stories, does the sharing of those stories bring healing rather than further trauma? Participants of research have historically (especially in the early days of the Stanford Prison Experiment or the Tuskegee Experiment) been deliberately harmed and neglected, their welfare never designed into the study itself. So evidence that isn't gathered within a study may not be counted because of the inequitable dynamics of the research conducted. For me, one specific experience that I find difficult to demonstrate is the extent to which I have experienced sexual violence throughout my life. For me, it's not the number of times something that has happened (though of course that impacts me), but rather the significance of specific incidents where I was not supported or heard. I can experience racist catcalling, but what I remember more are the times when people I know and trusted have broken my trust. This experience becomes further complicated when I take into consideration the multiple ways I am marginalized. If a study was, say, conducted on women of colour and sexual assault, it would be difficult for me to count all the evidence into the study because I do not consider all my experiences of sexual violence to be sexual assault, and there are times when my gender is not strictly "woman." Given the way that court systems, for instance, continuously refuse to recognize this kind of personal testimony, I may feel compelled to minimize my experience or not talk about it at all due to the feeling of being ignored or not taken seriously, or not feeling like telling my story would bring me further healing. The difference between seeing sexual assault as a single act vs sexual violence as a spectrum of experiences can significantly change how people offer up evidence of their experiences.

sb. | October 15, 2021

When it comes to experiences that are difficult to demonstrate or prove, micro-aggressions always come to the top of my mind first. When I attempt to convey my experience with these to others, I primarily employ oral storytelling. If I'm telling this story to a fellow disabled or fat or queer person, I'm often met with understanding and sympathy. If this story is being shared with someone more privileged (an abled, thin, and/or straight cis person) however, the gravity of the experience is usually overlooked or dismissed. I think this is a replication of what we see in an institutional context–like statistical evidence requirements for arguing down bills, for example. Micro-aggressions seem near-impossible to document. However, I could imagine isolating one experience (e.g. getting stared at by strangers in public) and noting down how many times this occurs over a given period of time–although it's not hard "proof," this might at least be one concrete way of demonstrating for others the frequency and consistency of the experience.

John Stapleton | October 15, 2021

Lived experience of poverty, racism, disability and its articulation is a first person psychological report that is not subject to refutation. Storytelling from lived experience is 'proof' and should be accepted at face value. Critical thinking is another matter. For me, the 5 most important elements are: deductive thinking that relies on e.g. mathematical 'proof'; inductive thinking that relies of scientific or empirical proof closely aligned to what we understand to be evidenced based; - analogical thinking that relies on reasoning that is closely aligned but not the same e.g. if we have vaccines, why do we need face masks? Analogy: would we stop using seat belts because we have air bags in cars?' phenomenological, gestalt or deconstructive thinking that reframes what we see and reveals that values frame evidence e.g. the cost of a program that helps the poor can be re framed by thinking of the cost of poverty: How much do we pay to do nothing?; Dialectical thinking that poses a hypothesis, its negation and the negation of the negation e.g. this thinking very evident in Covid-19 thinking. I would think through all five of these forms of critical thinking to help structure my storytelling of the value or the pain or position I hold.