Full Methodology
Data Collection
Communities Against Hate has collected data since November 2016 by way of a convenience sample of individuals who desire to share their stories with the Communities Against Hate team. They may report to us directly through our site or through one of our partner sites who have the same or similar forms for reporting of incidents. They may also call us on the hotline and speak with individuals at the Lawyers’ Committee for Civil Rights Under Law who record their story and then provide legal, mental, or social services.
Additionally, Communities Against Hate builds on this aggregation of data by supplementing our reports with news clips that exhibit hate incidents from around the country.
Direct Reports
Communities Against Hate and national, local, and state partner organizations include a link to the form on their websites which is advertised through community events, social media, listservs, emailed newsletters, and/or distributed palm cards at events or at the front desk. Through this form, individuals can report an event in the way that best describes their experience. Employees at the national organization level and The Leadership Conference Education Fund review the incident and determine if it should be approved for the database.
Determining the Report Should be Approved:
Step 1: Deciding if the Incident was Hate or Bias Motivated
Hate Incident: A bias-motivated incident committed, in whole or in part, because of actual or perceived race, color, religion, national origin, gender, sexual orientation, gender identity, disability and/or ethnicity. Hate incidents may or may not constitute a crime.
Step 2: Verification
There is a general belief that people are truthful in sharing their stories with CAH. It remains a founding principle of CAH to promise a space for those impacted to share their story about how they have either experienced or witnessed a hate incident. The first step for many organizations is to ensure that the incident occurred or that the incident is not “spam.” Some organizations have a verification process to follow-up with the individual, if contact information is available, to verify that reports are real and to get a better sense of what occurred. Additionally, some of these organizations may also search for a corresponding news story to verify the report.
If there is any doubt about the validity, the entry will be disapproved and not included in the database. However, if there is reasonable consensus that the entry is true, the incident will be approved for inclusion in the database.
News Clips
Our data collection also includes news clips traditionally found by The Leadership Conference Education Fund through a search of news clips and various newsletters, and the ProPublica news stories database. Relevant stories hosted on major news sources are evaluated against the approval criteria and entered into the database. Local news sources for smaller towns and cities are also used. If law enforcement, a community, or the family perceive the incident to be motivated by hate, the incident will be entered and approved in the database. If there is a question regarding the true motivation of the incident, the incident is followed until more information surfaces. Then, the incident can be entered into the database if found to be hate motivated. News stories are also researched and flagged for the database via partner organizations.
Data Analysis
The Leadership Conference Education Fund houses a data team that has created a methodology under the context of this aggregation of data incidents as a convenience sample of individuals. While our database currently has more than 4,500 entries, we have only reviewed entries that were said to have occurred between November 2016 and May 2018. Incidents that are determined to be spam or out-of-scope by the database manager are removed from the analysis. Three-thousand, nine-hundred-eighteen entries were qualitatively analyzed for several codes including the action that occurred in the incident, whether multiple or singular incidents were experienced by an individual, invocations of hate groups or politicians, emotions felt by witnesses or impacted individuals, immediate actions taken in response by impacted individuals or witnesses, knowledge of bias motivation, long-term actions taken after, and evidence of outcomes or longer-term effects. Additionally, the analysis collected categories within the form including the cited motivation for the incident and the location type where the incident occurred as conveyed by the individual who submitted the report.
The codebook for qualitative analysis was developed during multiple explorations of the database by the lead analyst who remarked on themes as they became more apparent. Additionally, national partners contributed categories that they were interested in learning more about from the database. Finally, throughout the process, a team of analysts contributed to the codebook when a new potential theme arose. The codebook was approved by The Leadership Conference Education Fund, following consultation with national partners.
Data analysis was conducted by three analysts. One 15 percent sample of co-coded entries was conducted by all three analysts to ensure high-level coder agreement. The three analysts were then each given a portion of entries to code separately. A final 5 percent sample of co-coded entries was studied for coder agreement at the end of the coding process as well. Where differences in interpretation of definition occurred, the analysts conferred collectively to decide on the outcome of the code.
Coder agreement was consistently high across both samples. In the 15 percent sample, percent agreement occurred over 85 percent in all categories, with most codes occurring over 95 percent agreement between the three analysts. In the 5 percent sample, percent agreement was over 95 percent for all codes.
Hate Incidence Poll
The Leadership Conference Education Fund commissioned a poll to better understand the reality of hate incidents from a representative sample across the United States. These findings are from a proprietary survey conducted by brilliant corners Research & Strategies on behalf of The Leadership Conference Education Fund. This nationwide survey consisted of 800 adults, as well as oversamples of 200 African Americans, 200 Hispanic Americans, and 200 Arab American/Middle Eastern Americans. The survey was conducted by phone and online, starting on Sept. 30, 2018 and ending on Oct. 16, 2018 and requested information regarding experiencing, feeling, and witnessing hate incidents or expressions over the last two years. The data was weighted slightly to adhere to population demographics of the population in the country. The margin of error for the sample is +/- 3.5 percent, with a 95 percent confidence interval. Notably, the margin of error for sub-groups will be slightly higher depending on the size of the sub-group sample and the size of the actual response to any given question. The poll survey instrument was designed based on the collected data from the Communities Against Hate reporting system as well as codes from the findings of the qualitative report. Findings and data taken from the poll and data from the Communities Against Hate database are distinguishable throughout the report.
Limitations
Our database has collected incidents from a variety of sources. Anyone who wishes to share a story with the Communities Against Hate initiative has been welcomed to do so. Where possible, we attempt to confirm that a story has truly taken place; however, it is our founding principle/belief that people are truthful when they share their story with CAH. We believe that sharing a story is itself therapeutic. Additionally, it is not the mission of Communities Against Hate to serve as judge and jury over alleged incidents that occur. We merely strive to share and report on these stories to help inform the public narrative on what is actually happening in this country as it relates to hate.
Our analysis here relies on a convenience sample delivered to us through individuals’ own words. Any conclusions made in this analysis of the CAH database are not meant to be generalized to the entire nation, but instead offered as a way of explaining the hate incidents that have come to our aggregate database. As our database is not entirely exhaustive, we are certain that more hate incidents occur throughout the nation than what our database holds. No organization or government institution has yet been able to accurately capture the exact amount of hate incidents that occur across our country each year. Offered throughout are numbers from the Hate Incidence Poll, which help to speak to the representative sample to provide context to the stories we have discovered throughout the database.
Additionally, we are not proposing that this database can be compared to the FBI database, which tracks hate crimes as defined by federal law. We track hate incidents, some of which may be hate crimes under a state or federal statute, but because we seek to capture a broader spectrum of hate, our numbers are likely to be very different than federal numbers. All figures used here are meant to describe what our database holds, with the understanding that subjectivity has been mitigated by operating with a team of analysts.
Motivation for incidents in the CAH database is determined at the point of intake of the form by the individual who witnessed or experienced a hate incident. Motivation is used to determine who was impacted by the incident. It may also be interpreted by the database manager or by the hotline manager, if not explicitly stated.
In the CAH database, more than one motivation can be selected for each entry. In these incidents, anti-Black and anti-Semitic motivations are chosen together. Rather than this indicating that Black Jewish people are being targeted, it is likely because some of the same alt-right language that was historically targeting Jewish populations during the Holocaust was adopted by alt-right hate groups in the modern and post-modern era. For example, although the swastika was historically used by Nazis during the Holocaust and associated with anti-Semitism, it is now used by many white supremacist groups targeting not just Jewish communities, but also African American and other minority communities. For most other motivations that were selected simultaneously in the CAH submissions, there are obvious correlations. Anti-Muslim, anti-Hispanic, anti-Asian, or anti-South Asian motivations were frequently selected alongside anti-immigrant. Often, these groups are targeted with language such as “Go back to your own country” or “This is my country” during verbal attacks. Many reports in the CAH database involving anti-Asian rhetoric involve the attacker using language insisting on the target’s foreignness. Similarly, 34 percent of Arab/Middle Eastern poll respondents stated they experienced or witnessed nativism or xenophobic attacks in the survey.
Location is determined at the point of intake of the CAH form by the individual who witnessed or experienced a hate incident. It may also be interpreted by the database manager or by the hotline manager, if not explicitly stated within the original report.
“Other” location is also possible and typically involves public locations not represented on the form, such as public parks, government buildings, or other public property. On some occasions, “Other” is selected if the location is unknown or happened in multiple places.
Incidents were coded as clean-up when they met the following definition: An incident in which community or friends and family support an individual though helping to clean-up after an incident has occurred. All hate incidents of this nature involved damage to property and/or written abusive language. Some of the incidents are less direct such as those that involved witnesses removing recruitment flyers that a hate group had placed on a campus or in a community.
Definitions not covered in Codebook
Convenience Sample
"Convenience sampling is a type of nonprobability sampling in which people are sampled simply because they are ‘convenient’ sources of data for researchers.” Encyclopedia of Survey Research Methods, Sage Publications
Alt-Right
The Alternative Right, commonly known as the "alt-right," is a set of far-right ideologies, groups and individuals whose core belief is that “white identity” is under attack by multicultural forces using “political correctness” and “social justice” to undermine white people and “their” civilization (Southern Poverty Law Center).
Hate Group
The Southern Poverty Law Center defines a hate group as an organization that – based on its official statements or principles, the statements of its leaders, or its activities – has beliefs or practices that attack or malign an entire class of people, typically for their immutable characteristics.
