RightsCon Toronto 2018

Automated decisions increasingly mediate civic life. Governments use algorithms to screen immigrants and allocate social services. Corporations rely on software to help in making decisions in vital areas like hiring, credit scoring, and political discourse. There is a growing desire to “open the black box” of complex algorithms and hold the institutions using them accountable. But across the globe, civil society faces a range of challenges as they pursue these goals. Testing the societal impact of algorithms from the outside is one of the biggest challenges for advocacy organizations: A critical amount of data may be needed and privacy provisions must be accounted for. Analysis can be time-consuming and expensive. Additionally, the potential of reidentification and different data formats can make it difficult for researchers to share results and effectively collaborate. Nevertheless, many researchers, journalists, and public interest groups have been able to make real progress in understanding these systems.

In this session, we’ll explore what it means to scrutinize automated systems in the real world, using both technical and non-technical methods. We’ll look at case studies where public actors were able to discover meaningful insights about an automated decision — without having access to the underlying code. And we’ll get a flavor for the many different ways to "hold algorithms accountable." The moderator will first share an overview of findings from research about different methods of scrutiny — both technical and nontechnical — and invited session participants will then describe the work they each did to investigate an automated system. One case study presented will be AlgorithmWatch's “openSCHUFA” project, investigating Germany’s leading credit scoring company. A low SCHUFA score means banks will reject your credit card application, car rentals will reject you as a customer and network providers will say ‘no’ to a new Internet contract. No one knows how accurate SCHUFA’s data is and how it computes its scores, and OpenSCHUFA wants to change this by analyzing thousands of credit records. The project has successfully crowdfunded more than 43,000 Euros and almost 18,000 people have requested their score.

Session leaders will then encourage attendees to share their own work investigating algorithms to promote an understanding that there are many ways to gain more insight into automated decision systems without having access to or being able to understand code. Participants will then have the opportunity to brainstorm other systems worth scrutinizing and how to do so.