<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
    <meta name="description" content="">
    <meta name="author" content="">

    <title>The Bias Report</title>

    <!-- Bootstrap core CSS -->
    <link href="../static/css/bootstrap.min.css" rel="stylesheet">

    <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
    <link href="../static/assets/css/ie10-viewport-bug-workaround.css" rel="stylesheet">

    <!-- Custom styles for this template -->
    <link href="../static/css/jumbotron-narrow.css" rel="stylesheet">

    <!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
    <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
    <script src="../static/assets/js/ie-emulation-modes-warning.js"></script>

    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.0/jquery.min.js"></script>
    <script src="../static/js/toggle_radio.js"></script>

    <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
    <!--[if lt IE 9]>
      <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
      <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
    <![endif]-->
  </head>


  <body>

    <div class="container">
      <div class="header clearfix">
        <nav>
          <ul class="nav nav-pills pull-right">
            <li role="presentation"><a href="{{ url_for('home')}}">Home</a></li>
            <li role="presentation" class="active"><a href="{{ url_for('about')}}">About</a></li>
          </ul>
        </nav>
        <h3 class="text-muted">The Bias Report</h3>
      </div>

      <div class="row marketing">
        <div class='col-sm-4'>
          <h1> The Bias Report in Action </h1>

         Using a clean version of the <a href="https://www.propublica.org/datastore/dataset/compas-recidivism-risk-score-data-and-analysis">COMPAS dataset</a>, we demostrate the use of The Bias Report web app.

         <h2>The Process</h2>

         <h3> Upload data </h3>

         First, we upload the data. The cleaned dataset is available on the upload page and follows the format described <a href="https://dssg.github.io/aequitas/input_data.html"> here</a>.

         <h3> Select Protected Groups</h3>

         Following the Propublica-Northpointe debate (discussed below) we focus on race. We select a custom reference group and use Caucasian as the reference group. Our metrics will thus reflect fairness in relation to the historically dominant group.

         <h3> Select Fairness Metrics</h3>

         Again following the below debate, we select False Positive Rates, False Negative Rates and False Discovery Rates.

         <h2> Background</h2>

         In 2016, Propublica <a href="https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing">reported on racial inequality</a> in  COMPAS, a risk assessment tool. They showed the algorithm led to unfair disparities in False Negative and False Positive Rates. In particular, they showed black defendants who would not go on to recidivate faced disproportionately high risk scores, while white defendants who would recidivate received disproportionately low risk scores. Northpointe, the company responsible for the algorithm, <a href="https://www.documentcloud.org/documents/2998e91-ProPublica-Commentary-Final-070616.html">responded</a> by arguing they callibrated the algorithm to be fair in terms of False Discovery Rate, also known as calibration. With the Bias Report, we get metrics on each type of disparity, adding clarity to the bias auditing process.

         <h2> Analysis </h2>
         The African-American false discovery rates are within the bounds of fairness. This result is expected because COMPAS is calibrated. (The overall FDR fairness returns false, because Asian and Native American defendants did not fall within the fairness threshholds for FDR).

         On the other hand, African-Americans are roughly twice as likely to have false positives and 40 percent less likely to false negatives. In real terms, 44.8% of African-Americans who did not recidivate were marked high or medium risk (with potential for associated penalties), compared with 23.4% of Caucasian non-reoffenders. This is unfair and is marked False below.

         These findings mark an inherent trade-off between FPR Fairness, FNR Fairness and calibration, which is present in any decision system where base rates are not equal. See Chouldechova (2017). Aequitas helps bring this trade-off to the forefront with clear metrics and asks system designers to make a reasoned decision based on their use case.
       </div>

       <div class='col-sm-8'>
         <iframe src="http://aequitas.dssg.io/audit/eg20cip9/compas_for_aequitas/report-1.html" width="100%" height="2000"></iframe>
         See the report in full screen <a href="http://aequitas.dssg.io/audit/eg20cip9/compas_for_aequitas/report-1.html">here</a>.
       </div>
     </div>
    </div> <!-- /container -->


    <!-- IE10 viewport hack for Surface/desktop Windows 8 bug -->
    <script src="../static/assets/js/ie10-viewport-bug-workaround.js"></script>
  </body>
