{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating and grading assignments"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. contents:: Table of Contents\n",
    "   :depth: 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Developing assignments with the assignment toolbar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: As you are developing your assignments, you should save them\n",
    "into the `source/{assignment_id}/` folder of the nbgrader hierarchy,\n",
    "where `assignment_id` is the name of the assignment you are creating\n",
    "(e.g. \"ps1\")."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. seealso::\n",
    "\n",
    "    :doc:`philosophy`\n",
    "        More details on how the nbgrader hierarchy is structured.\n",
    "        \n",
    "    :doc:`/configuration/student_version`\n",
    "        Instructions for customizing how the student version of the assignment looks.\n",
    "\n",
    "Before you can begin developing assignments, you will need to actually\n",
    "install the nbgrader toolbar. If you do not have it installed, please\n",
    "first follow the :doc:`installation instructions <installation>`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the toolbar has been installed, you should see it in the drop down \"Cell toolbar\" menu:\n",
    "\n",
    "![](images/assignment_toolbar.png)\n",
    "\n",
    "Selecting the \"Create Assignment\" toolbar will create a separate toolbar\n",
    "for each cell which by default will be a dropdown menu with the \"-\" item\n",
    "selected. For markdown cells, there are two additional options to choose\n",
    "from, either \"Manually graded answer\" or \"Read-only\":\n",
    "\n",
    "![](images/markdown_cell.png)\n",
    "\n",
    "For code cells, there are four options to choose from, including\n",
    "\"Manually graded answer\", \"Autograded answer\", \"Autograder tests\", and\n",
    "\"Read-only\":\n",
    "\n",
    "![](images/code_cell.png)\n",
    "\n",
    "The following sections go into detail about the different cell types,\n",
    "and show cells that are taken from a complete example of an assignment\n",
    "generated with the nbgrader toolbar extension:\n",
    "\n",
    "-   [source/ps1/problem1.ipynb](source/ps1/problem1.html)\n",
    "-   [source/ps1/problem2.ipynb](source/ps1/problem2.html)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. _manually-graded-cells:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"Manually graded answer\" cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you select the \"Manually graded answer\" option (available for both\n",
    "markdown and code cells), the nbgrader extension will mark that cell as\n",
    "a cell that contains an answer that must be manually graded by a human\n",
    "grader. Here is an example of a manually graded answer cell:\n",
    "\n",
    "![](images/manually_graded_answer.png)\n",
    "\n",
    "The most common use case for this type of cell is for written\n",
    "free-response answers (for example, which interpret the results of code\n",
    "that may have been written and/or executed above)."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "When you specify a manually graded answer, you must additionally tell nbgrader how many points the answer is worth, and an id for the cell. Additionally, when creating the release version of the assignment (see :ref:`assign-and-release-an-assignment`), the bodies of answer cells will be replaced with a code or text stub indicating to the students that they should put their answer or solution there. Please see :doc:`/configuration/student_version` for details on how to customize this behavior."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: the blue border only shows up when the nbgrader extension toolbar\n",
    "is active; it will not be visible to students.*"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. _autograded-answer-cells:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"Autograded answer\" cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you select the \"Autograded answer\" option (available only for code\n",
    "cells), the nbgrader extension will mark that cell as a cell that\n",
    "contains an answer which will be autograded. Here is an example of an\n",
    "autograded graded answer cell:\n",
    "\n",
    "![](images/autograded_answer.png)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "As shown in the image above, solutions can be specified inline, through the use of a special syntax such as ``### BEGIN SOLUTION`` and ``### END SOLUTION``. When creating the release version (see :ref:`assign-and-release-an-assignment`), the region between the special syntax lines will be replaced with a code stub. If this special syntax is not used, then the entire contents of the cell will be replaced with the code stub. Please see :doc:`/configuration/student_version` for details on how to customize this behavior."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike manually graded answers, autograded answers aren't worth any\n",
    "points: instead, the points for autograded answers are specified for the\n",
    "particular tests that grade those answers. See the next section for\n",
    "further details.\n",
    "\n",
    "*Note: the blue border only shows up when the nbgrader extension toolbar\n",
    "is active; it will not be visible to students.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"Autograder tests\" cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you select the \"Autograder tests\" option (available only for code\n",
    "cells), the nbgrader extension will mark that cell as a cell that\n",
    "contains tests to be run during autograding. Here is an example of two\n",
    "test cells:\n",
    "\n",
    "![](images/autograder_tests.png)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Test cells should contain ``assert`` statements (or similar). When run through ``nbgrader autograde`` (see :ref:`autograde-assignments`), the cell will pass if no errors are raised, and fail otherwise. You must specify the number of points that each test cell is worth; then, if the tests pass during autograding, students will receive the specified number of points, and otherwise will receive zero points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lock icon on the left side of the cell toolbar indicates that the\n",
    "tests are \"read-only\". See the next section for further details on what\n",
    "this means."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "For tips on writing autograder tests, see :ref:`autograding-resources`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: the blue border only shows up when the nbgrader extension toolbar\n",
    "is active; it will not be visible to students.*"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. _read-only-cells:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"Read-only\" cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you select the \"Read-only\" option (available for both code and\n",
    "markdown cells), the nbgrader extension will mark that cell as one that\n",
    "cannot be modified. This is indicated by a lock icon on the left side of\n",
    "the cell toolbar:\n",
    "\n",
    "![](images/read_only.png)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "However, this doesn't actually mean that it is truly read-only when opened in the notebook. Instead, what is means is that during the ``nbgrader assign`` step (see :ref:`assign-and-release-an-assignment`), the source of these cells will be recorded into the database. Then, during the ``nbgrader autograde`` step (see :ref:`autograde-assignments`), nbgrader will check whether the source of the student's version of the cell has changed. If it has, it will replace the cell's source with the version in the database, thus effectively overwriting any changes the student made.\n",
    "\n",
    ".. versionadded:: 0.4.0\n",
    "    Read-only cells (and test cells) are now truly read-only! However, at the moment this functionality will only work on the master version of the notebook (5.0.0.dev)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This functionality is particularly important for test cells, which are\n",
    "always marked as read-only. Because the mechanism for autograding is\n",
    "that students receive full credit if the tests pass, an easy way to get\n",
    "around this would be to simply delete or comment out the tests. This\n",
    "read-only functionality will reverse any such changes made by the\n",
    "student."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. _assign-and-release-an-assignment:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assign and release an assignment"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. seealso::\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-assign`\n",
    "        Command line options for ``nbgrader assign``\n",
    "        \n",
    "    :doc:`philosophy`\n",
    "        Details about how the directory hierarchy is structured\n",
    "\n",
    "    :doc:`/configuration/config_options`\n",
    "        Details on ``nbgrader_config.py``"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "After an assignment has been created with the assignment toolbar, you will want to create a release version of the assignment for the students.\n",
    "\n",
    "As described in :doc:`philosophy`, you need to organize your files in a particular way. For releasing assignments, you should have the master copy of your files saved (by default) in the following source directory structure:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "{course_directory}/source/{assignment_id}/{notebook_id}.ipynb\n",
    "```\n",
    "\n",
    "Note: The `student_id` is not included here because the source and release versions of the assignment are the same for all students.\n",
    "\n",
    "After running `nbgrader assign`, the release version of the notebooks will be:\n",
    "\n",
    "```\n",
    "{course_directory}/release/{assignment_id}/{notebook_id}.ipynb\n",
    "```\n",
    "\n",
    "As a reminder, the instructor is responsible for distributing this release version to their students using their institution's existing student communication and document distribution infrastructure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Workflow example: Instructor generating an assignment release"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example walks an instructor through the workflow for generating an assignment and preparing it for release to students:\n",
    "\n",
    "1. Add problem set to the source folder\n",
    "2. Verify that the tests pass in the instructor version\n",
    "3. Configure assignments and students in the config file\n",
    "4. Assign a problem set using nbgrader\n",
    "5. Inspect the release folder\n",
    "6. Verify that the tests fail in the student version\n",
    "7. Release files to students"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Add problem set to the source folder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify this example, two notebooks of the assignment have already been stored in the `source/ps1` folder:\n",
    "\n",
    "* [source/ps1/problem1.ipynb](source/ps1/problem1.ipynb)\n",
    "* [source/ps1/problem2.ipynb](source/ps2/problem2.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Verify that the tests pass in the instructor version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ideally, the solutions in the instructor version should be correct and pass all the test cases to ensure that you are giving your students tests that they can actually pass. To verify this is the case, run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success! Your notebook passes all the tests.\n",
      "Success! Your notebook passes all the tests.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "nbgrader validate source/ps1/*.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the notebook passes all the test cases, you should see the message \"Success! Your notebook passes all the tests.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Configure assignments and students in the config file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a `nbgrader_config.py` file and configuring the list of your assignments is the first workflow step to create a release version of an assignment. These assignments will be automatically added to the sqlite database `gradebook.db`. If this database does not already exist, nbgrader will automatically create it. By default, the nbgrader commands (like `nbgrader assign`) will assume that this database is called `gradebook.db` and that it lives in the root of your course directory, though its name and path can be configured in `nbgrader_config.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing nbgrader_config.py\n"
     ]
    }
   ],
   "source": [
    "%%file nbgrader_config.py\n",
    "\n",
    "c = get_config()\n",
    "c.NbGrader.db_assignments = [dict(name=\"ps1\", duedate=\"2015-02-02 17:00:00 UTC\")]\n",
    "c.NbGrader.db_students = [\n",
    "    dict(id=\"bitdiddle\", first_name=\"Ben\", last_name=\"Bitdiddle\"),\n",
    "    dict(id=\"hacker\", first_name=\"Alyssa\", last_name=\"Hacker\")\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also add students and assignments to the database using the `nbgrader db` command line tool:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[DbStudentAddApp | INFO] Creating/updating student with ID 'reasoner': {'last_name': 'Reasoner', 'email': None, 'first_name': 'Louis'}\n",
      "[DbAssignmentAddApp | INFO] Creating/updating assignment with ID 'ps2': {'duedate': None}\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "nbgrader db student add reasoner --last-name=Reasoner --first-name=Louis\n",
    "nbgrader db assignment add ps2"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "See the documentation on :doc:`managing_the_database` for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.  Assign a problem set using nbgrader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the gradebook is set up, run `nbgrader assign`. When running `nbgrader assign`, the assignment name (which is \"ps1\") is passed. We also specify a *header* notebook (`source/header.ipynb`) to prepend at the beginning of each notebook in the assignment. By default, this command should be run from the root of the course directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[AssignApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/source/./ps1/jupyter.png -> /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/release/./ps1/jupyter.png\n",
      "[AssignApp | INFO] Updating/creating assignment 'ps1': {'duedate': '2015-02-02 17:00:00 UTC'}\n",
      "[AssignApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/source/./ps1/problem1.ipynb to notebook\n",
      "[AssignApp | INFO] Writing 8034 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/release/./ps1/problem1.ipynb\n",
      "[AssignApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/source/./ps1/problem2.ipynb to notebook\n",
      "[AssignApp | INFO] Writing 2187 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/release/./ps1/problem2.ipynb\n",
      "[AssignApp | INFO] Setting destination file permissions to 644\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "nbgrader assign \"ps1\" --IncludeHeaderFooter.header=source/header.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.  Inspect the release folder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upon completion of `nbgrader assign` in the previous step, there will be a new folder called `release` with the same structure as `source`. The `release` folder contains the actual release version of the assignment files:\n",
    "\n",
    "* [release/ps1/problem1.ipynb](release/ps1/problem1.ipynb)\n",
    "* [release/ps1/problem2.ipynb](release/ps1/problem2.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Verify that the tests fail in the student version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ideally, all the tests should fail in the student version if the student hasn't implemented anything. To verify that this is in fact the case, we can use the `nbgrader validate --invert` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success! The notebook does not pass any tests.\n",
      "Success! The notebook does not pass any tests.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "nbgrader validate --invert release/ps1/*.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the notebook fails all the test cases, you should see the message \"Success! The notebook does not pass any tests.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Release files to students"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "At this point you will be able to take the files in the ``release`` folder and distribute them to students. If you are using nbgrader in a shared server environment, you can do this with the ``nbgrader release`` command (see :doc:`managing_assignment_files`). Otherwise, you will need to do this manually:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please note**: Students must use version 3 or greater of the IPython/Jupyter notebook for nbgrader to work properly. If they are not using version 3 or greater, it is possible for them to delete cells that contain important metadata for nbgrader. With version 3 or greater, there is a feature in the notebook that prevents cells from being deleted. See [this issue](https://github.com/jupyter/nbgrader/issues/424) for more details.\n",
    "\n",
    "To ensure that students have a recent enough version of the notebook, you can include a cell such as the following in each notebook of the assignment:\n",
    "\n",
    "```python\n",
    "import IPython\n",
    "assert IPython.version_info[0] >= 3, \"Your version of IPython is too old, please update it.\"\n",
    "```"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. _autograde-assignments:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Autograde assignments"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. seealso::\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-autograde`\n",
    "        Command line options for ``nbgrader autograde``\n",
    "        \n",
    "    :doc:`philosophy`\n",
    "        Details about how the directory hierarchy is structured\n",
    "\n",
    "    :doc:`/configuration/config_options`\n",
    "        Details on ``nbgrader_config.py``"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "After assignments have been submitted by students, you will want to save them into a ``submitted`` directory. As described in :doc:`philosophy`, you need to organize your files in a particular way. For autograding assignments, you should have the submitted versions of students' assignments organized as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "submitted/{student_id}/{assignment_id}/{notebook_id}.ipynb\n",
    "```\n",
    "\n",
    "After running `nbgrader autograde`, the autograded version of the notebooks will be:\n",
    "\n",
    "```\n",
    "autograded/{student_id}/{assignment_id}/{notebook_id}.ipynb\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Workflow example: Instructor autograding assignments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example, we have an assignment with two notebooks. There are two submissions of the assignment:\n",
    "\n",
    "Submission 1:\n",
    "\n",
    "* [submitted/bitdiddle/ps1/problem1.ipynb](submitted/bitdiddle/ps1/problem1.ipynb)\n",
    "* [submitted/bitdiddle/ps1/problem2.ipynb](submitted/bitdiddle/ps1/problem2.ipynb)\n",
    "\n",
    "Submission 2:\n",
    "\n",
    "* [submitted/hacker/ps1/problem1.ipynb](submitted/hacker/ps1/problem1.ipynb)\n",
    "* [submitted/hacker/ps1/problem2.ipynb](submitted/hacker/ps1/problem2.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can actually start grading, we need to actually specify who the students are. We did this earlier in the `nbgrader_config.py` file, where we also specified the list of assignments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "c = get_config()\r\n",
      "c.NbGrader.db_assignments = [dict(name=\"ps1\", duedate=\"2015-02-02 17:00:00 UTC\")]\r\n",
      "c.NbGrader.db_students = [\r\n",
      "    dict(id=\"bitdiddle\", first_name=\"Ben\", last_name=\"Bitdiddle\"),\r\n",
      "    dict(id=\"hacker\", first_name=\"Alyssa\", last_name=\"Hacker\")\r\n",
      "]"
     ]
    }
   ],
   "source": [
    "!cat nbgrader_config.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the config file has been set up with the students, we can run the autograder (and as with the other nbgrader commands for instructors, this must be run from the root of the course directory):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/bitdiddle/ps1/jupyter.png -> /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/jupyter.png\n",
      "[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/bitdiddle/ps1/timestamp.txt -> /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/timestamp.txt\n",
      "[AutogradeApp | INFO] Creating/updating student with ID 'bitdiddle': {'first_name': 'Ben', 'last_name': 'Bitdiddle'}\n",
      "[AutogradeApp | INFO] SubmittedAssignment<ps1 for bitdiddle> submitted at 2015-02-02 22:58:23.948203\n",
      "[AutogradeApp | WARNING] SubmittedAssignment<ps1 for bitdiddle> is 21503.948203 seconds late\n",
      "[AutogradeApp | INFO] Overwriting files with master versions from the source directory\n",
      "[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/source/./ps1/jupyter.png -> /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/jupyter.png\n",
      "[AutogradeApp | INFO] Sanitizing /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/bitdiddle/ps1/problem1.ipynb\n",
      "[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/bitdiddle/ps1/problem1.ipynb to notebook\n",
      "[AutogradeApp | WARNING] Attribute 'checksum' for cell correct_squares has changed! (should be: 8f41dd0f9c8fd2da8e8708d73e506b3a, got: 845d4666cabb30b6c75fc534f7375bf5)\n",
      "[AutogradeApp | WARNING] Attribute 'checksum' for cell squares_invalid_input has changed! (should be: 23c2b667d3b60eff3be46eb3290a6b4a, got: 123394e73f33a622ec057e2eae51a54a)\n",
      "[AutogradeApp | INFO] Writing 8093 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/problem1.ipynb\n",
      "[AutogradeApp | INFO] Autograding /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/problem1.ipynb\n",
      "[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/problem1.ipynb to notebook\n",
      "[AutogradeApp | INFO] Executing notebook with kernel: python\n",
      "[AutogradeApp | WARNING] SubmittedAssignment<ps1 for bitdiddle> is 21503.948203 seconds late\n",
      "[AutogradeApp | INFO] Using late submission penalty method: none\n",
      "[AutogradeApp | WARNING] Late submission penalty: None\n",
      "[AutogradeApp | INFO] Writing 13509 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/problem1.ipynb\n",
      "[AutogradeApp | INFO] Sanitizing /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/bitdiddle/ps1/problem2.ipynb\n",
      "[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/bitdiddle/ps1/problem2.ipynb to notebook\n",
      "[AutogradeApp | INFO] Writing 2228 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/problem2.ipynb\n",
      "[AutogradeApp | INFO] Autograding /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/problem2.ipynb\n",
      "[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/problem2.ipynb to notebook\n",
      "[AutogradeApp | INFO] Executing notebook with kernel: python\n",
      "[AutogradeApp | WARNING] SubmittedAssignment<ps1 for bitdiddle> is 21503.948203 seconds late\n",
      "[AutogradeApp | INFO] Using late submission penalty method: none\n",
      "[AutogradeApp | WARNING] Late submission penalty: None\n",
      "[AutogradeApp | INFO] Writing 2225 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/problem2.ipynb\n",
      "[AutogradeApp | INFO] Setting destination file permissions to 444\n",
      "[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/hacker/ps1/jupyter.png -> /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/jupyter.png\n",
      "[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/hacker/ps1/timestamp.txt -> /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/timestamp.txt\n",
      "[AutogradeApp | INFO] Creating/updating student with ID 'hacker': {'first_name': 'Alyssa', 'last_name': 'Hacker'}\n",
      "[AutogradeApp | INFO] SubmittedAssignment<ps1 for hacker> submitted at 2015-02-01 17:28:58.749302\n",
      "[AutogradeApp | INFO] Overwriting files with master versions from the source directory\n",
      "[AutogradeApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/source/./ps1/jupyter.png -> /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/jupyter.png\n",
      "[AutogradeApp | INFO] Sanitizing /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/hacker/ps1/problem1.ipynb\n",
      "[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/hacker/ps1/problem1.ipynb to notebook\n",
      "[AutogradeApp | INFO] Writing 8830 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/problem1.ipynb\n",
      "[AutogradeApp | INFO] Autograding /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/problem1.ipynb\n",
      "[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/problem1.ipynb to notebook\n",
      "[AutogradeApp | INFO] Executing notebook with kernel: python\n",
      "[AutogradeApp | INFO] Writing 9460 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/problem1.ipynb\n",
      "[AutogradeApp | INFO] Sanitizing /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/hacker/ps1/problem2.ipynb\n",
      "[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/submitted/hacker/ps1/problem2.ipynb to notebook\n",
      "[AutogradeApp | INFO] Writing 2358 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/problem2.ipynb\n",
      "[AutogradeApp | INFO] Autograding /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/problem2.ipynb\n",
      "[AutogradeApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/problem2.ipynb to notebook\n",
      "[AutogradeApp | INFO] Executing notebook with kernel: python\n",
      "[AutogradeApp | INFO] Writing 2355 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/problem2.ipynb\n",
      "[AutogradeApp | INFO] Setting destination file permissions to 444\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "nbgrader autograde \"ps1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When grading the submission for `Bitdiddle`, you'll see some warnings that look like \"Checksum for grade cell correct_squares has changed!\". What's happening here is that nbgrader has recorded what the *original* contents of the grade cell `correct_squares` (when `nbgrader assign` was run), and is checking the submitted version against this original version. It has found that the submitted version changed (perhaps this student tried to cheat by commenting out the failing tests), and has therefore overwritten the submitted version of the tests with the original version of the tests.\n",
    "\n",
    "You may also notice that there is a note saying \"ps1 for Bitdiddle is 21503.948203 seconds late\". What is happening here is that nbgrader is detecting a file in Bitdiddle's submission called `timestamp.txt`, reading in that timestamp, and saving it into the database. From there, it can compare the timestamp to the duedate of the problem set, and compute whether the submission is at all late.\n",
    "\n",
    "Once the autograding is complete, there will be new directories for the autograded versions of the submissions:\n",
    "\n",
    "Autograded submission 1:\n",
    "\n",
    "* [autograded/bitdiddle/ps1/problem1.ipynb](autograded/bitdiddle/ps1/problem1.ipynb)\n",
    "* [autograded/bitdiddle/ps1/problem2.ipynb](autograded/bitdiddle/ps1/problem2.ipynb)\n",
    "\n",
    "Autograded submission 2:\n",
    "\n",
    "* [autograded/hacker/ps1/problem1.ipynb](autograded/hacker/ps1/problem1.ipynb)\n",
    "* [autograded/hacker/ps1/problem2.ipynb](autograded/hacker/ps1/problem2.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manual grading and formgrade"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. seealso::\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-formgrade`\n",
    "        Command line options for ``nbgrader formgrade``\n",
    "\n",
    "    :doc:`philosophy`\n",
    "        More details on how the nbgrader hierarchy is structured.\n",
    "\n",
    "    :doc:`/configuration/config_options`\n",
    "        Details on ``nbgrader_config.py``"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "After assignments have been autograded, they will saved into an ``autograded`` directory (see :doc:`philosophy` for details):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running `nbgrader autograde`, the autograded version of the\n",
    "notebooks will be:\n",
    "\n",
    "    autograded/{student_id}/{assignment_id}/{notebook_id}.ipynb\n",
    "\n",
    "To grade the assignments with an HTML form, all we have to do is run:\n",
    "\n",
    "```bash\n",
    "nbgrader formgrade\n",
    "```"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "This will launch a server at ``http://localhost:5000`` that will provide you with an interface for hand grading assignments that it finds in the directory listed above. Note that this applies to *all* assignments as well -- as long as the autograder has been run on the assignment (see :ref:`autograde-assignments`), it will be available for manual grading via the formgrader."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The formgrader doesn't actually modify the files on disk at all; it only\n",
    "modifies information about them in the database. So, there is no\n",
    "\"output\" step for the formgrader."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Important note: if you run the formgrader on a public IP address and port (for example, by running ``nbgrader formgrade --ip=<public_ip>``), then *anyone* will be able to access the formgrader at its URL! So, you probably want to only run the formgrader on a private network that only the instructors have access to (e.g., behind a VPN). Alternately, if students are accessing their notebooks via JupyterHub, then you can configure the formgrader to authenticate users through JupyterHub so that only specified graders may access the formgrader. See :doc:`/configuration/jupyterhub_config` for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feedback on assignments"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. seealso::\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-feedback`\n",
    "        Command line options for ``nbgrader feedback``\n",
    "        \n",
    "    :doc:`philosophy`\n",
    "        Details about how the directory hierarchy is structured\n",
    "\n",
    "    :doc:`/configuration/config_options`\n",
    "        Details on ``nbgrader_config.py``"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "After assignments have been autograded and/or manually graded, they will located in the `autograded` directory (see :doc:`philosophy` for details):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "autograded/{student_id}/{assignment_id}/{notebook_id}.ipynb\n",
    "```\n",
    "\n",
    "After running `nbgrader feedback`, HTML versions of these notebooks will be saved to:\n",
    "\n",
    "```\n",
    "feedback/{student_id}/{assignment_id}/{notebook_id}.html\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Workflow example: Instructor returning feedback to students"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example, we have an assignment with two notebooks. There are two submissions of the assignment that have been graded:\n",
    "\n",
    "Autograded submission 1:\n",
    "\n",
    "* [autograded/bitdiddle/ps1/problem1.ipynb](autograded/bitdiddle/ps1/problem1.ipynb)\n",
    "* [autograded/bitdiddle/ps1/problem2.ipynb](autograded/bitdiddle/ps1/problem2.ipynb)\n",
    "\n",
    "Autograded submission 2:\n",
    "\n",
    "* [autograded/hacker/ps1/problem1.ipynb](autograded/hacker/ps1/problem1.ipynb)\n",
    "* [autograded/hacker/ps1/problem2.ipynb](autograded/hacker/ps1/problem2.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generating feedback is fairly straightforward (and as with the other nbgrader commands for instructors, this must be run from the root of the course directory):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[FeedbackApp | WARNING] Config option `display_data_priority` not recognized by `GetGrades`.\n",
      "[FeedbackApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/jupyter.png -> /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/feedback/bitdiddle/ps1/jupyter.png\n",
      "[FeedbackApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/timestamp.txt -> /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/feedback/bitdiddle/ps1/timestamp.txt\n",
      "[FeedbackApp | WARNING] Config option `display_data_priority` not recognized by `GetGrades`.\n",
      "[FeedbackApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/problem1.ipynb to html\n",
      "[FeedbackApp | INFO] Writing 274746 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/feedback/bitdiddle/ps1/problem1.html\n",
      "[FeedbackApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/bitdiddle/ps1/problem2.ipynb to html\n",
      "[FeedbackApp | INFO] Writing 253697 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/feedback/bitdiddle/ps1/problem2.html\n",
      "[FeedbackApp | INFO] Setting destination file permissions to 444\n",
      "[FeedbackApp | WARNING] Config option `display_data_priority` not recognized by `GetGrades`.\n",
      "[FeedbackApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/jupyter.png -> /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/feedback/hacker/ps1/jupyter.png\n",
      "[FeedbackApp | INFO] Copying /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/timestamp.txt -> /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/feedback/hacker/ps1/timestamp.txt\n",
      "[FeedbackApp | WARNING] Config option `display_data_priority` not recognized by `GetGrades`.\n",
      "[FeedbackApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/problem1.ipynb to html\n",
      "[FeedbackApp | INFO] Writing 271053 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/feedback/hacker/ps1/problem1.html\n",
      "[FeedbackApp | INFO] Converting notebook /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/autograded/hacker/ps1/problem2.ipynb to html\n",
      "[FeedbackApp | INFO] Writing 253827 bytes to /Users/jhamrick/project/tools/nbgrader/nbgrader/docs/source/user_guide/feedback/hacker/ps1/problem2.html\n",
      "[FeedbackApp | INFO] Setting destination file permissions to 444\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "nbgrader feedback \"ps1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the feedback has been generated, there will be new directories and HTML files corresponding to each notebook in each submission:\n",
    "\n",
    "Feedback for submission 1:\n",
    "\n",
    "* [feedback/bitdiddle/ps1/problem1.html](feedback/bitdiddle/ps1/problem1.html)\n",
    "* [feedback/bitdiddle/ps1/problem2.html](feedback/bitdiddle/ps1/problem2.html)\n",
    "\n",
    "Feedback for submission 2:\n",
    "\n",
    "* [feedback/hacker/ps1/problem1.html](feedback/hacker/ps1/problem1.html)\n",
    "* [feedback/hacker/ps1/problem2.html](feedback/hacker/ps1/problem2.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting grades from the database"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. versionadded:: 0.4.0\n",
    "\n",
    ".. seealso::\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-export`\n",
    "        Command line options for ``nbgrader export``\n",
    "        \n",
    "    :doc:`/plugins/export-plugin`\n",
    "        Details on how to write your own custom exporter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to creating feedback for the students, you may need to upload grades to whatever learning management system your school uses (e.g. Canvas, Blackboard, etc.). nbgrader provides a way to export grades to CSV out of the box, with the `nbgrader export` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ExportApp | INFO] Using exporter: CsvExportPlugin\n",
      "[ExportApp | INFO] Exporting grades to grades.csv\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "nbgrader export"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running `nbgrader export`, you will see the grades in a CSV file called `grades.csv`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "assignment,duedate,timestamp,student_id,last_name,first_name,email,raw_score,late_submission_penalty,score,max_score\n",
      "ps2,,,bitdiddle,Bitdiddle,Ben,,0.0,0.0,0.0,0.0\n",
      "ps2,,,hacker,Hacker,Alyssa,,0.0,0.0,0.0,0.0\n",
      "ps2,,,reasoner,Reasoner,Louis,,0.0,0.0,0.0,0.0\n",
      "ps1,2015-02-02 17:00:00,2015-02-02 22:58:23.948203,bitdiddle,Bitdiddle,Ben,,1.5,0.0,1.5,9.0\n",
      "ps1,2015-02-02 17:00:00,2015-02-01 17:28:58.749302,hacker,Hacker,Alyssa,,3.0,0.0,3.0,9.0\n",
      "ps1,2015-02-02 17:00:00,,reasoner,Reasoner,Louis,,0.0,0.0,0.0,9.0\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cat grades.csv"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "If you need to customize how the grades are exported, you can :doc:`write your own exporter </plugins/export-plugin>`."
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 0
}
