Encodings and Unicode
=====================

Character encodings remain a fractious and often exasperating part of IT.

If you are deal with more than ASCII characters with any regularity
whatsoever, for the love of God and all that is holy, use Python 3.
It has *greatly* superior support for Unicode characters, and will
generally make your life much eaiser.

``say()`` and ``fmt()`` try to avoid encoding gotchas by working with
Unicode strings.

In Python 3, all strings are Unicode strings, and all files and I/O streams are
inherently smart enough to read and write to reasonable encodings needed to
store Unicode on disk. But in Python 2, there is a choice between ``str`` and
``unicode``, and most files are *not* smart enough to use rational encodings.
Indeed, files that appear to have an ``encoding`` attribute will not let you set
that attribute, and they will not enforce that encoding when doing file IO.
*!@#$%^&!!!*

So if you must use Python 2:

* Seriously reconsider what led you to this point. Work to upgrade to modern
  Python (e.g. 3.7+) ASAP.

* Use ``unicode`` strings whenever possible.

* Seriously consider ``from __future__ import unicode_literals`` to automate
  the upleveling of string literals to Unicode.

* If you use the basie ``str`` type, include *only* ASCII characters, not
  encoded bytes from UTF-8 or whatever. If you don't do this, any trouble results
  be on your head.

* If ``say`` opens a file for you, it will do it with the ``codecs`` module
  with a default encoding of UTF-8. If you have ``say``
  write to a file that you open, you **must**  use
  ``codecs.open()``, ``io.open()``, or a similar mechanism that supports
  proper encoding. Else errors will result.

``say`` has a long history of trying to make Python 2 automatically "do the
right thing" even when basic Python 2 facilities do not. We have discovered,
like so many others before us, that was a fool's errand. Python 2 is simply
ill-prepared for day-in, day-out use of Unicode characters that are all around
us in the modern global world. While ``say`` continues some of this with respect
to the default standard output (``stdout``) stream, many of the previous
back-bends to support auto-encoding have been withdrawn. If you choose to use
Python 2, *you* are responsible for opening files in a responsible,
encoding-friendly way.::

    from __future__ import unicode_literals
    from codecs import open

    contents = u'Contains\u2012Unicode!'
    with open('outfile.txt', 'w', encoding='utf-8') as f:
        say(contents, file=f)
