[schooltool-dev] Fwd: About tests

[Ignas Mikalajunas]

Forwarding a very nice writeup of doctest types by Marius Gedminas.

Ignas

----- Forwarded message from Marius Gedminas <mgedmin at b4net.lt> -----

From: Marius Gedminas <mgedmin at b4net.lt>
To: zope3-dev at zope.org
Subject: Re: [Zope3-dev] Re: Windows eggs
Date: Sat, 14 Jul 2007 11:24:36 +0300
Mail-Followup-To: zope3-dev at zope.org
X-GPG-Fingerprint: 8121 AD32 F00A 8094 748A  6CD0 9157 445D E7A6 D78F
X-GPG-Key: http://gedmin.as/mg-pgp-key.txt
X-URL: http://gedmin/as/
X-Message-Flag: If you do not see this message correctly, stop using Outlook.

On Fri, Jul 13, 2007 at 11:09:30PM +0200, Wichert Akkerman wrote:
> Previously Tres Seaver wrote:
> > Stephan Richter wrote:
> > > I think in the long term it will be most beneficial, if we convert all tests
> > > to doctests; then a reasonable on-line documentation is not that hard to
> > > provide.
> >
> > - -1.  "Good tests" don't always make "good documentation";  I prefer the
> > isolation of traditional unittests for anything other than "main line"
> > use cases, for which doctests are best suited.
>
> Amen. I find failing doctests to be much harder to debug as well. I use
> doctests as a method to make sure examples in my documentation are
> correct, which is very useful.

Doctests are hard to get right.  I've finally settled on classifying my
tests into four categories and using different styles for them:

  - API documentation: human readability is the primary concern, doctests
    are in there just to make sure the documentation stays up to date.
    These are .txt files.

  - Short usage examples: sometimes it's simpler to show an example than
    to describe the function in words:

        def parse_date(date_string):
            """Parses an ISO-8601 date.

                >>> parse_date('2007-07-14')
                datetime.date(2007, 7, 14)

            """

    This is the only case when I allow doctests in code modules.
    Putting long test suites in class/function docstrings clutters the
    code and makes it harder to read.

  - Unit tests: there are many of those, they're independent (thus a
    single .txt for a collection of tests is a Bad Idea), they're short
    (so understanding and debugging is easy) and expressive.  I put
    those into .py files full with functions that look like

        def doctest_FooClass_does_this():
            """Test that FooClass is able to ...

                >>> foo = FooClass()
                >>> results = foo.do_this()
                >>> for fruit, score, comments in results:
                ...     print fruit.ljust(10), '|', score.ljust(5),
'|', comments
                Orange     | 9     | Tastes good, a bit sour
                Apple      | 8     | Varies

            """

    and have a traditional test_suite() function at the end that returns
    a DocTestSuite with the appropriate setUp/tearDown/optionflags.

    In effect this is a more or less direct translation of the
    traditional unittest tests.  I find that rewriting the assertions
    into doctest format helps me make the tests more readable.  Compare
    the above with

        class TestFooClass(unittest.TestCase):

            def test_does_this(self):
                foo = FooClass()
                results = foo.do_this()
                self.assertEquals(results,
                                  [('Orange', 9, 'Tastes good, a bit sour'),
                                   ('Apple', 8, 'Varies')])

    and especially compare the output you get when the test fails.

  - Functional tests: these are .txt files that use zope.testbrowser and
    are the hardest to debug.  There ought to be a better way to make
    assertions about HTML output than using ELLIPSIS and then pulling
    your hair out looking at huge and incomprehensible diffs.

        Digression: syntax highlighting the diffs helps immensely.
        Check out http://svn.zope.org/zope.testing/branches/colorized-output/

        Another digression: this is why I want Zope 3 to run on Python 2.5.
        I want to make XPath queries on the HTML, and I hope I'll be
        able to do that with cElementTree.

Marius Gedminas
--
Added mysterious, undocumented --scanflags and --fuzzy options.
        -- nmap 3.0 announcement



----- End forwarded message -----


Also check out https://storm.canonical.com/Tutorial for a very very good
example of an API documentation doctest.


Marius Gedminas
--
Windows 98: Not Plug & Play, but Bug & Pay!

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFGmIiQkVdEXeem148RAo+rAJ9NWePXgR2WqhL4E+XfbX51aIwbGACfahKr
8P9bpRQp0CuV6kqI2Lo9QNk=
=PSkv
-----END PGP SIGNATURE-----