Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for running tests in documentation #24

Open
14 tasks
oubiwann opened this issue Jan 22, 2015 · 1 comment
Open
14 tasks

Add support for running tests in documentation #24

oubiwann opened this issue Jan 22, 2015 · 1 comment

Comments

@oubiwann
Copy link
Member

One of the classic problems with putting code in documentation is that it, sooner or later, begins to bit rot. If code in documentation can be run as part of a test suite (e.g., prior to commits or as a gate in a continuous integration platform), then this can be checked (and hopefully prevented).

The Python programming language pioneered this with doctests. Though sometimes hailed as a great abuse of testing (which it can be), when used appropriately (e.g., ensuring that code in documentation continues to be checked as working) it can be a great boon to software projects which strive to provide good example usage of APIs, etc., in documentation.

I suspect this will actually be comprised of a sizable number of other tickets. The following tasks will likely need to be completed to make this a reality:

  • Parse Markdown files for code blocks tagged with "cl" or "lisp"
    • possibly add doctest metadata and/or support for doctest directives in the first line of the code block
  • Parse commented code in LFE modules - this will definitely need some sort of metadata marker indicating that the code to follow should be parsed by ltest's doctest mechanism
  • Parse docstrings in LFE functions - I have doubts about this one; one can only put so much code in a docstring before it becomes awkward to read (much more to parse and execute as a test ...)
  • Execute parsed code, line-by-line
  • Ensure that executed code returns the same result as provided in the parsed code example/documentation
  • Add testing infrastructure
    • Add a behaviour for "doctest"
    • Add a (defdoctest ...) macro
    • Add supporting helper functions for this macros which:
      • Add support for running all code in provided Markdown files (code that is marked as "doctest", that is)
      • Add support for running all code in the code comments of provided (or discovered) files
      • Add support for running all code in the docstrings of provided (or discovered) files (again, not sure about this one)
  • Update test runner to include these types of test "suites"
@oubiwann
Copy link
Member Author

Note that, as the number of Markdown-based docs continue to grow (e.g., the gitbook books), this -- or something like it -- becomes increasingly important.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant