Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for class methods documentation #9751

Merged
merged 4 commits into from
Apr 2, 2018
Merged

Add support for class methods documentation #9751

merged 4 commits into from
Apr 2, 2018

Conversation

StefanoCappellini
Copy link
Contributor

@StefanoCappellini StefanoCappellini commented Mar 25, 2018

I noticed that the recent commits related to the doc (the ones that try to automate the doc generation, for example #9656 ) break the old doc style. For example, the new ImageDataGenerator page does not document the ImageDataGenerator methods (fit, flow, ...) anymore, we only have some examples.
This PR adds:

  • Support for automatic generation of class methods: the classes marked as "classes_with_methods" in the autogen.py file will now have all their methods (obviously not the private or the excluded ones) fully documented
  • Some style enrichments in order to make the doc clearer:
    • The documented class methods will have a "method" prepended to their name: this helps distinguish between a method and a module function .
    • All the classes will have a "class" prepended to their name. This names are added using JS and so they will not pollute the menu bar.

I tried to follow a style similar to the one already adopted by the TensorFlow's docs.
In particular:

  • I decided to add an additional custom label, "classes_with_methods" in order to be able to decided for which class generate the inner methods docs. At them moment, only the ImageDataGenerator class uses it
  • I decided to add the "class" prefix to the class names in order to make easier to distinguish them from simple and plain methods
  • The solution makes uses of some tricks in order to add the style. I decided to go for them instead of using some extensions (like for example pymdownx.extra) that would have make the generation more cumbersome.

All these decisions can obviously be changed and improved but I think PR could serve as a good base for future improvements. I have tested the docs on a mac and it is rendered wonderfully.

@Dref360 Dref360 self-requested a review March 25, 2018 20:45
@fchollet
Copy link
Member

This appears to have been overlooked in the recent changes to the preprocessing docs, but we already have a way to document class methods, which we use for instance for here. It involves listing explicitly all methods that you want to generate docs for, in autogen.py.

In order to achieve a mechanism where we automatically document a class by printing 1) the main docstring, 2) individual methods, under a "methods" header, we could reuse much of the same code.

@StefanoCappellini
Copy link
Contributor Author

@fchollet Thank you for your feedback. As regards the code reuse, yes, I totally agree. However let me explain:

  1. I was thinking about completely automated doc generation (so not partial template filling)
  2. When you have a page like the one you mentioned (you have only methods from a single class and no functions), yes, I completely agree with you: you can obtain the same effect almost effortless
  3. However, you can see some problems when you start having pages, like the "Image Preprocessing" and the "Text Preprocessing" pages, where you have both methods from a single class and functions. Here the process start to become slightly cumbersome:
    • First of all, without any additional style enhancement I think it would be slightly hard to visually distinguish one from the other (yes, functions could have the module name prepended [now not added], but one should check). In particular I find counter-intuitive the idea of having a separating line between the class and its methods. In the old doc we used to have clear dotted list.
    • We could prepend the class name to the methods but then they could be confused with static methods.
    • Both the methods and the functions would be listed in the side menu, without a clear way to distinguish them (this does not happened in the old doc)
  • We should enforce a particular order: first the class description, then its methods and then the functions. This could be obtained by sorting all the functions appropriately but I find it pretty error-prone. Yes, you could probably obtain this effect by playing with the "all_module_functions" and "function" parts...
  1. Everything breaks whenever we have two classes (obviously with methods to be documented). Here in order to have an automatic class-methods, class-methods and then function interleaving we clearly should change the code, making the class generating its own docs. Then comes the code reuse problem and that's why I put the rendering functions outside. This could be see as feature creeping but comes free trying to solve the first problems.

A possible solution would be to avoid pages mixing methods and functions.
In addition, a good edit of my code would be to allow the filtering of class methods.

Let me know what you think and if there is something I am still missing...

@fchollet
Copy link
Member

fchollet commented Mar 26, 2018

We can definitely go with your PR. Please first edit it with these 3 things in mind:

  • We should generate pure markdown, without HTML/CSS for formatting.
  • We don't need to edit the site's CSS.
  • We would need the ability to explicitly list which methods we would like to document for a class. For instance, we could add the ability to include dicts in the classes list, with entries "path", "methods". A possible value for "methods" would then be '*' (document all public methods of the class).

@StefanoCappellini
Copy link
Contributor Author

@fchollet PTAL.

  • The generated doc is fully markdown
  • It is possible to (I added a comment to explain this syntax):
    • Document only the class: classes = [classA, classB, ...]
    • Document all class methdos: classes = [classA, (classB, "*"), ...]
    • Document some class methdos: classes = [classA, (classB, ["methodA"]), ...]
      The methods are listed as string this to save the typing of full signatures. [we could change this]
  • It reuses the old classes field
  • It doesn't use external CSS or JS files
  • It adds a class label to all the classes (true classes)
  • It adds a method label to all the methods
  • In order to hide the class labels from the menu and to hide the methods, it extends the default theme

Let me know what you think, thanks.

@StefanoCappellini
Copy link
Contributor Author

New commit: now it is possible to document class methods:

  • Using strings: classes = [classA, (classB, ["methodA"]), ...]
  • Using qualified names: classes = [classA, (classB, [module.classB.methodA]), ...]

Probably the second is easy to use during refactoring.

@fchollet
Copy link
Member

Please note that the Keras docs (online) are already using a custom theme.

The changes are reasonable, but please handle this feature without changes to the theme/markup, just by generating markdown (in particular, no need for docs/custom_theme/toc.html). E.g. following this model: https://github.com/keras-team/keras/blob/master/docs/templates/models/model.md

@StefanoCappellini
Copy link
Contributor Author

@fchollet Ok it should be ok now. PTAL again, thanks

Copy link
Member

@fchollet fchollet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

@fchollet fchollet merged commit 886c021 into keras-team:master Apr 2, 2018
@StefanoCappellini StefanoCappellini deleted the class-method-docs branch April 3, 2018 15:20
dschwertfeger added a commit to dschwertfeger/keras that referenced this pull request Apr 6, 2018
…ack-embeddings-from-layer-outputs

* upstream/master: (68 commits)
  fit/evaluate_generator supporting native tensors (keras-team#9816)
  keras-team#9642 Add kwarg and documentation for dilation_rate to SeparableConvs (keras-team#9844)
  Document that "same" is inconsistent across backends with strides!=1 (keras-team#9629)
  Improve tests by designating dtype of sample data (keras-team#9834)
  Add documentation for 'subset' and interpolation' arguments (ImageDataGenerator) (keras-team#9817)
  Revert default theme to readthedocs
  Various docs fixes.
  Fix conflict
  Add support for class methods documentation (keras-team#9751)
  Add missing verbose opt for evaluate_generator (keras-team#9811)
  Added `data_format` to flatten layer. (keras-team#9696)
  Allow saving models directly to binary stream (keras-team#9789)
  Fix ctc_batch_cost() error when batch_size = 1 (keras-team#9775)
  Fix keras-team#9802 (keras-team#9803)
  Fix error in ImageDataGenerator documentation (keras-team#9798)
  fix typo (keras-team#9792)
  keras-team#9733: Extend RemoteMonitor to send data as application/json (keras-team#9734)
  Fixed inconsistencies regarding ReduceLROnPlateau (keras-team#9723)
  Fix doc issue.
  General stateful metrics fixes (keras-team#9446)
  ...
@StefanoCappellini
Copy link
Contributor Author

StefanoCappellini commented Apr 7, 2018

Hi @fchollet , I took a look to the 1e82a71 commit and I think that it re-introduces the problems we had before:

  1. Now the doc for important utilities (load_img, img_to_array, ...) is missing. Is it intentional? Probably they should be added
  2. The horizontal rules separating the class methods makes hard to distinguish them from plain module functions. Probably we should remove them (as in the original commit).

I have already a pull request ready, I just wanted to know if this was intentional, in order to avoid useless PR.
Thank you

Vijayabhaskar96 pushed a commit to Vijayabhaskar96/keras that referenced this pull request May 3, 2018
* Add support for class methods documentation

* Documentation: remove the need of external CNN and JS files, using plain markdown

* Documentation: allow to decide which class method to document using a string or a fully qualified name

* Documentation: remove the need of a custom theme
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants