We developed a lexical analyzer for python using JFlex library.
JFlex is a lexical analyzer generator (also known as scanner generator) for Java, written in Java.
We're using the IntelliJ IDEA. But, you can do this on command line if you want.
Clone this rep by using
$ git clone https://github.com/chrismachado/Python-Lexer-Analyzer.git
First of all, you need the Lexical class for it. So, you can get it with by running the bin/LexerGenMain, that should be the result:
After that, you need to run the GUI view/InputPyFileScreen, than choose your Python file. We have some examples in files/, you could use it or run your own code.
The Python Tokens are listed in to class bin/PyTokens as we show.
PS*: This list will be modified.
After you load your code, hit the Analyze button and you can see the Token, type and line of it. On bottom side, we show a table with the amount of each token. Like this.
- I'd like to thank Professor Bruno Correia da Silva.