@ritikjain51 there are also acceptance tests and notebooks to update with your new change - the notebooks are under the notebook folder. Also, look for tests under the slow-tests folder. This is how you run the test coverage shell script:

cd [root of the project folder]
./test-coverage.sh "tests slow-tests"

This will generate tests run report and test coverage report.

@ritikjain51 please also add a bit of description explaining the background what you are doing, what is the outcome and what are you using to get the results, this will help me and others reading this PR.

As you can see from #15, you have opened up a potential, something I knew about but didn't give it a priority.

@ritikjain51 could you also please add a description to the PR as this will help me and others who look at it later.

You may have also missed out re-running the other notebook (nlp_profiler.ipynb), please take a look. You can skip the nlp_profiler-large-dataset.ipynb for now.

Thnx

@ritikjain51 Thanks for your patience and following through with the code review.

I will write-up the Developer guide soon so it can be a standard way to do things. There are also a couple of things to be automated which will also help.

@ritikjain51 your PR is out of sync with master, you may need to rebase with master

Are you using nbdime or another Jupyter/notebook version control plugin, it can help understand the differences?

There are 3 Cases are failing because I don't have datasets to validate those.
Those 3 Cases are being used in test_apply_text_profiling.py

...

There are 3 Cases are failing because I don't have datasets to validate those.
Those 3 Cases are being used in test_apply_text_profiling.py

The dataset is generated in the test itself, have a read of the tests that are failing, there is a method that creates the dataset.

These failures are normal if you look closely the Acceptance test is trying to compare old results with your new results (has one extra column) so it will fail. I suggest you generate the expected datasets and save them in the https://github.com/neomatrix369/nlp_profiler/tree/master/tests/acceptance_tests/data.

It's easy to generate the new test data, but you do it only when you know the new dataset is correct in every form. For each of the test cases in https://github.com/neomatrix369/nlp_profiler/blob/master/tests/acceptance_tests/test_apply_text_profiling.py save the results generated by the actual_dataframe using this line:

        def test_case() # for all the relevant test cases
                     ...
                     actual_dataframe.to_csv(csv_filename, index=False)  ### remove this line when you have finished generating the csv file
                     ...

Compare the old results with new using git diff and you can see the changes, in your case only your new column should change not any other aspect of the .csv files.

Let me know if you don't follow this step.

I hope to make this easier in future releases. So we can generate our tests data easily. But generating test data is always a manual task.

There are 3 Cases are failing because I don't have datasets to validate those.
Those 3 Cases are being used in test_apply_text_profiling.py

The dataset is generated in the test itself, have a read of the tests that are failing, there is a method that creates the dataset.

In the test code, We are fetching data from expected data path location. Which is on available with me

@ritikjain51

Not true, the failing tests in https://github.com/neomatrix369/nlp_profiler/blob/master/tests/acceptance_tests/test_apply_text_profiling.py are creating their own dataset and also using the data in the folder https://github.com/neomatrix369/nlp_profiler/tree/master/tests/acceptance_tests/data. Please look again.

There is no external data used here.

@ritikjain51

Not true, the failing tests in https://github.com/neomatrix369/nlp_profiler/blob/master/tests/acceptance_tests/test_apply_text_profiling.py are creating their own dataset and also using the data in the folder https://github.com/neomatrix369/nlp_profiler/tree/master/tests/acceptance_tests/data. Please look again.

There is no external data used here.

These Issues are there because u have made structural changes in your code.

Thanks @neomatrix369. for your guidance. I am closing this Pull Request. It's getting difficult to understand the issues.
Hope you understand.
Regards,
Ritik Jain

@ritikjain51 the master branch did move on Friday as I was refactoring the code (implementation) - sorry about that but it had to be done for better readability and maintenance.

The merge conflicts were mainly due to changes in the data and notebook files.

It is normal in a project for code to change in branch to be out of step with master as development continues. I see you have also deleted your original changes, maybe someone could have helped you fix the changes.

PyCharm or VSCode have good git functionalities to help in the process - it's something out of scope of my project to help you with.

You can always create a new fork and then a new branch with the changes and regenerate the notebooks - it's up to you, otherwise myself or someone else might implement this functionality separately.

Your tests were failing on CI/CD due to the absence of this line:

nltk.download('averaged_perceptron_tagger')

in the noun_phase_count.py module (see 08e54c4#diff-071796e721082b14e9eb7c22c771ca452fe1f83652dc4bcce6d1c599409a6c41R10)

The CI/CD logs did indicate that.

After fixing this line the tests pass:

tests/acceptance_tests/test_apply_text_profiling.py .....                                                                [  2%]
tests/granular/test_alphanumeric.py ........                                                                             [  7%]
tests/granular/test_chars_and_spaces.py ......                                                                           [ 10%]
tests/granular/test_dates.py .........                                                                                   [ 15%]
tests/granular/test_duplicates.py .........                                                                              [ 21%]
tests/granular/test_emojis.py ........                                                                                   [ 25%]
tests/granular/test_non_alphanumeric.py ........                                                                         [ 30%]
tests/granular/test_nounphase.py .........                                                                               [ 35%]
tests/granular/test_numbers.py ........                                                                                  [ 39%]
tests/granular/test_punctuations.py ........                                                                             [ 44%]
tests/granular/test_sentences.py ..................                                                                      [ 54%]
tests/granular/test_stop_words.py ........                                                                               [ 59%]
tests/granular/test_words.py ........                                                                                    [ 63%]
tests/high_level/test_grammar_check.py ..........                                                                        [ 69%]
tests/high_level/test_sentiment_polarity.py ................                                                             [ 78%]
tests/high_level/test_sentiment_subjectivity.py ................                                                         [ 87%]
tests/high_level/test_spelling_check.py ..................                                                               [ 97%]
slow-tests/acceptance_tests/test_apply_text_profiling.py .                                                               [ 98%]
slow-tests/performance_tests/test_perf_grammar_check.py .                                                                [ 98%]
slow-tests/performance_tests/test_perf_noun_phase.py .                                                                   [ 99%]
slow-tests/performance_tests/test_perf_spelling_check.py .                                                               [100%]

I would also suggest using the PR page i.e. #13 to read and respond to messages, lots of information on this page helps with development process.

@ritikjain51
Since you put so much time and effort into this work I restored the branch for you see https://github.com/neomatrix369/nlp_profiler/tree/addNounPhraseCount - your commits are preserved, I added some fixes to it.

Also, please have a look at the CI/CD process https://github.com/neomatrix369/nlp_profiler/runs/1271145643?check_suite_focus=true

Please do a thorough check of it to see if anything has been missed out. Happy for you to create a new PR as the community values your hard work - let's coordinate this time and try to stay in sync with master.

Again if you are unfamiliar with something its best to say it and ask for help - only way for us to come forward and step in.