3014 - Outdated submissions banner

[jtimpe]

Summary of Changes

Pull request closes #3014

• adds outdated submissions banner for files submitted before 5/31/2024
• removes the error report link for those files (replaces with "Not Available")

How to Test

cd tdrs-backend && docker compose up
cd tdrs-frontend && docker compose up --build

1. Open http://localhost:3000/ and sign in.
2. Submit a data file. Make sure the banner does not appear, and the error reports download link is shown.
3. Modify the created_at field of the datafile to before 5/31/2024

   docker compose exec web python manage.py shell_plus
   
   f = DataFile.objects.all().first()
   from django.utils import timezone
   from datetime import datetime
   f.created_at = datetime(2023, 10, 11)
   f.save()

4. Return to the datafiles page. Ensure the banner displays and the error report download link is replaced with "Not Available"

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• Banner appears on submission history if 1 or more files are present for viewed quarter that were submitted prior to May 31, 2024
• Banner does not appear if a given quarter doesn't have data submitted before May 31 2024
• Language matches spec "Please note that error reports and submission history content for files submitted prior to May 31, 2024 may be outdated. Please resubmit to get access to updated information."
• Testing Checklist has been run and all tests pass
• lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[codecov]

Codecov Report

Attention: Patch coverage is 74.00000% with 13 lines in your changes missing coverage. Please review.

Project coverage is 90.59%. Comparing base (4fa79a1) to head (d7291fb).
Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
...components/SubmissionHistory/SubmissionHistory.jsx	66.66%	4 Missing and 1 partial ⚠️
...ponents/SubmissionHistory/TotalAggregatesTable.jsx	61.53%	4 Missing and 1 partial ⚠️
...mponents/SubmissionHistory/CaseAggregatesTable.jsx	76.92%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3231      +/-   ##
===========================================
- Coverage    90.65%   90.59%   -0.07%     
===========================================
  Files          299      299              
  Lines         8490     8526      +36     
  Branches       794      802       +8     
===========================================
+ Hits          7697     7724      +27     
- Misses         676      684       +8     
- Partials       117      118       +1     

Flag	Coverage Δ
dev-backend	90.39% <100.00%> (+0.01%)	⬆️
dev-frontend	91.99% <68.29%> (-0.67%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
tdrs-backend/tdpservice/data_files/serializers.py	100.00% <100.00%> (ø)
tdrs-frontend/src/reducers/reports.js	90.00% <ø> (ø)
...mponents/SubmissionHistory/CaseAggregatesTable.jsx	82.60% <76.92%> (+1.35%)	⬆️
...components/SubmissionHistory/SubmissionHistory.jsx	86.84% <66.66%> (-13.16%)	⬇️
...ponents/SubmissionHistory/TotalAggregatesTable.jsx	63.63% <61.53%> (-3.04%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a10638d...d7291fb. Read the comment docs.

[elipe17]

One comment, doesn't halt anything. LGTM

[victoriaatraft]

LGTM @jtimpe

[ADPennington]

LGTM @jtimpe

per standup today, @jtimpe noted that submitted by column is blank because his user's first and last name is blank in user profile.

[ADPennington]

@jtimpe this PR delivers the following expected results 🥇 :

• Banner appears when files were submitted prior to May 31, 2024 ✔️
• error report is unavailable for abovementioned files ✔️

However, we believe this change is now out-of-scope given the reparsing work. In particular:

• when files are reparsed, the submission history metadata, including the error reports, could be refreshed, but if the reparsed file was submitted prior to May 31, 2024, this report will still be unavailable (see evidence below)

• The data team considered some possible conditions to apply that could help make this workable, including:
  • changing the date to something much earlier (e.g. Dec 31, 2023)
  • adding a condition to display the error report if the file has been reparsed
  • making the error report unavailable if a file has been resubmitted

All of these alternatives scope-creep the ticket and adds unnecessary complexity to an otherwise great solution to a problem (i.e. STTs reviewing and asking Qs about outdated reports) that we think is now resolved with the reparsing feature.

cc: @ttran-hub @klinkoberstar @lfrohlich @vlasse86

[jtimpe]

@ADPennington we could potentially make use of the new ReparseFileMeta to see if a particular file has newer reparses available. I have to think more about how that would work, and it definitely increases scope a bit, but gut feeling says it should be doable.

[jtimpe]

@ADPennington i made the change to include information about the datafile's reparses inside of the api response. The Submission History page now checks the ReparseFileMeta's finished_at field if the original file submission date is outdated. If there is a recent reparse, it will not show the banner and the error report will be available. If there is no recent reparse, it will show the banner and disable the report.

One caveat is that the Submission History page maintains the original created date. So, if that date is prior to May 31 2024, but the file has a more recent reparse, it may be slightly confusing. Ideally there would be a way to show the file's version history a little more clearly with respect to reparse. Open to suggestions on that front.

[ADPennington]

@ADPennington i made the change to include information about the datafile's reparses inside of the api response. The Submission History page now checks the ReparseFileMeta's finished_at field if the original file submission date is outdated. If there is a recent reparse, it will not show the banner and the error report will be available. If there is no recent reparse, it will show the banner and disable the report.

One caveat is that the Submission History page maintains the original created date. So, if that date is prior to May 31 2024, but the file has a more recent reparse, it may be slightly confusing. Ideally there would be a way to show the file's version history a little more clearly with respect to reparse. Open to suggestions on that front.

thank you for this update @jtimpe -- I plan to retest with @ttran-hub and @klinkoberstar early next week. in the meantime, if @victoriaatraft and @reitermb would like to explore this solution in a deployed environment, please feel free 😄