chore: Add py.typed file and resolve mypy errors in run.py #252

lewisjared · 2023-08-22T07:27:09Z

Description

Adds py.typed to the package to indicate to distribute type information.

The type hints are not 100% complete with some additional work to be done in some of the modules, particularly groupby.py, but I think we have good coverage for the most commonly used parts of the library (namely scmdata.ScmRun).

This PR ensures that mypy src/scmdata/run.py passes successfully with our current mypy config. These are quite relaxed compared to our new defaults though

Pull request

Please confirm that this pull request has done the following:

Tests added
Documentation added (where applicable)
Example added (either to an existing notebook or as a new notebook, where applicable)
Description in CHANGELOG.rst added

lewisjared · 2023-08-22T07:29:28Z

@mikapfl Can you try using this branch today and see if there are any big problems

mikapfl · 2023-08-22T07:45:15Z

I don't know if I will get to it today, but I'll do it ASAP.

lewisjared · 2023-08-22T07:50:46Z

All good - no rush

codecov · 2023-08-22T08:02:16Z

Codecov Report

Patch coverage: 92.56% and project coverage change: -0.29% ⚠️

Comparison is base (4d95b7d) 95.68% compared to head (575c706) 95.40%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #252      +/-   ##
==========================================
- Coverage   95.68%   95.40%   -0.29%     
==========================================
  Files          24       24              
  Lines        2132     2153      +21     
  Branches      395      400       +5     
==========================================
+ Hits         2040     2054      +14     
- Misses         73       79       +6     
- Partials       19       20       +1

Files Changed	Coverage Δ
src/scmdata/groupby.py	`82.25% <ø> (ø)`
src/scmdata/testing.py	`70.58% <ø> (ø)`
src/scmdata/run.py	`95.07% <90.21%> (-0.87%)`	⬇️
src/scmdata/__init__.py	`77.77% <100.00%> (+2.77%)`	⬆️
src/scmdata/filters.py	`100.00% <100.00%> (ø)`
src/scmdata/netcdf.py	`98.11% <100.00%> (ø)`
src/scmdata/offsets.py	`100.00% <100.00%> (ø)`
src/scmdata/pyam_compat.py	`26.31% <100.00%> (+4.09%)`	⬆️
src/scmdata/time.py	`100.00% <100.00%> (ø)`
src/scmdata/timeseries.py	`86.81% <100.00%> (+0.14%)`	⬆️
... and 1 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mikapfl · 2023-09-07T13:02:10Z

I get lots and lots of errors, the root seems to be:

src/ndcs/variant/historical_emissions.py:194: error: Module "scmdata" does not explicitly export attribute "ScmRun"  [attr-defined]

mikapfl · 2023-09-08T14:47:18Z

A lot better already; however, we still have a solid number of warnings in ndc-quantification:

MYPYPATH=stubs poetry run mypy --install-types --no-warn-unused-ignore src
src/ndcs/timeseries.py: note: In function "clip":
src/ndcs/timeseries.py:68: error: Argument "variable" to "filter" of "BaseScmRun" has incompatible type "dict_keys[str, LimitConfig]"; expected "str | int | float"  [arg-type]
src/ndcs/timeseries.py: note: In function "convert_to_gwp":
src/ndcs/timeseries.py:161: error: Invalid index type "tuple[str | int | float, str]" for "_LocIndexerFrame"; expected type "slice | ndarray[Any, dtype[integer[Any]]] | Index | list[int] | Series[int] | Series[bool] | ndarray[Any, dtype[bool_]] | list[bool] | Callable[[DataFrame], slice | ndarray[Any, dtype[integer[Any]]] | Index | list[int] | Series[int] | Series[bool] | ndarray[Any, dtype[bool_]] | list[bool] | list[<nothing>]] | list[<nothing>] | tuple[slice | ndarray[Any, dtype[integer[Any]]] | Index | list[int] | Series[int] | Series[bool] | ndarray[Any, dtype[bool_]] | list[bool] | list[<nothing>] | tuple[Index | Series[bool] | ndarray[Any, dtype[bool_]] | list[bool] | str | bytes | date | datetime | timedelta | datetime64 | timedelta64 | bool | int | float | Timestamp | Timedelta | complex | list[Any] | slice | tuple[str | bytes | date | datetime | timedelta | datetime64 | timedelta64 | bool | int | float | Timestamp | Timedelta | complex, ...], ...] | Callable[..., Any], list[<nothing>] | slice | Series[bool] | Callable[..., Any]]"  [index]
src/ndcs/timeseries.py:164: error: Invalid index type "tuple[str | int | float, str]" for "_LocIndexerFrame"; expected type "slice | ndarray[Any, dtype[integer[Any]]] | Index | list[int] | Series[int] | Series[bool] | ndarray[Any, dtype[bool_]] | list[bool] | Callable[[DataFrame], slice | ndarray[Any, dtype[integer[Any]]] | Index | list[int] | Series[int] | Series[bool] | ndarray[Any, dtype[bool_]] | list[bool] | list[<nothing>]] | list[<nothing>] | tuple[slice | ndarray[Any, dtype[integer[Any]]] | Index | list[int] | Series[int] | Series[bool] | ndarray[Any, dtype[bool_]] | list[bool] | list[<nothing>] | tuple[Index | Series[bool] | ndarray[Any, dtype[bool_]] | list[bool] | str | bytes | date | datetime | timedelta | datetime64 | timedelta64 | bool | int | float | Timestamp | Timedelta | complex | list[Any] | slice | tuple[str | bytes | date | datetime | timedelta | datetime64 | timedelta64 | bool | int | float | Timestamp | Timedelta | complex, ...], ...] | Callable[..., Any], list[<nothing>] | slice | Series[bool] | Callable[..., Any]]"  [index]
src/ndcs/timeseries.py:180: error: Returning Any from function declared to return "ScmRun"  [no-any-return]
src/ndcs/timeseries.py:180: error: Call to untyped function "apply" in typed context  [no-untyped-call]
src/ndcs/excel_utils.py: note: In function "extract_table":
src/ndcs/excel_utils.py:114: error: Unsupported operand types for | ("dict[str, Any]" and "dict[str, Any]")  [operator]
src/ndcs/missing.py: note: In function "calc_missing_leaf_timeseries":
src/ndcs/missing.py:87: error: Argument 1 to "chain" has incompatible type "*tuple[tuple[str], Hashable, tuple[str]]"; expected "Iterable[str]"  [arg-type]
src/ndcs/missing.py:94: error: Argument 3 to "ScmRun" has incompatible type "dict[str, tuple[Any, ...]]"; expected "dict[Hashable, str | Iterable[str]] | None"  [arg-type]
src/ndcs/variant/projected_emissions.py: note: In function "load_projected_emissions":
src/ndcs/variant/projected_emissions.py:107: error: Argument 1 to "filter" of "BaseScmRun" has incompatible type "**dict[str, str]"; expected "bool"  [arg-type]
src/ndcs/variant/projected_emissions.py:111: error: Argument "year" to "filter" of "BaseScmRun" has incompatible type "ndarray[Any, dtype[signedinteger[Any]]]"; expected "str | int | float"  [arg-type]
src/ndcs/variant/projected_emissions.py:113: error: No overload variant of "dropna" of "DataFrame" matches argument types "str", "str"  [call-overload]
src/ndcs/variant/projected_emissions.py:113: note: Possible overload variants:
src/ndcs/variant/projected_emissions.py:113: note:     def dropna(self, *, axis: Literal['index', 0] | Literal['columns', 1] = ..., how: Literal['any', 'all'] = ..., thresh: int | None = ..., subset: Sequence[Any] | ndarray[Any, Any] | Series[Any] | Index | str | bytes | date | datetime | timedelta | datetime64 | timedelta64 | bool | int | float | Timestamp | Timedelta | complex | None = ..., inplace: Literal[True]) -> None
src/ndcs/variant/projected_emissions.py:113: note:     def dropna(self, *, axis: Literal['index', 0] | Literal['columns', 1] = ..., how: Literal['any', 'all'] = ..., thresh: int | None = ..., subset: Sequence[Any] | ndarray[Any, Any] | Series[Any] | Index | str | bytes | date | datetime | timedelta | datetime64 | timedelta64 | bool | int | float | Timestamp | Timedelta | complex | None = ..., inplace: Literal[False] = ...) -> DataFrame
src/ndcs/variant/projected_emissions.py:113: note:     def dropna(self, *, axis: Literal['index', 0] | Literal['columns', 1] = ..., how: Literal['any', 'all'] = ..., thresh: int | None = ..., subset: Sequence[Any] | ndarray[Any, Any] | Series[Any] | Index | str | bytes | date | datetime | timedelta | datetime64 | timedelta64 | bool | int | float | Timestamp | Timedelta | complex | None = ..., inplace: bool | None = ...) -> DataFrame | None
src/ndcs/variant/projected_emissions.py:121: error: Argument "year" to "filter" of "BaseScmRun" has incompatible type "ndarray[Any, dtype[signedinteger[Any]]]"; expected "str | int | float"  [arg-type]
src/ndcs/variant/projected_emissions.py: note: In function "calculate_lulucf_projections":
src/ndcs/variant/projected_emissions.py:163: error: Argument "region" to "filter" of "BaseScmRun" has incompatible type "list[str | int | float]"; expected "str | int | float"  [arg-type]
src/ndcs/aggregations.py: note: In function "remove_redundant_totals":
src/ndcs/aggregations.py:119: error: Argument 2 to "set_meta" of "BaseScmRun" has incompatible type "ndarray[Any, dtype[signedinteger[Any]]]"; expected "str | int | float"  [arg-type]
src/ndcs/aggregations.py:135: error: Argument "__id" to "filter" of "BaseScmRun" has incompatible type "Series[Any]"; expected "str | int | float"  [arg-type]
src/ndcs/aggregations.py: note: In function "remove_totals":
src/ndcs/aggregations.py:217: error: Argument "variable" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/aggregations.py:217: error: Argument "category" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/aggregations.py: note: In function "append_missing_totals_dataset":
src/ndcs/aggregations.py:272: error: Returning Any from function declared to return "ScmRun"  [no-any-return]
src/ndcs/aggregations.py:272: error: Call to untyped function "apply" in typed context  [no-untyped-call]
src/ndcs/aggregations.py: note: In function "_find_derivation":
src/ndcs/aggregations.py:336: error: Function is missing a type annotation for one or more arguments  [no-untyped-def]
src/ndcs/aggregations.py: note: In function "calc_missing_totals":
src/ndcs/aggregations.py:424: error: Argument "variable" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/aggregations.py:425: error: Argument "category" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/aggregations.py:435: error: Argument "category" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/aggregations.py:445: error: Argument "variable" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/aggregations.py:456: error: Argument "variable" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/aggregations.py: note: In function "aggregate_categories":
src/ndcs/aggregations.py:524: error: Argument "category" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/aggregations.py: note: In function "aggregate_gases":
src/ndcs/aggregations.py:583: error: Argument "variable" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/coverage.py: note: In function "calculate_coverage":
src/ndcs/coverage.py:170: error: Call to untyped function "apply" in typed context  [no-untyped-call]
src/ndcs/coverage.py:177: error: Returning Any from function declared to return "ScmRun"  [no-any-return]
src/ndcs/coverage.py: note: In function "_with_coverage":
src/ndcs/coverage.py:186: error: Argument "variable" to "covers" of "ForcingAgentCoverage" has incompatible type "str | int | float"; expected "str | HierarchicalCategory"  [arg-type]
src/ndcs/coverage.py:186: error: Argument "category" to "covers" of "ForcingAgentCoverage" has incompatible type "str | int | float"; expected "str | HierarchicalCategory | None"  [arg-type]
src/ndcs/coverage.py: note: In function "calculate_coverage_emissions_dataset":
src/ndcs/coverage.py:235: error: Call to untyped function "apply" in typed context  [no-untyped-call]
src/ndcs/coverage.py: note: In function "_with_coverage_both":
src/ndcs/coverage.py:247: error: Argument "variable" to "covers" of "ForcingAgentCoverage" has incompatible type "str | int | float"; expected "str | HierarchicalCategory"  [arg-type]
src/ndcs/coverage.py:247: error: Argument "category" to "covers" of "ForcingAgentCoverage" has incompatible type "str | int | float"; expected "str | HierarchicalCategory | None"  [arg-type]
src/ndcs/variant/bau_emissions.py: note: In function "load_bau_emissions":
src/ndcs/variant/bau_emissions.py:132: error: Argument "category" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/variant/bau_emissions.py: note: In function "fill_missing_country_data_from_groups":
src/ndcs/variant/bau_emissions.py:268: error: Argument "variable" to "filter" of "BaseScmRun" has incompatible type "set[str]"; expected "str | int | float"  [arg-type]
src/ndcs/sources/projected_emissions.py: note: In member "process_raw_data" of class "UNBRCTFProjectedEmissionsCreator":
src/ndcs/sources/projected_emissions.py:73: error: Argument "category" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/sources/historical_emissions.py: note: In member "process_raw_data" of class "PRIMAPhistCreator":
src/ndcs/sources/historical_emissions.py:71: error: Argument "year" to "filter" of "BaseScmRun" has incompatible type "range"; expected "str | int | float"  [arg-type]
src/ndcs/sources/historical_emissions.py:81: error: Argument "category" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/sources/historical_emissions.py:82: error: Argument "category" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/sources/historical_emissions.py: note: In function "process_raw_data":
src/ndcs/sources/historical_emissions.py:93: error: Argument 1 to "convert_to_target_unit" has incompatible type "str | int | float"; expected "str | Unit | Quantity[float]"  [arg-type]
src/ndcs/sources/historical_emissions.py:96: error: Returning Any from function declared to return "ScmRun"  [no-any-return]
src/ndcs/sources/historical_emissions.py: note: In member "process_raw_data" of class "PRIMAPhistCreator":
src/ndcs/sources/historical_emissions.py:98: error: Call to untyped function "apply" in typed context  [no-untyped-call]
src/ndcs/sources/baseline_emissions.py: note: In member "process_raw_data" of class "PRIMAPDownscaledSSPCreator":
src/ndcs/sources/baseline_emissions.py:81: error: Argument "region" to "filter" of "BaseScmRun" has incompatible type "list[str]"; expected "str | int | float"  [arg-type]
src/ndcs/targets/models.py: note: In member "calculate" of class "CountryTarget":
src/ndcs/targets/models.py:151: error: List item 0 has incompatible type "bool"; expected "str"  [list-item]
src/ndcs/targets/models.py:155: error: List item 0 has incompatible type "Conditionality"; expected "str"  [list-item]
src/ndcs/targets/models.py:156: error: List item 0 has incompatible type "Ambition"; expected "str"  [list-item]
src/ndcs/targets/models.py:159: error: List item 0 has incompatible type "str | int | float"; expected "str"  [list-item]

I have a patch which reduces this somewhat, I'll post it as a PR to this PR.

lewisjared · 2023-09-12T01:55:31Z

Merging this once the checks pass. The type-hints will be a source for continuous improvement

Co-authored-by: Jared Lewis <jared@jared.kiwi.nz>

lewisjared and others added 11 commits September 12, 2023 11:57

chore: Add py.typed file and resolve mypy errors in run.py

dbae68a

chore: Expand mypy rules further

7200b1d

chore: Fix up CI

ab412fe

chore: Fix up some more type hints

371676c

fix: better types for apply, filter, __init__, and set_meta

79163fe

style: black

ce70b95

fix: delete benchmark.py, it was included inadvertently

1234c5b

Update src/scmdata/run.py

fd46e4c

Co-authored-by: Jared Lewis <jared@jared.kiwi.nz>

chore: Fix mistake in netcdf notebook

6130494

chore: Fix mistake in netcdf notebook

a7b71b2

chore: changelog

2b167fd

lewisjared force-pushed the 251-py-typed branch from bea7bff to 2b167fd Compare September 12, 2023 01:59

lewisjared self-assigned this Sep 12, 2023

lewisjared added 3 commits September 12, 2023 12:29

chore: Few more cleanups

37ae293

chore: isort

6be2a14

chore: linting

575c706

lewisjared merged commit 83d9990 into master Sep 12, 2023
26 of 28 checks passed

lewisjared deleted the 251-py-typed branch September 12, 2023 03:51

lewisjared mentioned this pull request Sep 13, 2023

Add py.typed marker to export types to users #251

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Add py.typed file and resolve mypy errors in run.py #252

chore: Add py.typed file and resolve mypy errors in run.py #252

lewisjared commented Aug 22, 2023 •

edited

Loading

lewisjared commented Aug 22, 2023

mikapfl commented Aug 22, 2023

lewisjared commented Aug 22, 2023

codecov bot commented Aug 22, 2023 •

edited

Loading

mikapfl commented Sep 7, 2023

mikapfl commented Sep 8, 2023

lewisjared commented Sep 12, 2023

chore: Add py.typed file and resolve mypy errors in run.py #252

chore: Add py.typed file and resolve mypy errors in run.py #252

Conversation

lewisjared commented Aug 22, 2023 • edited Loading

Description

Pull request

lewisjared commented Aug 22, 2023

mikapfl commented Aug 22, 2023

lewisjared commented Aug 22, 2023

codecov bot commented Aug 22, 2023 • edited Loading

Codecov Report

mikapfl commented Sep 7, 2023

mikapfl commented Sep 8, 2023

lewisjared commented Sep 12, 2023

lewisjared commented Aug 22, 2023 •

edited

Loading

codecov bot commented Aug 22, 2023 •

edited

Loading