Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loading a Report snapshot fails for RecSys metrics with k>=10 #1233

Open
soogui opened this issue Jul 31, 2024 · 0 comments
Open

Loading a Report snapshot fails for RecSys metrics with k>=10 #1233

soogui opened this issue Jul 31, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@soogui
Copy link
Contributor

soogui commented Jul 31, 2024

Hi!

I am trying to use the snapshot functionality to store my reports and later loading and visualizing them somewhere else. I am using metrics to evaluate a recommender and run into a problem when I change the k parameter.

To reproduce the issue, I modified the following notebook:
examples/how_to_questions/how_to_run_recsys_metrics.ipynb

I modified the cell in the Metrics section that creates the report to keep only a single HitRate metric:

report = Report(metrics=[
    HitRateKMetric(k=5),
])

With k=5, running the report, loading it back and displaying works fine:

report.save('report.json')
loaded_report = Report.load('report.json')
loaded_report

However, when I change the k=10 parameter:

report = Report(metrics=[
    HitRateKMetric(k=10),
])

I get the following error when I attempt to display the loaded_report:


ValueError                                Traceback (most recent call last)
File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/indexes/range.py:345, in RangeIndex.get_loc(self, key)
    344 try:
--> 345     return self._range.index(new_key)
    346 except ValueError as err:

ValueError: 10 is not in range

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/IPython/core/formatters.py:344, in BaseFormatter.__call__(self, obj)
    342     method = get_real_method(obj, self.print_method)
    343     if method is not None:
--> 344         return method()
    345     return None
    346 else:

File ~/Documents/repos/evidently/src/evidently/suite/base_suite.py:216, in Display._repr_html_(self)
    215 def _repr_html_(self):
--> 216     dashboard_id, dashboard_info, graphs = self._build_dashboard_info()
    217     template_params = TemplateParams(
    218         dashboard_id=dashboard_id,
    219         dashboard_info=dashboard_info,
    220         additional_graphs=graphs,
    221     )
    222     return self._render(inline_iframe_html_template, template_params)

File ~/Documents/repos/evidently/src/evidently/report/report.py:230, in Report._build_dashboard_info(self)
    228 # set the color scheme from the report for each render
    229 renderer.color_options = color_options
--> 230 html_info = renderer.render_html(metric)
    231 set_source_fingerprint(html_info, metric)
    232 replace_widgets_ids(html_info, id_generator)

File ~/Documents/repos/evidently/src/evidently/metrics/recsys/base_top_k.py:79, in TopKMetricRenderer.render_html(self, obj)
     77 metric_result = obj.get_result()
     78 k = metric_result.k
---> 79 counters = [CounterData.float(label=\"current\", value=metric_result.current[k], precision=3)]
     80 if metric_result.reference is not None:
     81     counters.append(CounterData.float(label=\"reference\", value=metric_result.reference[k], precision=3))

File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/series.py:1007, in Series.__getitem__(self, key)
   1004     return self._values[key]
   1006 elif key_is_scalar:
-> 1007     return self._get_value(key)
   1009 if is_hashable(key):
   1010     # Otherwise index.get_value will raise InvalidIndexError
   1011     try:
   1012         # For labels that don't resolve as scalars like tuples and frozensets

File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/series.py:1116, in Series._get_value(self, label, takeable)
   1113     return self._values[label]
   1115 # Similar to Index.get_value, but we do not fall back to positional
-> 1116 loc = self.index.get_loc(label)
   1118 if is_integer(loc):
   1119     return self._values[loc]

File ~/Documents/repos/evidently/venv/lib/python3.8/site-packages/pandas/core/indexes/range.py:347, in RangeIndex.get_loc(self, key)
    345         return self._range.index(new_key)
    346     except ValueError as err:
--> 347         raise KeyError(key) from err
    348 if isinstance(key, Hashable):
    349     raise KeyError(key)

KeyError: 10"
@emeli-dral emeli-dral added the bug Something isn't working label Aug 8, 2024
mike0sv added a commit that referenced this issue Aug 26, 2024
emeli-dral pushed a commit that referenced this issue Aug 28, 2024
* fix recsys stuff
#1233
#1231

* fix tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants