Availability of the Ax6's V1.5.11

[Xwang1976]

Hi,
I've the ZIP file downloaded by Ax6 github before it disappeared.
It should be v.1.5.11 with some modification made after this year winter testing.
It seems to be better commented and have some differences with your code.
Fast-F1-master-v1.5.11.zip

[theOehrly]

Very interesting! I'll take a look at it and will try to get the changes integrated into my current version.

(I also moved this issue. I just noticed that issues weren't enabled in the Fast-F1 repo.)

[theOehrly]

All changes should now be merged in branch "ax6-latest-integration". Branch "ax6-latest" holds the code as you provided it.

From my testing, everything looks fine. If you have some scripts that you can easily run, maybe you could run them with the new code too. Just to make sure. My number of test cases is currently a bit limited. Then I can be a bit more confident to merge the changes into master.

[Xwang1976]

Hi,

I've still not used the ax6-latest-integration, but in the last weeks when I've used the ax6-latest, I've had to make some changes to core.py which I send you as an attachment.

The first one is to manage the exception that is raised when data are not available on ergast.

The second one is from this code:

        try:
            result = pd.merge_asof(d1, d2, on='Time', by='Driver')
        except:

to this one:

        try:
            result = pd.merge_asof(d1.sort_values('Time'), d2.sort_values('Time'), on='Time', by='Driver')
        except:

and I had to do that because downloading some of this year sessions or some of the last one (I've downloaded all the 2019 and 2020 GPs and 2020 testing) it complained that the merge was possible only if the arguments were ordered.

I think that this latest one should be added to the latest-integration (if you have not done something similar yet).

Do you agree?
core.zip

[theOehrly]

Yes, I think it's a good idea to fix both. The ergast one I have already fixed.

For the pd.merge_asof problem, I've never had that happen so far. Do you know a session where it happens? Because I'd like to take a look at where the problem is coming from in the first place.

[Xwang1976]

2020 Hungarian Grand Prix Practice 2 highlights the pd.merge_asof problem.
Meantime I've used the attached file to download with the ax6-latest-integration branch all the sessions from the beginning of 2019 to the latest GP.
I've had to apply the pd.merge fix and I've had to manually put the two csv files in the patched_ sessions folder (they are not in the branch but are needed when you get the testing days).
Doing so the python script has behaved in the same way as the ax6-latest + mine patches, but the binary .pkl files are different.
Is it correct? I expected that the .pkl files would have been identical to the ones I downloaded before.
getDataFromF1.zip

[theOehrly]

I took a look at this. I don't really want to use the sorting fix because the problem is actually somewhere else.

This is 2020, Hungary, FP2, Driver 16

Take a look at laps 5,6,7,8

	Time	Driver	Error	LastLapTime	NumberOfLaps
0	0 days 01:03:30.892000000	16			1
1	0 days 01:05:18.433000000	16			2
2	0 days 01:07:04.145000000	16	-0.115	0 days 00:01:45.827000000	3
3	0 days 01:08:47.853000000	16	-0.017	0 days 00:01:43.725000000	4
4	0 days 01:10:38.276000000	16	-0.004	0 days 00:01:50.427000000	5
5	0 days 01:30:56.146000000	16	0.153	0 days 00:20:17.717000000	6
6	0 days 01:51:13.720000000	16	-0.143	0 days 00:20:17.717000000	7
8	0 days 01:32:46.545000000	16	-1213.456	0 days 00:01:46.281000000	7
9	0 days 01:34:28.735000000	16	-4.257	0 days 00:01:46.447000000	8
10	0 days 01:36:13.315000000	16	0.003	0 days 00:01:44.577000000	9
11	0 days 01:38:03.320000000	16	-0.001	0 days 00:01:50.006000000	10
12	0 days 01:40:46.100000000	16	-0.018	0 days 00:02:42.798000000

I added the error column. It is the difference between the "Last Lap Time" column and the lap duration you get when subtracting to subsequent values from the "Time" column. It basically shows where the length of the lap is wrong.
And the length is (very) wrong for two columns!

The reason seems to bee that the API sends
['01:30:56.146', {'Lines': {'16': {'NumberOfLaps': 6, 'Sectors': {'2': {'PreviousValue': '31.362'}}, 'LastLapTime': {'Value': '20:17.717'}}}}]
and a bit later
['01:32:42.415', {'Lines': {'16': {'NumberOfLaps': 6, 'Sectors': {'2': {'PreviousValue': '30.280'}}, 'LastLapTime': {'Value': '20:17.717'}}}}]

Not quite sure what this is intended to be... Maybe a correction of the sector time?
Because the message is late by close to two minutes, the lap time is registered for the wrong lap. And this causes the time column to be wrong.

The cleanest way to correct this would probably be a major overhaul of api._timing_data_laps_entry so that errors like this are processed correctly.

I'll be on vacation for a few days and will only be able to look at it properly after that.

Concerning the missing patched files, they are in the branch. Did you install it using pip? If yes, that problem should be fixed now. I had to add a manifest and change the setup.py file.

[Xwang1976]

I've installed the latest-integration branch by downloading the .zip file.
As far as the bug, I agree with you that it should be solved differently, but I don't know if I'm personally able to help doing that.

[theOehrly]

I have decided that fixing the data merging/lap time error is outside of the scope of integrating this version of the code. I have modified the error handling slightly so that the loading of the data won't fail completely. Instead, if there is incorrect timing data, all data for this driver will be discarded and an error is printed. The rest of the drivers should be loaded fine.

This is not a very good solution as most of the drivers' laps might be correct. But I want to prioritize accurate data. Fixing this properly has a high priority but still might take some time.

Concerning the patch files, they are in the branch and I have fixed the issues when installing with pip.

I will merge this into master now as there is nothing in the branch that makes the overall functionality worse than what it currently is.

[Xwang1976]

How can I install the ax6-latest-integration branch with pip?

[theOehrly]

If you want to select a specific branch when installing, the command is
pip install git+https://github.com/theOehrly/Fast-F1.git@branch-name

Although you probably want to use master now that the changes have been merged.

[theOehrly]

After some major changes to the API data reading/processing code the merge problem should now be fixed properly.

This was actually caused because the data source is sometimes jumping back to the previous lap and then back to the current one again. This data which should belong to the previous lap was then added to the current one because there was no special handling for this kind of stuff.
In all cases which I analyzed, already known data was simply sent again. Therefore, if this happens the data is simply ignored. This gave the correct result in all my tests. Still, there will be a warning if this happens.

There are also some other changes, mainly in an attempt to make everything more robust against edge cases. You can take a look at the changelog if you're interested.

[Xwang1976]

I've update and trying to download the 2019 Australian Grand Prix Practice 3 I get the following error:

`core INFO Loading Australian Grand Prix Practice 3
core INFO Getting summary...
api WARNING The api attempted to rewrite history for driver 10. This was ignored! The data may not be entirely correct. (near lap 9)
api WARNING The api attempted to rewrite history for driver 77. This was ignored! The data may not be entirely correct. (near lap 13)
core INFO Formatting summary...
core INFO Getting telemetry data...
api INFO Fetching car data
api INFO Parsing car data
core INFO Getting position data...
api INFO Fetching position
api INFO Parsing position
core INFO Resampling telemetry...
Traceback (most recent call last):

File "/home/andreak/src/Python/F1data/getDataFromF1.py", line 23, in
ff1.get_session(2019,gp).get_practice(3).load_laps()

File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/utils.py", line 127, in decorator
laps = func(*args, **kwargs)

File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 347, in load_laps
telemetry, lap_start_date = self._load_telemetry()

File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 444, in _load_telemetry
self._augment_position()

File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 597, in _augment_position
lap = self._get_reference_lap()

File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 610, in _get_reference_lap
lap = self.laps.loc[times.index[i]].copy()

File "/usr/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 4095, in getitem
return getitem(key)

IndexError: index 296 is out of bounds for axis 0 with size 296`

[theOehrly]

Well, so much for more stability. I was hoping to not have issues like this anymore.
At one point a ran this for all sessions from 2019 and 2020 to make sure it doesn't crash. Guess some smaller change after that broke something.
I'll take a look at it I guess.

[Xwang1976]

Does the latest commit solves the issue above wihen getting Australian Grand Prix Practice 3 data?

[Xwang1976]

I've discovered another different issue with the 2019 Azerbaijan Grand Prix Practice 1:

core INFO Loading Azerbaijan Grand Prix Practice 1
core INFO Getting summary...
core INFO Formatting summary...
core INFO Getting telemetry data...
api INFO Fetching car data
api INFO Parsing car data
core INFO Getting position data...
api INFO Fetching position
api INFO Parsing position
core INFO Resampling telemetry...
core INFO Creating laps...
Traceback (most recent call last):

File "/home/andreak/src/Python/F1data/getDataFromF1.py", line 13, in
ff1.get_session(2019,gp).get_practice(1).load_laps()

File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/utils.py", line 127, in decorator
laps = func(*args, **kwargs)

File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 347, in load_laps
telemetry, lap_start_date = self._load_telemetry()

File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 450, in _load_telemetry
_log_progress(i, len(self.laps.index))

File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 951, in _log_progress
if logging.root.level >= logging.INFO and i % int(length / (c - 1)) == 0:

ZeroDivisionError: integer division or modulo by zero

[theOehrly]

Both of these bugs are fixed, and some others too. I have now tested Session.load_laps() for all sessions from 2019 and 2020. None of them should be crashing anymore.

[Xwang1976]

Ok, thank you.
I've seen that the SessionNotAvailableError derives from BaseException and not from Exception as is suggested
". In general, any custom exception class you create in your own code should inherit from Exception." (see https://airbrake.io/blog/python-exception-handling/class-hierarchy).
I'm not a python expert but it seems that it should derive from Exception. Do you agree?

[Xwang1976]

Both of these bugs are fixed, and some others too. I have now tested Session.load_laps() for all sessions from 2019 and 2020. None of them should be crashing anymore.

I've done the same including the 2020 testing days and I've found these two issues:

1. 2020 testing Day 1:

Traceback (most recent call last):
File "/home/andreak/src/Python/F1data/getDataFromF1.py", line 52, in
ff1.get_session(2020,'testing',gp).load_laps()
File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/utils.py", line 127, in decorator
laps = func(*args, **kwargs)
File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 346, in load_laps
self.laps = Laps(self._load_summary())
File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 380, in _load_summary
data, _ = api.timing_data(self.api_path)
File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/api.py", line 123, in timing_data
drv_laps_data = _laps_data_driver(resp_per_driver[drv], empty_laps, drv)
File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/api.py", line 257, in _laps_data_driver
drv_data['PitOutTime'][lapcnt+1] = _to_timedelta(time) # add to next lap
IndexError: list assignment index out of range

2. 2019 Abu Dhabi Grand Prix Practice 3:

Traceback (most recent call last):
File "/home/andreak/src/Python/F1data/getDataFromF1.py", line 28, in
ff1.get_session(2019,gp).get_practice(3).load_laps()
File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/utils.py", line 127, in decorator
laps = func(*args, **kwargs)
File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 347, in load_laps
telemetry, lap_start_date = self._load_telemetry()
File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 444, in _load_telemetry
self._augment_position()
File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 603, in _augment_position
driver_ahead = self._make_trajectory(lap)
File "/home/andreak/.local/lib/python3.8/site-packages/fastf1/core.py", line 735, in _make_trajectory
dmap[:, index] = fix_suzuka(projection_index.copy(), reference_s)
ValueError: could not broadcast input array from shape (49028) into shape (49029)

[theOehrly]

Ahhh, I forgot to test the testing days. So all in season events should not crash.

I spotted the BaseException error too already. It's going to be fixed soon(ish) depending on how much time I have.

[Xwang1976]

There is also the 2019 Abu Dhabi Grand Prix Practice 3 which give a ValueError. Does it happens to you too?

[theOehrly]

Yes it does. I didn't know that one though because I forgot that last season had 21 races not just 20.
It looks like the same problem as with the test session.