graduate card when user presses again or hard and has 0 learning steps

[L-M-Sherlock]

source: https://forums.ankiweb.net/t/fsrs-5-support-for-ankidroid/48328/10?u=l.m.sherlock

[brishtibheja]

This is a great change for people without steps. But perhaps the same logic should apply when you only have 1 step; will this work for those situations?

[mlidbom]

I'm wondering the same thing. I'm the one that posted the suggestion that hard should graduate when on the last step : https://forums.ankiweb.net/t/option-to-have-hard-be-a-passing-grade-for-the-last-or-single-re-learning-step/48313

If this fixes that I'll be doing a happy little dance :)

[brishtibheja]

This is for when you don't have any steps at all. I think your case was Hard graduating cards when no more steps left.

[mlidbom]

That's right. But peeking at the code ctx.steps.hard_delay_secs(self.remaining_steps) it seemed like it might just take that case into account. I'm not familiar with the code though so I'm anything but sure, hence my chiming in. I was just not able to resist commenting when that line got my hopes up. I want that change pretty badly.
Edit: And I do use only one step :)

[Expertium]

@dae we could only allow this for FSRS, but then that would confuse users. With SM-2, empty field = 1m. With FSRS, empty field = no steps. So it's probably better to have the same behavior for everything.
Personally, I really hope that you will merge this PR. Disabling learning steps is requested often enough that I'm sure plenty of people will find this useful. And it doesn't break the workflow for those who don't leave the field empty (which is the majority of users).
EDIT: as another user on Discord succinctly put it: "it's more customizability for no extra complexity"

[dae]

With FSRS on, Again/Hard are at least different to Good, which is not the case for SM2. But they're also the same - 1d/1d/4d/16d with the default parameters. Would it make sense for Hard to choose a middle value (e.g. 2 or 3 days) instead?

3 buttons with the same value is not great in SM2, but given this will only happen if the user removes all their learning steps, this is not a dealbreaker.

[Expertium]

Would it make sense for Hard to choose a middle value (e.g. 2 or 3 days) instead?

With FSRS? No, because then it would deviate from the optimal interval. You could enforce Again < Hard < Good < Easy, by making it so that Hard is at least Again + 1, and Good is at least Hard + 1, etc. But that would also deviate from optimal intervals.
Or you could rewrite all of the code to allow FSRS to schedule <1d intervals ¯\(ツ)/¯
Unrelated, but Dae, can you tell us when the next Anki version will come out? FSRS-5 is ready, Load Balance is ready, Easy Days and new FSRS stats are...not ready, but they will be in a month or even 2-3 weeks, probably. Idk if there is some other stuff you want to see done before the next major release. Anyway, my question is: when is the next major release?

[L-M-Sherlock]

FSRS-5 is ready

Please don't ignore #3369. It will fix a regression caused by FSRS-5.

Or you could rewrite all of the code to allow FSRS to schedule <1d intervals

As I mentioned in Discord, it's possible. But it is based on current PR.

if there is some other stuff you want to see done before the next major release.

Maybe the simulator is included?

[dae]

With FSRS? No, because then it would deviate from the optimal interval.

I think we should strike a balance between mathematical ideals, and what humans expect. We already deviate from ideal intervals with fuzz & load balancing. When you give the user 4 buttons and multiple buttons do the same thing, or are not strictly ascending, users tend to think something is wrong.

A beta is not too far off, but I want to get some more of the recent pending PRs in first.

[mlidbom]

We already deviate from ideal intervals with fuzz & load balancing.

However, fuzz and load balancing are both aimed at improving studying results by taking into account important factors other than the optimal intervals of individual cards. I don't see any such advantages here though...

[Expertium]

Also, enforcing Again < Hard < Good < Easy could result in unacceptably large deviations from the optimum, much larger than fuzz. Absolute worst case scenario: Easy could be 4 days instead of 1 day, 4x longer than necessary.
But anyway, I think right now this deserves more attention: #3375

[L-M-Sherlock]

The current PR is the prerequisite of #3375. So I hope we can focus on here. By the way, I will have a 10-day travel starting from 28th Aug. I'm not sure whether I'm available to improve this PR before the next release.

[brishtibheja]

I agree with @dae on this one. Instead of defending this behaviour, I think a good way of making this more intuitive should be thought of before merging. Although, I'm pretty sad dae never considered default 2-button mode as a potential solution to such problems.

[Expertium]

I think a good way of making this more intuitive should be thought of before merging

That depends on the other PR. If FSRS is allowed to schedule <1d intervals, the problem is solved. Though, that introduces another problem: what if the first interval is <1d but relatively long, like 12 hours? Solution: introduce a cap, such that any <1d interval longer than the cap gets rounded to 1d.
For example, if the cap is 4h, then any interval <4h will be scheduled as such, and any interval longer than that but still shorter than 1d is scheduled as 1d.
But then if both Again and Hard are between 4h and 1d, well...
Oh, and there is another problem. Sometimes S0 is inconsistent, for example, S0(Again)>S0(Hard). FSRS doesn't allow that, so it makes both of them the same. So even if FSRS is allowed to schedule <1d intervals, it's still possible that Again and Hard will be the same.

[brishtibheja]

We already do that sort of thing with day rollovers. If a card is due 5 hours later and 5 hours later is "tomorrow", it is automatically converted to a 1 day interval.

Though, that introduces another problem: what if the first interval is <1d but relatively long, like 12 hours

I don't think this is that much of a problem (but this conversation would belong there).

So even if FSRS is allowed to schedule <1d intervals, it's still possible that Again and Hard will be the same.

Expertium proposes a solution, then he says there is no solution 😄

I actually think we should,

1. Don't convert delays to days after day rollovers or at least visually distinguish them by showing next time in hours and not days.
2. If Again and Hard intervals are same, preferably change something in Hard (as Hard/Easy are least used buttons).

I also doubt how much of this would be a problem for "optimal intervals" as most people use desired retention pretty subjectively. FSRS came, and I think practically more people have ditched the idea of "optimal retention rate" etc.

[Expertium]

Expertium proposes a solution, then he says there is no solution 😄

Stuff is complicated, man. I can't think of a way to let FSRS do what it can do in the best way possible AND solve the Again = Hard (or even Again = Hard = Good in a really bad case) problem.

[mlidbom]

To my mind, trying to make spaced repetition intervals intuitive to the users is a counterproductive idea simply because it cannot be done. I believe they will always be unintuitive to the vast majority of people, me included. I don't think we should intentionally do sub-optimal scheduling trying to do the impossible. I think we should always do as optimal scheduling + fuzz + balancing as we can, and minimize user confusion in other ways.

Possible ways of minimizing confusion:

• Putting the message that intervals are unintuitive and you need to just trust them (unless your statistics show you have a real problem) front and center in the documentation, in the UI, and when answering confused users.
• Making the option to show the interval only available as a command line parameter or something similar, so that the great majority of users are never tempted to use it and be confused by it.
• Alternatively, strongly warning, in the UI, against showing the intervals as showing them is virtually guaranteed to be quite detrimental to your studying because.
  • You are not better at calculating intervals than the algorithms, so you are virtually guaranteed to make your intervals worse if you second guess the algorithms.
  • The statistics is where you see if you have a problem, not the intervals of individual cards
  • The time spent second guessing the algorithm is time not spent studying and it can easily add up to a really major detriment.
  • The concentration vital to effective studying is constantly disrupted by task switching to second guessing the algorithm.
  • Useful solutions to scheduling problems lies in settings, not in second guessing the algorithm on a per card studied basis.

And, of course, in this specific case doing optimal scheduling will be more intuitive, if we allow sub-day intervals to be set by FSRS then again < hard < good < easy

I think we are tying ourselves into logical knots here trying to solve an impossible problem. I suggest we give up on that and do what can be done as well as we can instead.

[brishtibheja]

I believe they will always be unintuitive to the vast majority of people, me included.

I think they are pretty intuitive in one sense.

I don't think we should intentionally do sub-optimal scheduling trying to do the impossible.

The point here is, even though parameters are optimised for your collection you still freely control your desired retention. And the vast majority of people, they do not set "optimal" intervals. They simply go with 0.9 DR. It is not a very convincing argument, when we don't even know how this is going to affect being "optimal". Is it going to reduce people's retention bringing it closer to CMRR recommendations?

---

Re: don't show intervals

The intervals are part of the design that clue new users into what the App does. To quote what David from AnkiDroid quotes,

Instructional onboarding should not be used to supplement poor design. Resources are better spent for making a UI more usable than for creating instructional content.

https://www.nngroup.com/articles/mobile-app-onboarding/

If you do not have the intervals people wouldn't know what the buttons are exactly doing.

[Expertium]

@L-M-Sherlock I think in order to minimize the confusion, we'll have to change the current behavior of FSRS-5 (and rebenchmark it). Currently, if S0(Again)>S0(Hard), FSRS sets both of them to the same value, whichever one is based on more data. If S0(Again) is based on more reviews, then the optimizer sets them both to S0(Again). But that can result in Again = Hard, or even Again = Hard = Good, or even Again = Hard = Good = Easy if the user is smoking crack and using the buttons in the most incorrect way imaginable.
Let's change that: instead, just sort all 4 S0 values and assign them to Again/Hard/Good/Easy after sorting. The lowest value will correspond to S0(Again), the highest value will correspond to S0(Easy). So if initially S0(Again) > S0(Hard), this will be fixed after sorting, and S0(Again) will be smaller than S0(Hard). This will probably be slightly less accurate, but it solves the Again = Hard problem.
And as for clamping (for gradient descent), I guess we could do something like this:

            w[0] = w[0].clamp(S_MIN, 100)
            w[1] = w[1].clamp(w[0] * 1.05, 100)
            w[2] = w[2].clamp(w[1] * 1.05, 100)
            w[3] = w[3].clamp(w[2] * 1.05, 100)

This ensures that all 4 values are distinct.
EDIT: I made a PR.
EDIT 2: I'll let LMSherlock handle it, seems like my idea wasn't good.

Here's the second problematic part: imagine that it's 5 hours until the day rollover. Again is 6 hours, Hard is 12 hours. So they both get rounded to 1d. So we still get the same problem. I don't know how to solve this one, tbh.

[mlidbom]

If you do not have the intervals people wouldn't know what the buttons are exactly doing.

Only if we believe that the user is better at calculating intervals than the algorithm does showing the intervals help the user make a good choice. I don't believe that, so the information in the linked article does not support showing the intervals to my mind.

Assuming the user is not better at calculating intervals, what Anki needs to help the user optimally is accurate information on what the user experienced when trying to recall the card. Thus the user should be guided to simply honestly select the button that matches what happened when they tried to recall the card. We could show something similar to this:

Button text: Hover text (Could be shown beneath the buttons on mobile with an option to hide them once familiar with them.)
Failed: I failed to recall the information correctly
Struggled: I recalled correctly but struggled to do so.
No biggie: I had no great difficulty.
Easy: I hardly had to try. I just knew.

When we show the intervals we are much less likely to get honest and consistent button presses, instead the button is quite likely pressed because the user intuitively liked the interval they saw above that button best. This is unlikely to be doing the user any favors for all of the reasons I listed in my previous comment. Unless we believe the user is not only better at calculating intervals than the algorithm, but that they do so with negligible cost in time, effort and distraction from actually studying.

I don't believe any of that is the case.

[Expertium]

Button text: Hover text (Could be shown beneath the buttons on mobile with an option to hide them once familiar with them.)
Failed: I failed to recall the information correctly
Struggled: I recalled correctly but struggled to do so.
No biggie: I had no great difficulty.
Easy: I hardly had to try. I just knew.

We'll add something like that to the manual. I have proposed adding it to Anki itself, Dae didn't like it.

[brishtibheja]

@mlidbom The goal is to clue users into what the app is doing. I'm repeating myself. The intervals don't help in selecting a grade. The intervals let people know Anki's buttons are scheduling cards into the future. This shouldn't be an assumption that new users will come with pre-existing knowledge of SRS systems. Don't argue this point. I didn't know about SRS when I started Anki, and a common question among new users is "why my cards have disappeared?!!".

This helps in new users' onboarding. Particularly if they are from outside the SRS niche.

[mlidbom]

@brishtibheja

This shouldn't be an assumption that new users will come with pre-existing knowledge of SRS systems. Don't argue this point.

I won't, I agree.

The intervals let people know Anki's buttons are scheduling cards into the future.

That information could be shown instead. Showing that information would come with none of the problems that showing the intervals does.

Imagine something like this UI welcoming a new user on the backside of cards:

When you press the buttons here Anki will schedule this card for you to see it again later in a way that lets you remember it with minimal time spent studying. Just answer honestly according to these simple guidelines and Anki will take care of the rest:

Failed: I failed to recall the information correctly, or this was a new card with information new to me.
Struggled: I recalled correctly but struggled to do so.
No biggie: I had no great difficulty.
Easy: I hardly had to try. I just knew.

[Please hide this information, I got it now]

Hitting the bottom button might leave a small question mark. If one clicked that that the text would reappear.

Finally, no one has addressed the problems I see with showing the intervals. If you disagree that those are significant problems, I would really appreciate hearing your reasoning.

[brishtibheja]

I think all of this is outside the scope of this PR. I won't discuss UI changes here.

[mlidbom]

Dang, I seem to have failed to make it clear why I am bringing this up here, annoying people with a seemingly irrelevant discussion. Sorry about that.

The reason I brought it up is that, to the best of my understanding, this pull request is being blocked by the consequences of showing the intervals in the UI and changes are being discussed that would prevent FSRS from scheduling cards to the best of its ability due to this. That is why I brought this up here.

So far I seem to just be annoying people though. I'm sorry about that too.

I think I've made my main points pretty clearly already. If nothing I've said resonates with anyone here I don't see much point in me saying more on the topic. Thank you all for your time.

[Expertium]

@dae even if me and @L-M-Sherlock find a way to make sure that Again < Hard < Good < Easy within FSRS, it could still lead to problems because of the day rollover.
Example I used above: imagine that it's 5 hours until the day rollover. Again is 6 hours, Hard is 12 hours. Then they both will be rounded up to 1d. I don't know how to solve this one, so I reluctantly agree to accept adding 1 day to Hard if it's equal to Again as a solution (same for Good and Easy). It's suboptimal, but if that's what it takes to make you merge this PR, then fine.
In case you are confused why I'm talking about <1d intervals in this PR: #3375. I'm assuming that if you merge this PR, then you will also merge that other PR.

Just to make everything absolutely clear, there are several possible cases:

​1)​ Again < Hard < Good < Easy, no <1d intervals. Everyone is happy.

​2)​ One of the intervals is equal to another one, like Hard = Good or Again = Hard, no <1d intervals. Then add 1 day to the interval that corresponds to the higher grade. If Again = Hard, add 1 to Hard. If Hard = Good, add 1 to Good. If Good = Easy, add 1 to Easy.

​3a)​ Again < Hard < Good < Easy, there is at least one <1d interval, but the time until the day rollover is longer than the interval. For example, Again = 2h and it's 5 hours until the day rollover. Again = 2h.

​3b)​ Again < Hard < Good < Easy, there is at least one <1d interval, but the time until the day rollover is shorter than the interval so the interval gets rounded up to 1d, but the interval of the next grade is greater than 1d.
Example, Hard = 2d, Again = 6h and it's 5 hours until the day rollover, so it's rounded up to 1d. Again = 1d, Hard = 2d.

​4)​ There is at least one <1d interval, but the time until the day rollover is shorter than the interval so the interval gets rounded up to 1d, and the interval of the next grade is the same. Then add one to the interval of the higher grade.
Example, Hard = 1d, Again = 6h and it's 5 hours until the day rollover, so it's rounded up to 1d. Again = 1d, but Hard is also 1d. Then add 1 to Hard. This basically gets reduced to number 2, which I described above.

Currently, learning steps have 2 issues: they add extra cognitive load (user has to think and worry about setting them), and they don't correspond to what FSRS considers to be optimal. Merging this PR and #3375 would solve both of those problems. This is going to be a bit messy, but this (or something similar) is better long-term, even with all the mess and tons of edge cases.

[YukiNagat0]

@Expertium, @L-M-Sherlock, is the ultimate goal to completely remove learning steps??
If this is indeed the case, then I really think that the Anki development community has chosen the wrong path of development.
This is no longer a simplification, but a complete destruction of the Anki's values, of what it is loved for.
Literally no one is requesting to remove learning steps, then why you are trying to do this?
Just because you think, that it is more "intuitive"?
For me it is seem, that you are just trying to make a memrise from Anki.

[L-M-Sherlock]

is the ultimate goal to completely remove learning steps??

IMO, of course not. I just think it's weird that Anki still gives me learning steps even when I set the learning step field to empty.

[mlidbom]

What about users that disable the "Show next review time above answer buttons" option?
When a user empties out learning steps, and hides the intervals in the UI, couldn't this allow FSRS to control short term scheduling freely to the very best of its current and future ability?

[YukiNagat0]

is the ultimate goal to completely remove learning steps??

IMO, of course not. I just think it's weird that Anki still gives me learning steps even when I set the learning step field to empty.

Phew, that's a relief. Yeah, completely agree - if user decides to empty learning step field, then, by doing it, he gives full control over intervals to FSRS, including short term scheduling.

[brishtibheja]

Medical students learn a lot of meaningless stuff requiring several repetitions per day to get it. It would be not prudent to remove them completely at this point.

The current defaults also mean less people will frustrate at "why my intervals are so long!".

[Expertium]

is the ultimate goal to completely remove learning steps??

If by "remove" you mean "let FSRS control them", then yes.
If by "remove" you mean "no <1d intervals under any circumstances ever", then no.

[YukiNagat0]

is the ultimate goal to completely remove learning steps??

If by "remove" you mean "let FSRS control them", then yes. If by "remove" you mean "no <1d intervals under any circumstances ever", then no.

@Expertium, from what you are saying it sounds like you want to remove/delete the Learning steps field altogether - that is the mistake.
User must have the right to set learning steps.
Only if user decides to empty Learning steps field, then and only then FSRS is granted the right to fully control all intervals (short and long term).

[brishtibheja]

(ping @Expertium)

make sure that Again < Hard < Good < Easy within FSRS, it could still lead to problems because of the day rollover.

The day rollover adds to the problem, true.

In case you are confused why I'm talking about <1d intervals in this PR: #3375. I'm assuming that if you merge this PR, then you will also merge that other PR.

The day rollover bit requires bringing up the other PR. It would've been nice to have them both in the same PR.

---

1. No subday intervals and intervals are in proper order.

blank

2. No subday intervals but at least two intervals match.

Is there any reason subtracting days haven't been considered? If this is hard to do in code, sure it's a bad idea. But ideally, if you are dealing with a person who keeps DR to CMRR recommendations, you're better off subtracting a few days than adding a few. Also, consider the opposite situation of a person with 0.99 DR.

I'm not necessarily suggesting peeking at CMRR. In a simple situation, we can simply look at DR and have 0.86 or 0.9 as threshold.

3. a. At least one subday interval but it doesn't cross day boundary. Intervals are in order.

blank

3. b. At least one subday interval that crosses day boundary. Intervals would still be in order.

blank

4. At least one subday interval that crosses day boundary. At least two intervals will match.

In this situation, we end up with two intervals that match. In your examples, an easier solution seems subtracting a few hours from the lower interval so that it doesn't cross the day boundary. Sure, this solves the optics problem we dont want and still allows the user to study at intervals closer to what they want.

But consider another situation. Again = 12h and Hard = 23h at 3:00 AM. In this situation, converting Hard to 2d is optimal instead of trying to make Again not cross the day boundary. This is because Hard is closer to a day boundary compared with how close Again is to a day boundary.

I am not completely sure about this situation though. For example, the level of precision I'm seeking would require answering whether 1d means a >24h interval or not.

---

I agree long term a lot more polish can be added here. Releasing this also means public feedback, which is good.

[Expertium]

Yuki

Only if user decides to empty Learning steps field, then and only then FSRS is granted the right to fully control all intervals (short and long term).

That's how it will be after this PR and the other PR are merged. But this isn't ideal. FSRS is better at determing the optimal interval length than the user, so the ideal solution would be to remove the field and just let FSRS handle it. If FSRS decides that a <1d interval is needed - ok.

sorata
Subtracting a few hours seems like a better idea in some cases, I agree.

[brishtibheja]

Let's focus on what's at hand for now and not talk about distant goals. It will be futile anyway.

[mlidbom]

Subtracting a few hours seems like a better idea in some cases, I agree.

Agreed, but also, if the user has chosen not to show the intervals above the buttons they cannot be confused by them and FSRS can be free to schedule optimally without any of these adjustments. Right?

Meaning that even if no satisfactory solution for how to handle this for users that show the intervals can be agreed upon, this can still be merged and enabled for users that don't show the intervals. Right? Which would enable the FSRS developers to get the data they need to further improve FSRS and let users that hide the intervals receive the full benefit of FSRS current abilities.

I don't see a downside. If I'm missing something that makes this a terrible idea, clue me in and I'll shut up.

[brishtibheja]

The intervals appear in stats/card info and people see them working in practice. Besides, it's already getting complex enough and this idea wouldn't be easy to document.

[user1823]

FSRS is better at determining the optimal interval length than the user

I agree that this is true for the long term intervals. But, for short-term intervals, this is NOT true.

FSRS was originally designed for long-term scheduling. Though L-M-Sherlock has added a short-term component in FSRS 5, it is mostly aimed at calculating the effect of the short-term reviews on DSR and not for scheduling short-term intervals.

He has also stated that modeling short-term memory requires more data and that such data is limited.

Besides, most of the researchers have focused on long-term memory rather than short-term memory. So, there is a very sparse knowledge base to build upon.

So, it can be said that FSRS may not be better at calculating short-term intervals than the user. It may even be worse.

All of this implies that it won't be prudent for Anki to remove the (re)learning steps setting, at least not in the near future.

However, if the user decides to clear the (re)learning steps field, then we can allow FSRS to control the short-term scheduling. In that case, it will be more like an experimental version of short-term FSRS scheduler.

PS:
With this comment, I don't want to undermine the efforts put into FSRS 5. Remember that the goal of FSRS 5 was to determine the impact of short-term reviews on the long-term memory and not to use FSRS for scheduling short-term reviews.

[brishtibheja]

I thought the contributing files said "Do One Thing" 😄

[dae]

This PR bumps the FSRS version. It should make the relevant changes on Anki's end at the same time. The error message should have ideally been updated in #3298, but was missed then.

Thank you to everyone who contributed to the discussion. I still think showing multiple buttons with the same intervals is not great, but as this will only happen when the user removes their learning steps, I'm not going to let it block a merge.

[brishtibheja]

@dae I think the same intervals issue needs to be done someday. maybe an issue to track this?

[Expertium]

I've said this before, I think it's ok to do Again + 1d <= Hard + 1d <= Good + 1d <= Easy. In other words, make sure that each interval is different by at least one day. It's suboptimal, but oh well.
And we'll need to do the same thing, but with hours for #3375