perf: Avoid updating the folder size if we know the size difference

perf: Avoid updating the folder size if we know the size difference #49701