Test Icons Not Updating when using pytest

[aaronsmith1234]

Issue Type: Bug

Hi,
While using the built-in Python test tab in VSCode, occasionally the test status icons (green checkmark or red X) get out of sync with the actual test results. As an example, I have tests that were once failing, and are now passing, but the icons still are red X's; when running tests, the icons dont update to be the spinning arrows, and after the tests are done, they are what they were before. At the bottom of the screen, the test status is correct, and inside the test file the test status is correct, its just the test explorer tab.

Note that closing VSCode entirely and re-opening it fixes the problem. I can't reliably reproduce, it just happens sometimes on some projects

Extension version: 2021.1.502429796
VS Code version: Code 1.52.1 (ea3859d4ba2f3e577a159bc91e3074c5d85c0523, 2020-12-16T16:34:46.910Z)
OS version: Windows_NT x64 10.0.18363

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-8365U CPU @ 1.60GHz (8 x 1896)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: disabled_off opengl: enabled_on protected_video_decode: enabled rasterization: enabled skia_renderer: disabled_off_ok video_decode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.75GB (6.86GB free)
Process Argv	--crash-reporter-id 72142afe-7e98-4075-91ad-09c713b0d234
Screen Reader	no
VM	0%

A/B Experiments

vsliv368:30146709
vsreu685:30147344
openlogontheside:30221877
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492:30211401
wsl2promptcf:30224613
pythonvsdeb440:30248342
unusedpromptcf:30224611
folderexplorer:30224614
openfilemenu:30224647
pythonvsded773:30248341
pythonvspyt600cf:30247032

[kimadeline]

Hi @aaronsmith1234,

Thanks for the bug report! We investigate issues in order based on priority and severity, which includes the impact it has on your ability to use the extension to do productive work, and the number of people affected. If other users come forward and leave a comment demonstrating they are seeing/reproducing the problem then we will raise this issue's priority.

If you do end up having a sample project that we could use to reliably repro this issue, along with extra info like which test framework you're using, or your general workspace setup, that would be very useful 🙂

Thank you for your understanding and patience!

[aaronsmith1234]

For sure! General workspace settings is minimal; really only saying what test framework I'm using (Pytest)

[rubenvdg]

Same here, the pytest icons are stale:

while all tests are actually passing:

[c-byte88]

Also experiencing the issue, icons don't update from ? when running using pytest.

Test running

Showing as passed in test output:

Environment Details

Running attached to a container via wsl2 (ubuntu-20.04)

Extension version: v2021.4.765268190
VS Code version: Code 1.52.2
OS version: Windows_NT x64 10.0.19042

Using Remote settings:

{
	"python.formatting.provider": "yapf",
	"python.formatting.yapfArgs": [
		" — style",
		"${workspaceFolder}/setup.cfg"
	],
	"python.linting.pylintArgs": [
		"--rcfile=${workspaceFolder}/setup.cfg"
	],
	"python.linting.mypyArgs": [
		"--config-file=${workspaceFolder}/setup.cfg"
	],
	"python.linting.enabled": true,
	"python.linting.pylintEnabled": true,
	"remote.autoForwardPorts": false,
	"remote.restoreForwardedPorts": false,
	"terminal.integrated.shell.linux": "/bin/bash",
	"python.languageServer": "Jedi",
	"python.jediEnabled": true,
	"python.autoComplete.addBrackets": true,
	"python.linting.mypyEnabled": true,
	"python.linting.flake8Enabled": true,
	"python.testing.pytestEnabled": true,
	"python.testing.pytestArgs": [
		"--rootdir=./src/python/"
	],
	"python.linting.banditEnabled": true,
	"python.formatting.blackPath": "/opt/conda/envs/xxxxxxxxx/bin/black",
	"python.linting.pylintPath": "/opt/conda/envs/xxxxxxxxx/bin/pylint",
	"python.pythonPath": "/opt/conda/envs/xxxxxxxxx/bin/python",
	"python.venvPath": "/opt/conda/envs/xxxxxxxxx/lib/python3.7/site-packages/pyspark/jars",
	"python.formatting.autopep8Path": "/opt/conda/envs/xxxxxxxxx/bin/autopep8",
	"python.formatting.yapfPath": "/opt/conda/envs/xxxxxxxxx/bin/yapf",
	"python.linting.banditPath": "/opt/conda/envs/xxxxxxxxx/bin/bandit",
	"python.linting.flake8Path": "/opt/conda/envs/xxxxxxxxx/bin/flake8",
	"python.linting.mypyPath": "/opt/conda/envs/xxxxxxxxx/bin/mypy",
	"python.linting.pycodestylePath": "/opt/conda/envs/xxxxxxxxx/bin/pycodestyle",
	"python.envFile": "${workspaceFolder}/.env",
	"files.watcherExclude": {
		"**/build/**": true,
		"**/dist/**": true,
		"**/.ipynb_checkpoints/**": true,
		"**/*.egg-info/**": true,
		"**/.pytest_cache/**": true,
		"**/__pycache__/**": true,
		"**/.mypy_cache/**": true,
		"**/.venv/**": true
	},
	"files.exclude": {
		"**/.pytest_cache/**": true,
		"**/.mypy_cache/**": true,
		"**/__pycache__/**": true,
		"**/*.egg-info/**": true
	},
	"workbench.editorAssociations": [
		{
			"viewType": "jupyter.notebook.ipynb",
			"filenamePattern": "*.ipynb"
		}
	],
	"jupyter.defaultCellMarker": "# COMMAND ----------",
	"jupyter.codeRegularExpression": "^(#\\s*%%|#\\s*\\<codecell\\>|#\\s*In\\[\\d*?\\]|#\\s*In\\[ \\]|#\\s*Databricks\\snotebook\\ssource)",
	"jupyter.jupyterServerType": "remote",
	"jupyter.allowUnauthorizedRemoteConnection": true,
	"jupyter.alwaysTrustNotebooks": true
}

[kimadeline]

Does any of you have a sample repo I could use to try and reproduce the issue? Also does it happen every time, or only with specific projects or configurations?

[c-byte88]

@kimadeline I've included the docker setup as it was pertinent to my environment. Thanks

https://github.com/nigelc-byte88/vscode-pytest-bugsample

This is happening every time on this project (the full version) and every project that follows this particular template layout. (i.e. using docker dev container)

[aaronsmith1234]

@kimadeline I've had it happen in a variety of projects and configurations, but haven't been able to reliably reproduce 😢Reloading the window always fixes it for me

[followben]

@kimadeline - did you manage t repro with @nigelc-byte88's repo?

I'm seeing the same here, but not using remote testing:

vscode:
054a9295330880ed74ceaedda236253b4f39a335
Date: 2021-05-12T17:44:30.902Z (1 wk ago)
Electron: 12.0.4
Chrome: 89.0.4389.114
Node.js: 14.16.0
V8: 8.9.255.24-electron.0
OS: Darwin x64 20.3.0

python extension: 2021.5.2

[kimadeline]

@followben Do you have a sample repo I could use as well?

[kimadeline]

Thank you all for your patience!

@nigelc-byte88 I sadly couldn't reproduce the issue with repo:

I followed these steps:

• cloned the repo
• installed pylint, flake8, yapf, xmlrunner, pytest
• opened the root folder
• ran test discovery with src as the source repo

And the tests were discovered and ran when using Python 3.9.0 and pytest 6.2.4. Are there any steps I am missing?

[followben]

Hi @kimadeline - for my case at least (pytest in venv under python 3.8+ running on macOS) it seems to be related to the addition of python.testing.pytestArgs. I've created a repo that reproduces the issue here: https://github.com/followben/vscode-pytest-icons

[c-byte88]

Thanks for checking @kimadeline, we use wsl/docker and use the vscode settings included in the repo here. Which does includes python.testing.Pytest args, so would align with @followben hypothesis.

[luabud]

This is fixed by #16769 😊