Visual Studio Bug: Clicking on test result sometimes kills VS2019

[bwandersen]

Hi

We are using ::testing::RegisterTest to dynamically register test cases. The cases show up in the Test Explorer and can be run from here without any issue. However, if you click on the test result, then Visual Studio gets stuck and has to be killed from the task manager. This works fine in Visual Studio 16.9, but no longer works in 16.10.

Now the strange part is that on machines with Resharper enabled, this issue goes away. Disabling Resharper makes the problem appear again. Our guess is the recent changes to the test explorer has introduced some minor discrepancy between google test adapter and the vs test explorer.

Does anyone know what the problem might be?

Here is an example that reproduces the problem:

`#include "stdafx.h"
#include <gtest/gtest.h>

#ifdef _MSC_VER
// Windows service specifics
#define WIN32_LEAN_AND_MEAN
#define NOMINMAX 1
#include <windows.h>
#endif

#include

using namespace std::string_literals;
using ::testing::ValuesIn;
using ::testing::Values;
using ::testing::Return;
using ::testing::An;
using ::testing::_;

class klXlTestRunner : public testing::Test
{
protected:

static void SetUpTestSuite() 
{
}

static void TearDownTestSuite() 
{
}

void SetUp() override
{
}

void TearDown() override
{

}

};

class XlTest : public klXlTestRunner {
public:
explicit XlTest(int ) : data(_) {}
void TestBody() override
{
}

private:
int data_;
};

bool RegisterXlTests()
{

std::string suite("suite");
std::string name("name");

::testing::RegisterTest(suite.c_str(),
	name.c_str(),
	nullptr,
	nullptr,
	__FILE__,
	__LINE__,
	[=]() -> klXlTestRunner* { return new XlTest(0); });

return true;

}

GTEST_API_ int main(int argc, char** argv) {
printf("Running main() from %s\n", FILE);
if (RegisterXlTests())
{
testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
return 0;
}`

[csoltenborn]

Sorry for the late answer, and thanks for the code to reproduce the issue! I assume that with "click on the test result" you mean selecting the test in the test explorer to see the test result below!?

I am on VS 16.10.3 right now. Unfortunately, your code does not compile out of the box. However, the SampleTests solution also contains dynamically registered tests. With those, I can not reproduce the issue, neither with ReSharper enabled nor disabled.

Could you try on your machine? And if the issue does not occur (my tests are registered differently), could you try to integrate your code into the Test project of that solution, check if the problem remains, and then attach the resulting code?

[bwandersen]

Thanks for taking the time to reply. Much appreciated

I assume that with "click on the test result" you mean selecting the test in the test explorer to see the test result below!?

Yes, correct

I am on VS 16.10.3 right now. Unfortunately, your code does not compile out of the box. However, the SampleTests solution also contains dynamically registered tests. With those, I can not reproduce the issue, neither with ReSharper enabled nor disabled. Could you try on your machine?

I build SampleTests solution and looked the Api.Created.Test you refer to. For some reason it will only show the last of the two testcases in the test explorer. You can switch them around, and then you will get the other one. In any case, I took the code above and modified it to run against 1.8.1 (we use 1.10). That worked nicely. The result can be "clicked" and the test case can run (regardless of resharper is on or off). I will try to use the style in the test case in our solution and see how it goes.

If interested, here is the h-file. It needs to be included in Tests project.
DynamicallyRegisteredTest.txt

The function to register the test cases needs to be added to main.cpp
Main.txt

I had change the extension .txt.

Thank you, Bjarke

[LPhil]

I have the same Problem with VS 16.10.3. If you try to access the source over the test explorer, VS stops and only the Taskmanger you help by kill the process away. I first noticed this behavior in 16.10.3.

[csoltenborn]

@bwandersen

I build SampleTests solution and looked the Api.Created.Test you refer to. For some reason it will only show the last of the two testcases in the test explorer. You can switch them around, and then you will get the other one.

I'm not sure what you mean by that. I must admit :-)

In any case, I took the code above and modified it to run against 1.8.1 (we use 1.10). That worked nicely. The result can be "clicked" and the test case can run (regardless of resharper is on or off). I will try to use the style in the test case in our solution and see how it goes.

Note that I'm not sure at all whether this is an appropriate way to create and register tests - I guess I picked up code somewhere just to have something to write tests against. At first sight, your approach seems cleaner (since mine e.g. uses internal API)...

If interested, here is the h-file. It needs to be included in Tests project.
DynamicallyRegisteredTest.txt

The function to register the test cases needs to be added to main.cpp
Main.txt

Yes, I tried the code, and it appears to work just fine - thanks for attaching! That leaves me with a case I cannot reproduce locally, though :-)

[csoltenborn]

@LPhil Is this also true for dynamically created tests in your case, or also other tests? What gtest version are you on? Can you maybe provide some code to reproduce the example, preferably as an extension of the SampleTests solution?

[bwandersen]

I build SampleTests solution and looked the Api.Created.Test you refer to. For some reason it will only show the last of the two testcases in the test explorer. You can switch them around, and then you will get the other one.

I'm not sure what you mean by that. I must admit :-)

There are two test cases in Api.Created.Test, and only one of them shows up in the test explorer. You can change which one shows up by making it the last case in the Api.Created.Test cpp file .. but not it is not important. Sry for the confusion.

In any case, I took the code above and modified it to run against 1.8.1 (we use 1.10). That worked nicely. The result can be "clicked" and the test case can run (regardless of resharper is on or off). I will try to use the style in the test case in our solution and see how it goes.

Note that I'm not sure at all whether this is an appropriate way to create and register tests - I guess I picked up code somewhere just to have something to write tests against. At first sight, your approach seems cleaner (since mine e.g. uses internal API)

I think RegsiterTest is an addition they did for 1.10+. It does almost the same the code in the test case. My code expanded on the example from their documentation Registering tests programmatically. It tried copying the example, and that still has the bug. Resharper still removes the bug when enabled.

If interested, here is the h-file. It needs to be included in Tests project.
DynamicallyRegisteredTest.txt
The function to register the test cases needs to be added to main.cpp
Main.txt

Yes, I tried the code, and it appears to work just fine - thanks for attaching! That leaves me with a case I cannot reproduce locally, though :-)

Yes, sry, but at least it shows that 1.8.1 works. It points to problem between 1.10 and VS 16.10.1-3. I will keep trying to get it working for 1.10, and let you know how it turns out.

[daveabramson]

I have the same issue where selecting the test in test explorer causes VS to hang forever, but it seems to be triggered differently than the dynamically registered test scenario above.

1. Tests defined using TEST do not cause VS to hang when selected in the test explorer
2. Tests defined using TEST_F cause VS to hang when selected in the test explorer

Posted this issue with devcommunity (https://developercommunity.visualstudio.com/t/visual-studio-freezes-when-selecting-test-in-test/1499235?from=email). They asked me to update to 16.11.0, which I did. On first startup after update I got the follow error, that I was not getting on my previous version (16.10.2):

And this message in the tests output pane:

The next time I started after the update, I only got the message in the test output pane.

Also, the same behavior regarding selection of tests in test explorer still exists.

I hope this is useful information.

[csoltenborn]

Thanks! This is indeed useful information, I would say, but probably more for the VS guys then for me - this is almost certainly a VS regression since there have been no changes to GTA for quite some time, and since it also appears to happen for TAfGT, the MS maintained fork of GTA.

Basically, GTA just reports the tests found (and then the tests' results) to the VS test framework - GTA has nothing to do with the Test Explorer. The only thing I could think of is that for a particular kind of tests, some fields of the "report objects" are filled with values which make VS crash recently. This would still be a VS regression, but I might have some influence on that. However, since I was not yet able to reproduce the issue at all, it's probably rather hard for me to figure that one out.

My suggestion would be to also report this at TAfGT's repository or the repository of the test framework itself, and to point to this issue from there...

[luke727]

I also experienced VS2019 locking up while clicking on an individual test in the text explorer. I was able to work around it by changing the Linker -> Debugging -> Generate Debug Info value from /DEBUG to /DEBUGFULL. Hope this helps anyone else running into the same issue.

[csoltenborn]

@luke727 That's interesting, thanks for sharing!

@LPhil @bwandersen @daveabramson Can you confirm this by any chance? I would then add this to the trouble shooting section of the docs (and I would also check whether I can see anything in the code which might cause this).

[bwandersen]

I just checked and the project already had /DEBUG:FULL on. So this sadly does not help on VS 16.10.2. I am going to upgrade to VS 16.11.4 tomorrow. One of my colleagues upgraded to this version and doesn't have the problem. I will let you know how it turns out.

[luke727]

I'm on VS 16.11.3 if that makes any difference.

[csoltenborn]

@bwandersen Any news from your side? Just being curious...