Allow just a single test to be run

Our run_tests.py presently has a '--test' argument that lets you only run just a single test module, or all test modules with a given prefix.

This is nice, but often we want to only run a single test. For example, just test/integ/control/controller.py's test_reattaching_listeners() method.

Presently you could test only this module with...

% ./run_tests.py --integ --test control.controller

But better would be to let you drill down even more like...

% ./run_tests.py --integ --test control.controller.test_reattaching_listeners

added component::archived/stem easy owner::atagar priority::low resolution::fixed status::closed testing type::enhancement labels

Hi!

I was taking a look at this and I have a few thoughts:

• There will be necessary to change the current test loading method (loadTestsFromTestCase) since it needs a valid class to run, and a method will not satisfy that.
• The test loading method I suggest is loadTestsFromName (doc) which needs a string pointing to what we want to test, and that will be perfect for testing single methods.
• With this approach it won't be necessary to dynamically load modules with !import!, since that is handled by loadTestsFromName itself ;)

I've implemented all of the above and I'll reply with the suggested code ASAP!

Trac:
Username: clv

Fantastic, thanks for taking this on!

Hi,

I pushed what I've done here: https://github.com/leivaburto/stem/commit/27c8673552ced6ca9daa3674249daa45698a202e

I basically changed from loadTestsFromTestCase to loadTestsFromName, added a try-except on run_tests and modified _get_tests in util.py in order to:

• Look for the given string (module_substring) in modules list.

• If nothing found, remove last fragment of the string (what is supposed to be a method) and try again. If something is found after that, it will return the module found + the name of the method.

To test the changes I made, I tried with the following arguments:

• --all
• --all --test test.integ.process (process module)
• --all --test process
• --integ --test test.integ.process
• --integ --test process
• --all --test test.integ.process.test_quiet_argument (process module + single method)
• --all --test process.test_quiet_argument
• --integ --test test.integ.process.test_quiet_argument
• --integ --test process.test_quiet_argument

Aditionally, there a couple of things to notice:

This will show a fail msg:

• --integ --test process.test_quiet_arguments ('s' at the end, non existing method)

This will show nothing (as it does now)

• --integ --test processs (triple 's' at the end, non existing module)

Finally, there is one thing I haven't been able to solve: testing a non existing module inside a sub folder. For example:

• --all --test test.integ.descriptor.microdescriptors (extra 's' at the end, non existing module)

As this module is not found, the program will remove the last fragment (microdescriptors) and try to look for test.integ.descriptor, which is the base for 5 sub modules that do exist, and will find all of them and try for the "method" microdescriptors on them, which will finally produce 5 "failed messages". The problem here is that tere is no way of knowing beforehand if the given string corresponds to a submodule or a method... so I've been thinking that there are two possible solutions for this:

1. List all method names in settings.cfg, as it is done now for module names.
2. Add a new argument for testing single methods, so we know beforehand that we were asked for that and not something else. It could be something like --test-method

What do you think?

Trac:
Username: clv

Thanks civ! I was able to simplify this a bit but generally looked great! Merged, let me know if I made any mistakes. :)

Trac:
Status: new to closed
Resolution: N/A to implemented

Hi civ, gonna reopen this. Minor issue I just noticed is that when tests have errors we report 'no such test' rather than a failure. For instance, try making a syntax error in a test module and run it.

This isn't a terribly big whoop (pyflakes catches these too) but probably a regression we should fix. :P

Trac:
Status: closed to reopened
Resolution: implemented to N/A

Hi atagar!

I checked on what you said and I just added a specific except for catching AttributeError in here.

For example, this will report no such test:

• --all --test process.test_quiet_arguments (non existing method)

If there is an error in process test class, then this will report failure:

• --all --test process.test_quiet_argument

Reporting failures will occur before no such test as it first try to load the module and then to look for the method.

Btw, my nick is clv (not with 'i' but 'l') :P

Trac:
Username: clv

Great, thanks! Fix pushed.

Trac:
Status: reopened to closed
Resolution: N/A to fixed

closed