Opened 10 months ago

Last modified 10 months ago

#28176 assigned task

Cleanup and add the testsuite to tor-browser-build.git

Reported by: boklm Owned by: boklm
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-rbm, tbb-qa, TorBrowserTeam201811
Cc: tbb-team Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Currently we have a Tor Browser testsuite in a separate git repository, at https://gitweb.torproject.org/user/boklm/tor-browser-bundle-testsuite.git/.

Rather than using a separate repository, I think it might be better to integrate that testsuite directly into tor-browser-build.git, as a sub-directory:

  • quite often some Tor Browser changes need corresponding changes in the testsuite. Having both in the same repository would make it possible to do both changes with the same commit (although we might still need two commits when the change is in tor-browser.git), and use the same review process.
  • some people don't know that we have a testsuite, or where it is. Having it in the same repository would make it easier to find and know that it exists.
  • having the testsuite in the same repository would make it easier to integrate it with the build process. For example we could add some makefile command such as make run-testsuite-testbuild, which would make it easier for developers to run the testsuite after doing a build.

One option would be to copy the current testsuite to some tor-browser-build.git sub-directory, and make necessary changes to fix it and integrate it better. However I think that over time the testsuite accumulated a lot of complexity that is not really needed. Adding the current testsuite and fixing it will probably result in adding even more complexity. I think that an other option is to do some important cleanup first and only import minimal testsuite tools, and then cleanup and import each of the tests one by one.

So I think the main two options are:
1) copy the current testsuite, fix all the tests, then try to integrate it better, and to clean it.
2) rewrite minimal testsuite tools, avoiding the complexity from the previous version, and import each test one by one after fixing and cleaning them.

To me it seems easier to re-start from a minimal testsuite and re-add the needed features and tests (sometimes copying them from the old testsuite, and sometimes re-writting them from scratch if it makes more sense) than taking the current testsuite and trying to clean it little by little, so I think option 2) is better.

Child Tickets

Change History (2)

comment:1 Changed 10 months ago by gk

Could you elaborate a bit what made the previous version too complex? How would minimal testsuite tools look like compared to what we have today?

I am not really convinced yet that we should put the whole testsuite stuff into our build stuff. My first reaction was "What do the tests we have to do with producing builds and configs/scripts defining those?". It does not speak anything against make run-testsuite-testbuild with a different repo, for instance. One could just clone the testsuite repo somewhere as a dependency during and then use that for this make target (like we already do with other dependencies). I think having such a make target is a good idea anyway, but for the cases where one needs to run the tests in addition to a build having an external repo should be working as well. That's for your third argument.

I am not overly worried about the scenario in the second one: it's easy to point folks writing patches to that repo or to explain how to run the testsuite somehwere. Devs should be used to that kind of thing.

Regarding the first one: I at least have not forgotten about the problem of making corresponding changes. We currently just don't have a policy of enforcing the rule of making changes to the code and, if needed, to the testsuite. I have the feeling most of the changes are in non-tor-browser-build repos, so I am not so sure that your proposed change is good for your first reason. At any rate, I think the right fix for that problem is to a) document exactly what we are testing right now and then enforce the policy that all new changes need to pass our tests, i.e. that tests need to get adapted when landing changes where needed. But for the latter to be effective we need to start from some baseline, meaning a running testsuite. :)

comment:2 Changed 10 months ago by gk

Keywords: TorBrowserTeam201811 added; TorBrowserTeam201810 removed

Moving our tickets to November.

Note: See TracTickets for help on using tickets.