Opened 5 years ago

Closed 4 years ago

#7843 closed enhancement (implemented)

Test and support python 3.x

Reported by: atagar Owned by: atagar
Priority: Medium Milestone:
Component: Core Tor/Stem Version:
Severity: Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Stem's setup.py does a python 2to3 conversion of our codebase, but we don't yet have a method for running our tests on the built output. We should do that and investigate testing failures in the 2to3 conversion.

Child Tickets

TicketTypeStatusOwnerSummary
#7835defectclosedatagarTypeError in __init__.py due to attempt to concatenate list with tuple (typo?)

Change History (8)

comment:1 Changed 5 years ago by atagar

I should have mentioned that the goal for this from a testing perspective is to have a '--python3' argument for the tests that...

  • does a 2to3 export of our codebase via setup.py (including the tests)
  • runs the tests on the 2to3 conversion of our codebase
  • optionally cleans up the build output

comment:2 Changed 5 years ago by robinson

Damian,

Some comments:

  1. remember to allow for specifying the python3 binary location
  2. run_tests.py also needs to be run through 2to3
  3. there are several cross-version changes I believe can be had now

For 2 (above), maybe python3 testing should be a separate command? For example, a shell script which does:

mkdir python3-tests
2to3 --write --no-diffs --nobackups --write-unchanged-files --output-dir=python3-tests stem test run_tests.py
( cd python3-tests; $PYTHON3 ./run_tests.py )

For 3 (above): are you interested in some low-hanging Py2->Py3 fruit? For example:

- import Queue
+ import Queue as queue

and change Queue.Queue initializations to queue.Queue. Then only one line (the import statement) is changed by 2to3 (because the Queue module was renamed to queue in Py3).

comment:3 follow-up: Changed 5 years ago by atagar

For 2 (above), maybe python3 testing should be a separate command? For example, a shell script which does:

I like that idea but lets do it through run_tests.py. I admit it's weird to have a python 2 instance of run_tests.py invoking a python 3 version, but it's probably a little nicer from an interface perspective.

For 3 (above): are you interested in some low-hanging Py2->Py3 fruit? For example:

If you're sure that it'll be a problem for python 3 and this maintains python 2.5 compatibility then go for it (dropping 2.5 might be ok if 2to3 forces us to, but that would be something to think about a bit).

comment:4 in reply to: ↑ 3 Changed 5 years ago by robinson

Replying to atagar:

For 2 (above), maybe python3 testing should be a separate command? For example, a shell script which does:

I like that idea but lets do it through run_tests.py. I admit it's weird to have a python 2 instance of run_tests.py invoking a python 3 version, but it's probably a little nicer from an interface perspective.

So... um... you mean, run a Py2 script which converts the code base, including itself, to Py3, then execs Py3 on the newly-converted version of the running script?

For 3 (above): are you interested in some low-hanging Py2->Py3 fruit? For example:

If you're sure that it'll be a problem for python 3 and this maintains python 2.5 compatibility then go for it (dropping 2.5 might be ok if 2to3 forces us to, but that would be something to think about a bit).

I'll put together some examples and send a public branch address to read.

comment:5 follow-up: Changed 5 years ago by atagar

So... um... you mean, run a Py2 script which converts the code base, including itself, to Py3, then execs Py3 on the newly-converted version of the running script?

Somehow I get the sense that you're not quite thrilled with this idea. :P

It would largely be the same as the shell script you mentioned. The only difference is that it lives in test_runner.py, in essence...

python3_destination = os.path.join(CONFIG["integ.test_directory"], 'python3')
os.makedirs(python3_destination)
system.call("2to3 --write --no-diffs --nobackups --write-unchanged-files --output-dir=%s stem test run_tests.py" % python3_destination)
system.call("python3 %s/run_tests.py %s" % (python3_destination, " ".join(sys.argv[1:]))
shutil.rmtree(python3_destination, ignore_errors = True)
sys.exit(0) # TODO: use the exit status of the above call instead

The advantages of this are...

  • '--python3' can be a testing argument just like any other.
  • No need to have yet another little shell script floating around (fewer of those mean less clutter, which I like).

Feel free to talk me down from the ledge. ;)

comment:6 in reply to: ↑ 5 Changed 5 years ago by robinson

Replying to atagar:

So... um... you mean, run a Py2 script which converts the code base, including itself, to Py3, then execs Py3 on the newly-converted version of the running script?

Somehow I get the sense that you're not quite thrilled with this idea. :P

[snipped]

Feel free to talk me down from the ledge. ;)

Ok. That's not as horrible as I was imagining. I can support your concept for this method of testing in Py3.

comment:7 Changed 5 years ago by atagar

I've started working on this a bit. The tests in the following branch now support a '--python3' argument and can at least get through a unit test run. Still lots of testing failures/errors though...

https://gitweb.torproject.org/user/atagar/stem.git/shortlog/refs/heads/python3

comment:8 Changed 4 years ago by atagar

  • Resolution set to implemented
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.