updateFallbackDirs.py thinks it is python 3 compatible but it is not

This comment would lead one to believe the script will run with python 3 but problems remain:

# Optionally uses ipaddress (python 3 builtin) or py2-ipaddress (package)
# for netblock analysis.

After running 2to3-3.7 on commit 6BC5C06D additional manual revisions were required per the attached patch. A subtle certificate validation problem exists, not enough time to fix so disabled it. Have good CA files in both system and Python directories and openssl s_client has no problem.

OUTPUT_CANDIDATES = True

is broken, wasn't prepared to spend the time hacking it.

added 043-must actualpoints::0.6 component::core tor/fallback scripts fallback-ci parent::30971 priority::medium resolution::fixed reviewer::asn severity::normal status::closed type::defect labels

Trac:
updateFallbackDirs.py-python3.patch

touch up to apply after 2to3.7

The TLS certificate validation problem could be the result of running with torsocks. Two years+ ago updateFallbackDirs.py ran fine this way, but much code has been added and changed since. Did not try running over direct connection.

Thanks for this patch.

We don't have CI for this script yet, so we can easily make changes that break some platforms. (I might wait to merge this patch until we have CI for python 2 and 3.)

Replying to starlight:

The TLS certificate validation problem could be the result of running with torsocks. Two years+ ago updateFallbackDirs.py ran fine this way, but much code has been added and changed since. Did not try running over direct connection.

Running the script over Tor was never supported, because the script is designed to measure direct consensus download speeds. We documented this requirement in April 2016:

# This script should be run from a stable, reliable network connection,
# with no other network activity (and not over tor).
# If this is not possible, please disable:
# PERFORM_IPV4_DIRPORT_CHECKS and PERFORM_IPV6_DIRPORT_CHECKS

I am not sure about the https change, the only https URL I can see is:

ONIONOO = 'https://onionoo.torproject.org/'

This URL validates fine in my (Tor) browser.

I also see this note in the Python 2 urllib2 documentation:

Note The urllib2 module has been split across several modules in Python 3 named urllib.request and urllib.error. The 2to3 tool will automatically adapt imports when converting your sources to Python 3.

https://docs.python.org/2/library/urllib2.html

Is your patch missing the imports for python 3?

Trac:
Milestone: N/A to Tor: 0.4.0.x-final
Parent: N/A to #28793 (moved)

Trac:
Status: new to needs_revision

Replying to teor:

Thanks for this patch.

welcome

...

Running the script over Tor was never supported, because the script is designed to measure direct consensus download speeds.

Sure, but it runs nice anyway.

I am not sure about the https change, the only https URL I can see is: {{{ ONIONOO = 'https://onionoo.torproject.org/' }}}

this is where it happens; on searching found many folk adapting python 2 to 3 baffled by similar behaviors; didn't have time to dig deeper

This URL validates fine in my (Tor) browser.

and with "torsocks openssl s_client -connect xxx", works fine

I also see this note in the Python 2 urllib2 documentation: {{{ Note The urllib2 module has been split across several modules in Python 3 named urllib.request and urllib.error. The 2to3 tool will automatically adapt imports when converting your sources to Python 3. }}} https://docs.python.org/2/library/urllib2.html

Is your patch missing the imports for python 3?

ran fine after the changes with python 3.7 (output at https://lists.torproject.org/pipermail/tor-relays/2018-December/016735.html); might work ok via direct but need adjustment to function using torsocks, but out of scope as you say

I'd like to merge this patch, but I don't want to disable security checks without understanding the problem better.

Has someone written a good summary of the issue with python 3, and the security impact of this change?

Trac:
Status: needs_revision to needs_information

I would not recommend keeping TLS disabled, was just tired after hours of hacking at it and trying to get the script running in an environment where compiling Python3 was the path of least resistance. Should be fixable but is probably a couple of hours of searching and hacking. While I respect Python I've vowed to do everything I would use Python for in modern C++ going forward, don't have enough space in my brain to be good at both.

Mark some tickets as postfreeze-ok, to indicate that I think they are okay to accept in 0.4.0 post-freeze. Does not indicate that they are all necessary to do postfreeze.

Trac:
Keywords: N/A deleted, postfreeze-ok added

Hi starlight,

In #27914 (moved), we split the fallback scripts out into https://git.torproject.org/fallback-scripts.git . We're still working on syncing it with https://github.com/torproject/fallback-scripts.git (#29101 (moved)).

After #29101 (moved) closes, we can check the security implications, then apply your patch to fallback-scripts. We might also get CI working before we merge any more script changes (#28797 (moved)).

(Removing tags, because this ticket no longer has any connection to tor releases.)

Trac:
Milestone: Tor: 0.4.0.x-final to N/A
Keywords: postfreeze-ok deleted, N/A added
Version: Tor: 0.3.5.5-alpha to N/A

We're not going to have time to do this any time soon.

Trac:
Parent: #28793 (moved) to #28797 (moved)

Trac:
Keywords: N/A deleted, fallback-ci added

We need to fix this at the end of 2019, because python 2 is not supported in 2020.

Trac:
Parent: #28797 (moved) to #30971 (moved)
Keywords: N/A deleted, 043-must added

I've done some research, here's what I've discovered:

• python 2's code freeze starts on 1 January 2020, the final release will be around April at pycon 2020
• if we want to continue testing using python 2 during the transition, we should use python-modernize or python-future, not 2to3
• python 3.6 is the earliest supported python 3; python 3.2, 3.3, and 3.5 added features that make porting easier
• we should try to use if PY2 for any code that won't run in both versions, in case there is a python 4 some day

See my PR:

• fallback master: https://github.com/torproject/fallback-scripts/pull/9

I tried to make minimal changes to get the 3 fallback scripts working with python 3.6 and later. I also made sure I maintained compatibility with python 2. We should see if any of the CI fails, it has a lot more python versions.

We might fix check_existing and OUTPUT_CANDIDATES modes in #31021 (moved), but they aren't a high priority, because we don't use them very often.

Trac:
Actualpoints: N/A to 0.5
Status: needs_information to needs_review

Trac:
Reviewer: N/A to asn

(CI is passing, I rewrote the lookup* and generate* scripts to make them more reliable.)

I need to add future statements at the start of the scripts:

from __future__ import print_function

And for the update script:

from __future__ import division

Trac:
Status: needs_review to needs_revision

I made this change by adding all the __future__ imports to every python script.

Trac:
Actualpoints: 0.5 to 0.6
Status: needs_revision to needs_review

No-one has reviewed these changes, so I'll just merge them after CI passes.

Trac:
Status: needs_review to merge_ready

Merged to master.

Trac:
Resolution: N/A to fixed
Status: merge_ready to closed

closed

added 4h 48m of time spent

mentioned in issue #31020 (moved)

mentioned in issue #31021 (moved)

mentioned in issue #32632 (moved)

mentioned in issue #32959 (moved)

mentioned in issue #30971 (moved)

moved to tpo/core/fallback-scripts#28863 (closed)