Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#8549 closed defect (fixed)

Mac OS X pluggable transports bundles need argparse for Python 2.6 and earlier

Reported by: dcf Owned by: dcf
Priority: Medium Milestone:
Component: Archived/Flashproxy Version:
Severity: Keywords:
Cc: mrphs, aallai, asn Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

pyobfsproxy doesn't work in the 2.4.11-alpha-1 bundles because argparse.py is missing. Looking at the bundle-macosx.txt instructions, it appears we are also missing the commands to copy at least twisted and zope too.

We need to add these copy commands to the bundle instructions, build new bundles, test them on a non-development machine, and publish the new bundles.

Child Tickets

Attachments (3)

download.php?i=3bDzyBR9 (6.3 KB) - added by rransom 4 years ago.
pastebin is not an archival system
0001-Add-argparse-to-bundle-instructions.patch (1.3 KB) - added by aallai 4 years ago.
Update instructions to copy argparse.
0001-Add-argparse-to-bundle-instructions.2.patch (1.6 KB) - added by aallai 4 years ago.

Download all attachments as: .zip

Change History (29)

comment:1 Changed 5 years ago by dcf

Priority: criticalnormal
Summary: Mac OS X pluggable transports bundles are missing argparse.py, twisted, zopeMac OS X pluggable transports bundles need Python 2.7 for argparse

Talking with Alex, it seems that argparse, twisted, and zope were omitted intentionally; argparse because it is part of the standard library (new in Python 2.7) and twisted and zope because they come with Mac OS X.

Therefore it seems this reduces to making sure our programs get run with python2.7. Probably the original error happened on a system where python2.6 is the default.

comment:2 Changed 4 years ago by mrphs

Yes, the default version of Python on OS X 10.6.8 (Snow Leopard) is 2.6.1

Now the question is, do we want to support 10.6.8 (which is not very old) or not?

--
Someone please update the blog post and mention that the latest "pluggable transport" only supports the latest Mac OS X.

comment:3 in reply to:  2 ; Changed 4 years ago by dcf

Replying to mrphs:

Yes, the default version of Python on OS X 10.6.8 (Snow Leopard) is 2.6.1

Now the question is, do we want to support 10.6.8 (which is not very old) or not?

I think we definitely want to support 10.6. I wonder if it is also missing Twisted. mrphs, does this command work for you?

python -c 'import twisted'

There should be no ImportError.

Someone please update the blog post and mention that the latest "pluggable transport" only supports the latest Mac OS X.

I added a note to the blog post.

comment:4 in reply to:  3 Changed 4 years ago by mrphs

Twisted was imported without any problem

comment:5 Changed 4 years ago by aallai

I made an updated bundle with argparse, mrphs can you please test it?

TorBrowser-Pluggable..>

TorBrowser-Pluggable..>

Thanks

comment:6 in reply to:  5 ; Changed 4 years ago by mrphs

Replying to aallai:

I made an updated bundle with argparse, mrphs can you please test it?

TorBrowser-Pluggable..>

Here's the Log file of this new bundle:
http://pastebin.com/UL8Ci3Bx

comment:7 in reply to:  6 ; Changed 4 years ago by dcf

Replying to mrphs:

Here's the Log file of this new bundle:
http://pastebin.com/UL8Ci3Bx

I see. Rats. I hoped that argparse.py copied from Python 2.7 would be compatible with Python 2.6, but it looks like not.

AttributeError: 'module' object has no attribute 'OrderedDict'

I think this leaves us with two options:

  1. Ship a Python interpreter like we do for Windows, using py2app or similar.
  2. Reduce pyobfsproxy's requirements to Python 2.6.

My vote is for (2), if argument parsing is the only thing that currently requires Python 2.7. Apart from this issue, the strategy of using the system Python has been working well. George, what do you think? Is it possible to rewrite option parsing to be compatible with Python 2.6?

comment:8 in reply to:  7 ; Changed 4 years ago by asn

Replying to dcf:

Replying to mrphs:

Here's the Log file of this new bundle:
http://pastebin.com/UL8Ci3Bx

I see. Rats. I hoped that argparse.py copied from Python 2.7 would be compatible with Python 2.6, but it looks like not.

AttributeError: 'module' object has no attribute 'OrderedDict'

I think this leaves us with two options:

  1. Ship a Python interpreter like we do for Windows, using py2app or similar.
  2. Reduce pyobfsproxy's requirements to Python 2.6.

My vote is for (2), if argument parsing is the only thing that currently requires Python 2.7. Apart from this issue, the strategy of using the system Python has been working well. George, what do you think? Is it possible to rewrite option parsing to be compatible with Python 2.6?

Hm, implementation of (2) is not trivial. Refactoring Pyobfsproxy to support Python 2.6 means that we will have to rewrite the cli parsing logic (which is kinda hard because it uses argparse subparsers), replace OrderedDicts in the integration tests, and change any other 2.7 constructs are lying around. Also, pyptlib is not meant to be ran in 2.6 either.

What happens if we ship a Python interpreter along with the bundles? How much will the bundle size increase?

comment:9 in reply to:  8 Changed 4 years ago by dcf

Replying to asn:

What happens if we ship a Python interpreter along with the bundles? How much will the bundle size increase?

Probably only a few MB, but it just means more pain trying to get py2app to work and checking the resulting executables against virus scanners. Basically another dose of the things we go through for the Windows bundle.

comment:10 in reply to:  7 ; Changed 4 years ago by aallai

Replying to dcf:

Replying to mrphs:

Here's the Log file of this new bundle:
http://pastebin.com/UL8Ci3Bx

I see. Rats. I hoped that argparse.py copied from Python 2.7 would be compatible with Python 2.6, but it looks like not.

AttributeError: 'module' object has no attribute 'OrderedDict'

I think this leaves us with two options:

  1. Ship a Python interpreter like we do for Windows, using py2app or similar.
  2. Reduce pyobfsproxy's requirements to Python 2.6.

My vote is for (2), if argument parsing is the only thing that currently requires Python 2.7. Apart from this issue, the strategy of using the system Python has been working well. George, what do you think? Is it possible to rewrite option parsing to be compatible with Python 2.6?

I could work on (2), it's probably more fun than py2app.

comment:11 Changed 4 years ago by hellais

You could also just add as a requirement argparse (https://pypi.python.org/pypi/argparse).

A perhaps better solution would be of using twisted.python.usage instead of argparse.

comment:12 in reply to:  11 Changed 4 years ago by dcf

Replying to hellais:

You could also just add as a requirement argparse (https://pypi.python.org/pypi/argparse).

Ah, hmm, interesting. If I use the argparse.py from http://argparse.googlecode.com/files/argparse-1.2.1.tar.gz, I can run pyobfsproxy with python2.6. This is not the case if I copy argparse.py from a 2.7 library tree; they are not the same file.

So Alex, would you try building another bundle using the argparse.py from the tarball above? If there are other Python 2.7 dependencies, perhaps we can remove them one by one. You might want to start with a GNU/Linux bundle, and change the shebang line to #!/usr/bin/python2.6, and make sure you can bootstrap, before remaking the Mac bundle.

comment:13 in reply to:  10 Changed 4 years ago by asn

Replying to aallai:

Replying to dcf:

Replying to mrphs:

Here's the Log file of this new bundle:
http://pastebin.com/UL8Ci3Bx

I see. Rats. I hoped that argparse.py copied from Python 2.7 would be compatible with Python 2.6, but it looks like not.

AttributeError: 'module' object has no attribute 'OrderedDict'

I think this leaves us with two options:

  1. Ship a Python interpreter like we do for Windows, using py2app or similar.
  2. Reduce pyobfsproxy's requirements to Python 2.6.

My vote is for (2), if argument parsing is the only thing that currently requires Python 2.7. Apart from this issue, the strategy of using the system Python has been working well. George, what do you think? Is it possible to rewrite option parsing to be compatible with Python 2.6?

I could work on (2), it's probably more fun than py2app.

As a developer, I would prefer not to change any code of Obfsproxy, especially if we can avoid it by using pypi's 2.6 argparse or adding a 2.7 interpreter inside the bundles. Changing the CLI code of Obfsproxy is unneeded implementation pain and might even introduce regressions bugs.

In any case, if you dig this task and you can come up with a reasonable patch that makes pyofsproxy 2.6-compliant, I would be up for merging it. Still, I consider it kind of a waste of effort and time (but maybe more exciting than playing with py2app).

Changed 4 years ago by rransom

Attachment: download.php?i=3bDzyBR9 added

pastebin is not an archival system

comment:16 Changed 4 years ago by hellais

It works fine for me.

Tested on Mac OS X 10.8.2.

Mar 28 13:26:54.791 [Notice] Tor v0.2.4.11-alpha (git-fcd9248387249d68) running on Darwin with Libevent 2.0.21-stable and OpenSSL 1.0.0k.
Mar 28 13:26:54.791 [Notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Mar 28 13:26:54.792 [Notice] This version is not a stable Tor release. Expect more bugs than usual.
Mar 28 13:26:54.792 [Notice] Read configuration file "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Library/Vidalia/torrc".
Mar 28 13:26:54.792 [Notice] Opening Socks listener on 127.0.0.1:9150
Mar 28 13:26:54.793 [Notice] Opening Control listener on 127.0.0.1:9151
Mar 28 13:26:54.793 [Notice] Parsing GEOIP IPv4 file ../../Contents/Resources/Data/Tor/geoip.
Mar 28 13:26:58.093 [Notice] Bootstrapped 5%: Connecting to directory server.
Mar 28 13:26:58.094 [Notice] Bootstrapped 10%: Finishing handshake with directory server.
Mar 28 13:27:01.111 [Notice] Learned fingerprint 94C9E691688FAFDEC701A0788BD15BE8AD34ED35 for bridge 70.182.182.109:54542 (with transport 'obfs2').
Mar 28 13:27:01.112 [Notice] Bootstrapped 15%: Establishing an encrypted directory connection.
Mar 28 13:27:01.118 [Notice] Learned fingerprint 58D91C3A631F910F32E18A55441D5A0463BA66E2 for bridge 209.141.36.236:60783 (with transport 'obfs2').
Mar 28 13:27:01.210 [Notice] Bootstrapped 20%: Asking for networkstatus consensus.
Mar 28 13:27:01.303 [Notice] Bootstrapped 50%: Loading relay descriptors.
Mar 28 13:27:01.406 [Notice] new bridge descriptor 'foolsgarden' (fresh): $58D91C3A631F910F32E18A55441D5A0463BA66E2~foolsgarden at 209.141.36.236
Mar 28 13:27:01.407 [Notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
Mar 28 13:27:01.462 [Notice] new bridge descriptor 'mana1' (fresh): $94C9E691688FAFDEC701A0788BD15BE8AD34ED35~mana1 at 70.182.182.109
Mar 28 13:27:01.462 [Notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
Mar 28 13:27:01.473 [Notice] Learned fingerprint BA61757846841D64A83EA2514C766CB92F1FB41F for bridge 208.79.90.242:55564 (with transport 'obfs2').
Mar 28 13:27:01.487 [Notice] Learned fingerprint CA7434F14A898C7D3427B8295A7F83446BC7F496 for bridge 128.31.0.34:1051 (with transport 'obfs2').
Mar 28 13:27:01.726 [Notice] Learned fingerprint 844B1F53FFD548C998F8D3B01B7E19FA07C3396E for bridge 109.105.109.163:46924 (with transport 'obfs2').
Mar 28 13:27:01.931 [Notice] Learned fingerprint 6F058CBEF888EB20D1DEB9886909F1E812245D41 for bridge 83.212.96.182:46602 (with transport 'obfs2').
Mar 28 13:27:01.932 [Notice] Learned fingerprint 1F01A7BB60F49FC96E0850A6BAD6D076DFEFAF80 for bridge 83.212.100.216:47870 (with transport 'obfs2').
Mar 28 13:27:02.230 [Notice] new bridge descriptor 'bridge' (fresh): $CA7434F14A898C7D3427B8295A7F83446BC7F496~bridge at 128.31.0.34
Mar 28 13:27:02.230 [Notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
Mar 28 13:27:02.369 [Notice] Learned fingerprint 2ADFE7AA8D272C520D1FBFBF4E413F3A1B26313D for bridge 83.212.101.2:42782 (with transport 'obfs3').
Mar 28 13:27:02.428 [Notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
Mar 28 13:27:02.480 [Notice] Learned fingerprint 00BC5E7111BD00E9AF463BE9BFE6255FE51CFCD9 for bridge 109.163.233.198:1051 (with transport 'obfs2').
Mar 28 13:27:02.623 [Notice] new bridge descriptor 'ndnop2' (fresh): $844B1F53FFD548C998F8D3B01B7E19FA07C3396E~ndnop2 at 109.105.109.163
Mar 28 13:27:02.623 [Notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
Mar 28 13:27:03.105 [Notice] Bridge 'arta' has both an IPv4 and an IPv6 address.  Will prefer using its IPv4 address (83.212.96.182:46602).
Mar 28 13:27:03.106 [Notice] new bridge descriptor 'arta' (fresh): $6F058CBEF888EB20D1DEB9886909F1E812245D41~arta at 83.212.96.182
Mar 28 13:27:03.106 [Notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
Mar 28 13:27:03.122 [Notice] new bridge descriptor 'PacificSunset' (fresh): $BA61757846841D64A83EA2514C766CB92F1FB41F~PacificSunset at 208.79.90.242
Mar 28 13:27:03.122 [Notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
Mar 28 13:27:03.205 [Notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/3313, and can only build 0% of likely paths. (We have 0% of guards bw, 0% of midpoint bw, and 0% of exit bw.)
Mar 28 13:27:03.246 [Notice] new bridge descriptor 'bzoum' (fresh): $2ADFE7AA8D272C520D1FBFBF4E413F3A1B26313D~bzoum at 83.212.101.2
Mar 28 13:27:03.246 [Notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/3313, and can only build 0% of likely paths. (We have 0% of guards bw, 0% of midpoint bw, and 0% of exit bw.)
Mar 28 13:27:03.363 [Notice] Bridge 'giofyri' has both an IPv4 and an IPv6 address.  Will prefer using its IPv4 address (83.212.100.216:47870).
Mar 28 13:27:03.364 [Notice] new bridge descriptor 'giofyri' (fresh): $1F01A7BB60F49FC96E0850A6BAD6D076DFEFAF80~giofyri at 83.212.100.216
Mar 28 13:27:03.364 [Notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/3313, and can only build 0% of likely paths. (We have 0% of guards bw, 0% of midpoint bw, and 0% of exit bw.)
Mar 28 13:27:05.745 [Notice] new bridge descriptor 'Unnamed' (fresh): $00BC5E7111BD00E9AF463BE9BFE6255FE51CFCD9~Unnamed at 109.163.233.198
Mar 28 13:27:05.746 [Notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/3313, and can only build 0% of likely paths. (We have 0% of guards bw, 0% of midpoint bw, and 0% of exit bw.)
Mar 28 13:27:09.099 [Notice] We now have enough directory information to build circuits.
Mar 28 13:27:09.117 [Notice] Bootstrapped 80%: Connecting to the Tor network.
Mar 28 13:27:09.118 [Notice] Bootstrapped 90%: Establishing a Tor circuit.
Mar 28 13:27:12.022 [Notice] Tor has successfully opened a circuit. Looks like client functionality is working.
Mar 28 13:27:12.023 [Notice] Bootstrapped 100%: Done.

comment:17 in reply to:  14 Changed 4 years ago by dcf

Replying to aallai:

I made a bundle as suggested above by dcf, mrphs can you please test it?

http://cs.mcgill.ca/~aallai2/bundles/2.4.11/TorBrowser-Pluggable-Transports-2.4.11-alpha-2-osx-i386-en-US.zip

http://cs.mcgill.ca/~aallai2/bundles/2.4.11/TorBrowser-Pluggable-Transports-2.4.11-alpha-2-osx-i386-en-US.zip.asc

I diffed this bundle against the 2.4.11-alpha-1, and found some extraneous .pyc files:

$ diff -ru 2.4.11-alpha-1/TorBrowser_en-US.app/ 2.4.11-alpha-2/TorBrowser_en-US.app/ 
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS: argparse.py
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/common: aes.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/common: heartbeat.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/common: hmac_sha256.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/common: __init__.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/common: log.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/common: rand.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/common: serialize.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy: __init__.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/managed: client.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/managed: __init__.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/managed: server.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/network: buffer.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/network: extended_orport.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/network: __init__.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/network: launch_transport.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/network: network.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/network: socks.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy: pyobfsproxy.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/transports: b64.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/transports: base.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/transports: dummy.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/transports: __init__.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/transports: obfs2.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/transports: obfs3_dh.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/transports: obfs3.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy/transports: transports.pyc
Only in 2.4.11-alpha-2/TorBrowser_en-US.app/Contents/MacOS/obfsproxy: _version.pyc

If I remember correctly, the .pyc files are platform-dependent and need to be deleted. Other than that, the only difference I saw was the addition of argparse.py. So since the bundles work, please make another without the .pyc files and then you are good to release. Like we discussed, for the other platforms, just copy the 2.4.11-alpha-1 packages and call them 2.4.11-alpha-2.

Changed 4 years ago by aallai

Update instructions to copy argparse.

comment:18 Changed 4 years ago by aallai

I have remade/renamed the bundles, pyc-free, they are at

http://cs.mcgill.ca/~aallai2/bundles/2.4.11/

George could you make a new blog post?

I also updated the patch to the instructions.

comment:19 Changed 4 years ago by asn

Great!

BTW, you think we should write a new blog post for these new bundles, or just do an update on the current blog post?

comment:20 in reply to:  19 Changed 4 years ago by dcf

Replying to asn:

Great!

BTW, you think we should write a new blog post for these new bundles, or just do an update on the current blog post?

Update to the current post along with updates to the download page are fine with me.

comment:21 in reply to:  18 Changed 4 years ago by dcf

Replying to aallai:

I also updated the patch to the instructions.

Could you add the sha256sum you see to the patch? I think it's also worth adding a link to this ticket to explain why we bother to include argparse as an additional dependency.

comment:22 Changed 4 years ago by aallai

I attached an updated patch. I forgot to check the overwrite checkbox so its been uploaded
as 0001-Add-argparse-to-bundle-instructions.2.patch.

comment:23 Changed 4 years ago by asn

Patch looks good. I'll try to find someone to upload the bundles in http://cs.mcgill.ca/~aallai2/bundles/2.4.11/ to www.tpo/dist .

comment:24 Changed 4 years ago by dcf

Resolution: fixed
Status: newclosed

I'm closing this ticket because you can download the bundles now. See #8644 and #8645 for issues uncovered by this release.

comment:25 Changed 4 years ago by dcf

Summary: Mac OS X pluggable transports bundles need Python 2.7 for argparseMac OS X pluggable transports bundles need argparse for Python 2.6 and earlier

comment:26 Changed 4 years ago by dcf

I updated https://blog.torproject.org/blog/new-pluggable-transports-bundles-02411-alpha-flashproxy-obfsproxy with the 2.4.11-alpha-2 bundles (which are really built on the 2.4.11-alpha-1 TBB; see #8644 which addresses this confusion in the future).

Note: See TracTickets for help on using tickets.