Opened 6 years ago

Closed 6 years ago

#8485 closed defect (duplicate)

Gettor: TBB too big now for Gmail

Reported by: mo Owned by: mo
Priority: Very High Milestone:
Component: Applications/GetTor Version:
Severity: Keywords:
Cc: sukhbir Actual Points:
Parent ID: #8542 Points:
Reviewer: Sponsor:

Description

TBB for Windows is 27MB now, TBB for Linux 40MB. Gmail accepts only 25MB attachments from external sources.

It does however support 10GB (!) attachments from other Gmail users, so one idea would be to use Gmail to send Gettor mails?

Child Tickets

Attachments (1)

gdrive.patch (7.7 KB) - added by sukhbir 6 years ago.

Download all attachments as: .zip

Change History (41)

comment:1 Changed 6 years ago by mo

Cc: gsathya added
Owner: set to mo
Status: newassigned

The quick fix really is to move SMTP from locally to a gmail account. We should do that now. I asked kaner for ssh axx so I can do it.

comment:2 Changed 6 years ago by mo

Priority: majorcritical

comment:3 Changed 6 years ago by kaner

What has changed to blow up TBB? I guess Firefox?

Another thing we could do is to implement a different packaging method for TBB alltogether (see #3920)

As I am not handing out GetTor host access anymore, you should ask weasel to do this.

comment:4 in reply to:  3 Changed 6 years ago by gsathya

Cc: sukhbir added; gsathya removed

Replying to kaner:

What has changed to blow up TBB? I guess Firefox?

Shipping Python as part of TBB(for the pyobfsproxy-flashproxy bundles, I think)

comment:5 Changed 6 years ago by runa

Using a Gmail account, I can successfully receive the following packages:

  • obfs-windows
  • macos-ppc
  • windows
  • source

I am unable to receive the following packages:

  • macos-i386
  • linux-i386
  • linux-x86_64
  • obfs-macos-i386
  • obfs-macos-x86_64
  • obfs-linux-i386
  • obfs-linux-x86_64

comment:6 Changed 6 years ago by kaner

I guess the reason you still can receive packages with GMail is that the auto-update mechanism of GetTor broke again due to changes in the package naming schemes, so that you're receiving old packages.

comment:7 in reply to:  6 ; Changed 6 years ago by kaner

Replying to kaner:

I guess the reason you still can receive packages with GMail is that the auto-update mechanism of GetTor broke again due to changes in the package naming schemes, so that you're receiving old packages.

Checking on the GetTor host confirms what I thought. GetTor is unable to find any current packages of the naming scheme tor-obfsproxy-browser-* or osx/TorBrowser-Obfsproxy-* or linux/tor-obfsproxy-browser-gnu-linux-i686-*.

I guess that changed?

All I can find in the dist/ directory is a tor-flashproxy-pyobfsproxy-browser package, but only for en-US.

comment:8 in reply to:  7 Changed 6 years ago by runa

Replying to kaner:

Replying to kaner:

I guess the reason you still can receive packages with GMail is that the auto-update mechanism of GetTor broke again due to changes in the package naming schemes, so that you're receiving old packages.

Checking on the GetTor host confirms what I thought. GetTor is unable to find any current packages of the naming scheme tor-obfsproxy-browser-* or osx/TorBrowser-Obfsproxy-* or linux/tor-obfsproxy-browser-gnu-linux-i686-*.

I guess that changed?

All I can find in the dist/ directory is a tor-flashproxy-pyobfsproxy-browser package, but only for en-US.

You're looking for "tor-pluggable-transports-browser". The bundle is only available in English, but it's the bundle that users in China, Iran, Syria, etc need.

comment:9 Changed 6 years ago by runa

See #8536 for a patch that fixes the filename issue.

comment:10 Changed 6 years ago by kaner

Thanks. Somehow my mail filters didn't fish the #8536 patch to my Inbox!

comment:11 Changed 6 years ago by runa

Revised list of packages you can receive with a Gmail account:

  • macos-ppc
  • windows
  • source

List of packages you cannot receive :

  • obfs-windows
  • macos-i386
  • linux-i386
  • linux-x86_64
  • obfs-macos-i386
  • obfs-macos-x86_64
  • obfs-linux-i386
  • obfs-linux-x86_64

comment:12 Changed 6 years ago by mo

The new Gettor will send individual Google Drive links (direct download links). Until then, I have now changed Gettor to send mails via Gmail, so at least Gmail users can receive our packages again. Sadly, it does not really work, and I have a hard time figuring out why.

  • I can send emails including large attachments via gmail SMTP fine (using gettor's From addresses).
  • Gettor can send all mails without attachments via gmail SMTP fine (using gettor's From addresses).
  • it cannot send mails with attachment (smtp auth failure)

So, sending packages is still broken. Now for a different reason than before.

/home/gettor/.gettor.conf

PROVIDER_ATTACHMENT_SIZES = { "gmail.com": 500,
                              "yahoo.com": 25,
                              "yahoo.cn": 25 }


/srv/gettor.torproject.org/gettor/lib/python2.6/site-packages/gettor/responses.py

def sendEmail(self, sendTo, message, smtpserver="localhost:25"):
        """Send out message via STMP. If an error happens, be verbose about·
           the reason
        """
        # modified by moritz to send via gmail (#8485)
        smtpserver = "smtp.gmail.com"
        smtpuser = "gettormail@gmail.com"
        smtppass = <>
        try:
            smtp = smtplib.SMTP(smtpserver,587)
            smtp.ehlo()
            smtp.starttls()
            smtp.ehlo()
            smtp.login(smtpuser, smtppass)
            smtp.sendmail(self.reqInfo['ouraddr'], sendTo, message.as_string())
            smtp.quit()

comment:13 in reply to:  12 Changed 6 years ago by kaner

Replying to mo:

The new Gettor will send individual Google Drive links (direct download links). Until then, I have now changed Gettor to send mails via Gmail, so at least Gmail users can receive our packages again. Sadly, it does not really work, and I have a hard time figuring out why.

  • I can send emails including large attachments via gmail SMTP fine (using gettor's From addresses).
  • Gettor can send all mails without attachments via gmail SMTP fine (using gettor's From addresses).
  • it cannot send mails with attachment (smtp auth failure)

So, sending packages is still broken. Now for a different reason than before.

Any progress on this?

comment:14 Changed 6 years ago by kaner

It seems GetTor is getting a "SMTP sender address refused" from GMail. This is to be expected when sending mails as "gettor@…" as a sender address. What was your exact plan concerning that? If we set "something@…" as a sender, people will reply to that address, too, and nothing ends up with GetTor.

Maybe we should either
a) shrink the package size again or
b) Use the transport mechanism pointed out using Google Drive?

All we need for b) is to replace some code in responses.py with the upload to Google Drive.

comment:15 Changed 6 years ago by kaner

Maybe it helps if we set reply-to to gettor@… for now?

But still, I think the Google Drive upload would be fairly easy to implement, too.

comment:16 in reply to:  15 Changed 6 years ago by kaner

Replying to kaner:

Maybe it helps if we set reply-to to gettor@… for now?

But still, I think the Google Drive upload would be fairly easy to implement, too

Still no luck :(

send: 'mail FROM:<gettormail@gmail.com> size=38447535\r\n'
reply: "552-5.2.3 Your message exceeded Google's message size limits. Please visit\r\n"
reply: '552-5.2.3 http://support.google.com/mail/bin/answer.py?answer=8770 to review\r\n'
reply: '552 5.2.3 our size guidelines. qn10sm10413700igc.6 - gsmtp\r\n'

I'm changing it back to original now - before your patch. With my tests so far, I didn't even get a error response or help text from GetTor, even though it said it sended those out in the logs.

comment:17 Changed 6 years ago by kaner

Changed back to original. Now users get "sorry attachment size to small" error mails as a reply, at least. We should work on this Drive upload asap!

comment:18 in reply to:  14 Changed 6 years ago by mo

Replying to kaner:

It seems GetTor is getting a "SMTP sender address refused" from GMail. This is to be expected when sending mails as "gettor@…" as a sender address. What was your exact plan concerning that? If we set "something@…" as a sender, people will reply to that address, too, and nothing ends up with GetTor.

No. You can set arbitrary From addresses with Gmail, even in the web interface - after you have confirmed them once. This is *not* the issue here, as I have described in my previous comment: I can send mails fine using gettor@tpo addresses as From address from command line (including attachments), from within the Gmail web interface (including attachments), and Gettor was also able to send mails via Gmail. Just not the ones with the attachment.

https://support.google.com/accounts/bin/answer.py?hl=en&answer=86635&topic=2382751&ctx=topic etc

a) shrink the package size again or
b) Use the transport mechanism pointed out using Google Drive?

All we need for b) is to replace some code in responses.py with the upload to Google Drive.

Either gsathya and/or sukhbir and/or nima wanted to do the Google Drive thing. You or me can do it as well, but I don't want to steal their chosen assignments.

comment:19 Changed 6 years ago by mo

Like I said, the first thing I would have tried is enable debug info in the smtp parts, and look at the actual response Google sends, and when. It does *not* reject Gettor From addresses in general. I have added and confirmed all of them to the Gmail account.

comment:20 in reply to:  19 Changed 6 years ago by kaner

Replying to mo:

Like I said, the first thing I would have tried is enable debug info in the smtp parts, and look at the actual response Google sends, and when. It does *not* reject Gettor From addresses in general. I have added and confirmed all of them to the Gmail account.

Thats precisely what I did. See attached log message from my previous reply. It seems we're simply too large for GMail, even when we send from a GMail account.

I don't know what went wrong with the missing mails without attachment from GetTor. I tested for about an hour an not a single mail came through. When I switched back to the getulum SMTP, mails were sent fine.

If gsathya and/or sukhbir and/or nima want to implement the Google Drive thing, great. Its just that we need a solution real soon and the fastest way to get it flying is to patch current GetTor. As far as I see it - unless I'm missing something, its an easy straightforward patch.

comment:21 Changed 6 years ago by sukhbir

So for everyone who is subscribed to this ticket: our idea (as of now) is to use Google Drive; we will upload the packages there (Google Drive API) and then distribute the URLs in the email. The maximum file we can upload is 10 GB, so it's perfect for us.

comment:22 in reply to:  21 ; Changed 6 years ago by Jason

Replying to sukhbir:

So for everyone who is subscribed to this ticket: our idea (as of now) is to use Google Drive; we will upload the packages there (Google Drive API) and then distribute the URLs in the email. The maximum file we can upload is 10 GB, so it's perfect for us.

Google Drive has been blocked by GFW since the beginning. If gettor has to use the URLs, I recommend other online file sharing tools, such as drobpox, box or evernote, which are not blocked in mainland China.

comment:23 in reply to:  22 ; Changed 6 years ago by runa

Replying to Jason:

Replying to sukhbir:

So for everyone who is subscribed to this ticket: our idea (as of now) is to use Google Drive; we will upload the packages there (Google Drive API) and then distribute the URLs in the email. The maximum file we can upload is 10 GB, so it's perfect for us.

Google Drive has been blocked by GFW since the beginning. If gettor has to use the URLs, I recommend other online file sharing tools, such as drobpox, box or evernote, which are not blocked in mainland China.

Ideally, we should make GetTor reply with a few different URLs: Google Drive, mirrors, other file sharing tools, etc.

comment:24 Changed 6 years ago by mrphs

The idea was to give user options to choose how to receive the package. #8542
so ppl from China should be able to get it from mirrors as Runa mentioned.

The amazing thing about Google Drive is that we can have a unique URL for every single request. The idea is to fetch the latest bundle from Tor dist and upload it to Google Drive *for every single request*.

But of course, it has it's own problems. For instance, every package we want to send is ~40MB and we'll run out of space as soon as we receive 250 requests.

The solution is to have like 10 accounts and randomly switch between them. There we can handle 2500 requests at a short time. and we can remove the uploaded files after 2hrs or so. to make sure there's always enough free space.

and the whole idea of this Google Drive thing came from SiNA's code: https://github.com/mrphs/Tor-XMPP-Bot

and if someone's got time to steal my idea, please do! All I care is to have GetTor back to ppl who need it.

comment:25 in reply to:  24 ; Changed 6 years ago by runa

Replying to mrphs:

The amazing thing about Google Drive is that we can have a unique URL for every single request. The idea is to fetch the latest bundle from Tor dist and upload it to Google Drive *for every single request*.

What happens if someone decides to flood us with 500 requests? Will Google Drive be happy if we suddenly switch between accounts and upload 500 copies of the bundle? What about the host uploading the files?

comment:26 in reply to:  25 Changed 6 years ago by mrphs

Replying to runa:

What happens if someone decides to flood us with 500 requests? Will Google Drive be happy if we suddenly switch between accounts and upload 500 copies of the bundle? What about the host uploading the files?

The attack should hit all the 10 accounts equally, 50 request for each (~2000MB).
Not sure how Google would feel about that. Theoretically, there shouldn't be a problem. But let's think of the opposite.

Here are some random thoughts:

  • We should check the header for patterns... IP space, protocol, whether it's from Gmail or not. If matched the pattern, we'll reply them with a message like this:

"We're receiving too many requests from your IP. Are you a human? please reply with the answer of this question:" 2+2=?

  • Or we can reply with mirror links only. with a note of course to clarify why they didn't get the unique url. Or a mix of this and previous idea.
  • Upgrading storage to 100GB ($10/m) or 1TB ($50/m) and then one account should be able to handle the whole traffic. But of course, there's no fun at it.
  • Using NginX to handle the attacks? not quite sure about this one tho.

comment:27 in reply to:  23 Changed 6 years ago by Jason

Replying to runa:

Replying to Jason:

Replying to sukhbir:

So for everyone who is subscribed to this ticket: our idea (as of now) is to use Google Drive; we will upload the packages there (Google Drive API) and then distribute the URLs in the email. The maximum file we can upload is 10 GB, so it's perfect for us.

Google Drive has been blocked by GFW since the beginning. If gettor has to use the URLs, I recommend other online file sharing tools, such as drobpox, box or evernote, which are not blocked in mainland China.

Ideally, we should make GetTor reply with a few different URLs: Google Drive, mirrors, other file sharing tools, etc.

Runa thinks I should put my suggetion here, so here it is:

BitTorrent Sync is one of p2p tools used by Chinese users to distribute censored videos/books. The download speed is much faster than magnet link, and it's easy to use. Download the software (the Windows version is only 630kb) and input the secret key (a character string generated by BTsync when sharing files), now you can download the files. 

Please note BT Sync is still a alpha software, there will be security issues.

comment:28 Changed 6 years ago by mo

From what I see, Bittorrent Sync simply uses magnet URIs. It thus cannot be "faster", although it might appear to be since for local sync you find peers using local peer discovery and not the DHT.

Magnet URIs for /dist are now available and should be sent out/used by Gettor. See #6978.

comment:29 in reply to:  28 Changed 6 years ago by runa

Replying to mo:

Magnet URIs for /dist are now available and should be sent out/used by Gettor. See #6978.

Too bad GetTor stopped working a few weeks ago (#8888).

comment:30 Changed 6 years ago by kaner

It would be good if someone could drop me an email if I don't react to "GetTor broken" tickets quickly, should this happen again in the future. I don't follow Trac too closely these days. TIA!

Changed 6 years ago by sukhbir

Attachment: gdrive.patch added

comment:31 Changed 6 years ago by sukhbir

kaner, can we please apply the patch that I emailed you so that we can the Google Drive thing going? (Yes, we are doing it!)

comment:32 in reply to:  24 Changed 6 years ago by arma

Replying to mrphs:

The amazing thing about Google Drive is that we can have a unique URL for every single request. The idea is to fetch the latest bundle from Tor dist and upload it to Google Drive *for every single request*.

How is this more helpful than just giving everybody the same url for a given bundle?

(Using the same url means they can share it with their friends. It also means there isn't a scary long URL that looks exactly like we're uniquely tracking them.)

comment:33 Changed 6 years ago by arma

Answer (as far as I can tell): making a new url for each request was intended to prevent the adversary from learning the URL and blocking it. But in most of these cases, the url will be https, so the adversary can't see enough of the URL (when an honest user fetches it) to block it.

So I think we should just stick with one URL for each bundle, and maybe rotate it every so often if that's helpful in some way (like if it's some other service that does allow http).

comment:34 in reply to:  33 ; Changed 6 years ago by kaner

Replying to arma:

Answer (as far as I can tell): making a new url for each request was intended to prevent the adversary from learning the URL and blocking it. But in most of these cases, the url will be https, so the adversary can't see enough of the URL (when an honest user fetches it) to block it.

So I think we should just stick with one URL for each bundle, and maybe rotate it every so often if that's helpful in some way (like if it's some other service that does allow http).

I believe someone recently said Google Drive is fully blocked in countries we're aiming for (China & Iran, mostly) since a long time anyway.

comment:35 in reply to:  34 ; Changed 6 years ago by kaner

Replying to kaner:

I believe someone recently said Google Drive is fully blocked in countries we're aiming for (China & Iran, mostly) since a long time anyway.

This probably also means that we should switch from the Google Drive mechanism (if go for that) back to sending out emails as soon as the TBB shrinks down past 25MB again. Mike said something that he's working on it.

comment:36 Changed 6 years ago by mrphs

Status: assignedneeds_review

Changing to needs_review. This ticket has a patch sitting here for 2months.

comment:37 in reply to:  35 Changed 6 years ago by mrphs

Replying to kaner:

Replying to kaner:

I believe someone recently said Google Drive is fully blocked in countries we're aiming for (China & Iran, mostly) since a long time anyway.

Not Iran. But I agree that we need more option than just GDrive. That's why we've been working on DropBox and other ways.

This probably also means that we should switch from the Google Drive mechanism (if go for that) back to sending out emails as soon as the TBB shrinks down past 25MB again. Mike said something that he's working on it.

And again as I've mentioned on IRC, I think, instead of switching from GDrive and throwing it away, we need to expand user's option on how to download Tor (See #8542)

We already have two of these options developed, tested and ready to integrate. Maybe it's just me, but I can't see why we don't already make use of them? Aren't these options better than what we currently have in GetTor, which only gives 3 hard coded mirrors away, that mostly confuses users rather than helping them? (see #9036)

comment:38 Changed 6 years ago by sukhbir

We now have Google Drive support for uploading bundles, along with Dropbox. The next step is integration with GetTor.

(Code here: https://gitweb.torproject.org/user/sukhbir/gettor.git)

comment:39 Changed 6 years ago by sukhbir

Parent ID: #8542

comment:40 Changed 6 years ago by sukhbir

Resolution: duplicate
Status: needs_reviewclosed

Discussion about this ticket is continued in #8542.

Note: See TracTickets for help on using tickets.