Opened 7 years ago

Closed 5 years ago

#6978 closed enhancement (wontfix)

Create Magnet links for download

Reported by: ioerror Owned by:
Priority: Medium Milestone:
Component: Webpages/Website Version:
Severity: Keywords:
Cc: g.koppen@…, admin@…, sukhbir.in@…, nima@…, adrelanos@…, erinn Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I've been talking with a lot of Free Software projects about how they don't want to serve stuff up over HTTPS - especially bulk data. I've heard a lot of user blaming, balking at even trying to be secure, the point of gpg signatures, etc.

I think that there is a middle ground and it is probably worthwhile to explore. We could also use it as a test - it seems like an easy test too.

Basically, I propose that over HTTPS, we have a page that links to all of our downloads, with GPG signatures and the file they wish to download as BitTorrent Magnet links.

Here's one I created that has no seeders and no need for them, I might add:

magnet:?xt=urn:btih:696513360665ad8bc398126cba2e688d882ed5cd&dn=TorBrowser&as=https%3A%2F%2Fwww.torproject.org%2Fdist%2Ftorbrowser%2Flinux%2Ftor-browser-gnu-linux-x86%5F64-2.3.22-alpha-1-dev-en-US.tar.gz&as=http%3A%2F%2Fj6im4v42ur6dpic3.onion%2Ftor-package-archive%2Ftorbrowser%2Ftor-obfsproxy-browser-2.3.22-alpha-1%5Fen-US.exe&as=https%3A%2F%2Farchive.torproject.org%2Ftor-package-archive%2Ftorbrowser%2Flinux%2Ftor-browser-gnu-linux-x86%5F64-2.3.22-alpha-1-dev-en-US.tar.gz

The nice thing about that Magnet URN is that it includes (three) urls as backups - so anyone who clicks on it will fetch it over a .onion via HTTP or via two different HTTPS urls. I think that means that merely by offering the files our first users will get a normal download via HTTPO or HTTPS. They then become the seeders - no seeding box needed!

I think it should be rather straight forward to automate the creation of urls for our use too.

The main question for me is how well the 'as=' ( see https://en.wikipedia.org/wiki/Magnet_URI#Normal_.28as.29 ) field actually works. Do BitTorrent clients actually download it over HTTPS properly?

Child Tickets

Change History (17)

comment:1 Changed 7 years ago by ioerror

I think if we did use this, I'd also make the urls include the following trackers:

udp://tracker.openbittorrent.com:80
udp://tracker.publicbt.com:80
udp://tracker.ccc.de:80
udp://tracker.istole.it:80

comment:2 Changed 7 years ago by gk

Cc: g.koppen@… added

comment:3 Changed 7 years ago by phoul

Cc: admin@… added

comment:4 Changed 7 years ago by ioerror

It seems that the xs parameter is more likely to work than the as parameter - though sadly, a number of clients do not respect or know about xs or as:

magnet:?xt=urn:btih:696513360665ad8bc398126cba2e688d882ed5cd&dn=TorBrowser&xs=https%3A%2F%2Fwww.torproject.org%2Fdist%2Ftorbrowser%2Flinux%2Ftor-browser-gnu-linux-x86%5F64-2.3.22-alpha-1-dev-en-US.tar.gz&xs=http%3A%2F%2Fj6im4v42ur6dpic3.onion%2Ftor-package-archive%2Ftorbrowser%2Ftor-obfsproxy-browser-2.3.22-alpha-1%5Fen-US.exe&xs=https%3A%2F%2Farchive.torproject.org%2Ftor-package-archive%2Ftorbrowser%2Flinux%2Ftor-browser-gnu-linux-x86%5F64-2.3.22-alpha-1-dev-en-
US.tar.gz

comment:5 Changed 7 years ago by rransom

Do magnet links work at all if no BitTorrent clients are already running for that BTIH and serving the torrent info dictionary? (Without the info dictionary, a client would not know the torrent's piece size or be able to verify the contents of the downloaded file.)

How do you expect BitTorrent clients to be able to download a file from a .onion address?

comment:6 Changed 7 years ago by phobos

Code talks. Submit something that actually automates this process, otherwise no one is going to do this.

comment:7 in reply to:  6 Changed 7 years ago by ioerror

Replying to phobos:

Code talks. Submit something that actually automates this process, otherwise no one is going to do this.

We're talking a bash script that hashes each file here. That and the file name are the unique properties in each url - it couldn't really get easier - it might even make sense to put this into the .wml website build process. I haven't decided yet.

comment:8 in reply to:  5 Changed 7 years ago by ioerror

Replying to rransom:

Do magnet links work at all if no BitTorrent clients are already running for that BTIH and serving the torrent info dictionary? (Without the info dictionary, a client would not know the torrent's piece size or be able to verify the contents of the downloaded file.)

I believe "work" in this case means it will simply keep trying to find something based on what was in the magnet link. I think if there is nothing there it will spin forever and when something shows up, it will notice and the file download will begin.

How do you expect BitTorrent clients to be able to download a file from a .onion address?

Via Tor over HTTP. That is the point of the xs/as parameter. It also means that there is a .onion url in the .magnet link that a user could simply manually copy and download. It doesn't hurt to include it.

comment:9 Changed 7 years ago by mo

#1 Magnets and torrents support the notion of a file, or a directory containing files. Ideally, a torrent/magnet would contain both the package and its signature. The sad thing about it is that if you add web URLs as sources, Bittorrent requires to download to be at http://weburl/directory/. So, we can create magnets for file+signature combined, and use web URLs as sources, but then the directory the user will get would not have a nice name: for example "dist" for plain Tor, "linux" for linux TBB etc.

#2 I was unable to find a torrent creation tool that I can point at multiple files. They all expect either a file, or a directory, and will add all files in that directory recursively to the torrent. The closed I could find is http://www.robertnitsch.de/projects/py3createtorrent , where you can specify regular expressions to exclude files. Unfortunately, it does not seem to support the notion of a regexp like "(?!package)*$", but looking at the sources the thing could easily be modified to simply support lists of files.

Conclusion: If we want it the easy way, we have to give the user separate torrents/magnets for a file and its signature. This is what I've been doing since 2011 at https://www.torservers.net/mirrors/torrents/ (script is at https://github.com/moba/createtortorrents ). If we want combined releases, we cannot use web URLs as source, and either would have to modify a torrent creation tool, or do some hard link magic (temporarily creating the directory structure we want on the seedbox).

Another caveat: If we want "nice" directory names, we have to derive them somehow from either the file name or the directory structure. Sounds easy, but actually is not if we want, say, "Tor Browser" or "torbrowser" as directory name (how would you programmatically derive that from /dist/torbrowser/linux/tor-obfsproxy-browser-gnu-linux-i686-2.4.9-alpha-1-dev-ar.tar.gz.asc). If we simply go with "filename without extension" (so the directory name would include version information), I don't know of a good way to remove extensions since we have some files with a single extension, whereas other files have multiple extensions like .tar.gz.

comment:10 Changed 6 years ago by mo

I now still generate the torrents using my old script, but also automatically seed them from the torservers.net box, and generate magnet URIs for the single files.

https://www.torservers.net/mirrors/torrents/magnets.txt is regenerated every 6 hours based on the content of /dist on that mirror. Old entries are purged and deleted files *removed* from the seed.

In my tests, it took my client a long time to find the seeder, but eventually it downloaded the file. We can add trackers and a filename to the magnet link, but I don't think that is necessary.

The scripts are at https://github.com/moba/createtortorrents

comment:11 Changed 6 years ago by sukhbir

Cc: sukhbir.in@… added

comment:12 Changed 6 years ago by mrphs

Cc: nima@… added

comment:13 Changed 6 years ago by proper

Cc: adrelanos@… added

Do BitTorrent clients actually download it over HTTPS properly?

I made a quick test to answer that question with transmission (and apache2, default-ssl, ssl snakeoil, tested https://127.0.0.1/testfile in browser and got an invalid certificate as expected). Then used "mktorrent --announce=http://announce.torrentsmd.com:6969/announce -w https://127.0.0.1/testfile ~/testfile".

Result: It downloaded from HTTPS, but the invalid ssl certificate was ignored.

Is this a problem?

To my knowledge, Bittorrent clients do hash verification, they still and only use SHA-1. If SHA-1 is still considered safe enough is another topic. And if Bittorrent clients do hash verification well or can be tricked, would probable be a fine topic for research.

comment:14 in reply to:  1 Changed 6 years ago by proper

Replying to ioerror:

The main question for me is how well the 'as=' ( see https://en.wikipedia.org/wiki/Magnet_URI#Normal_.28as.29 ) field actually works.

Maybe not the most reliable source to answer this, but see:
https://en.wikipedia.org/wiki/Magnet_URI#Clients_table

(xs and as columns)

Some bittorrent clients support it, some bittorrent clients do not support it, popular ones among them. For example, µTorrent doesn't support it.

So depending on how you implement it, it may happen that this magnet link works for some users while it doesn't work for others. Not sure if this would add confusion.

comment:15 Changed 6 years ago by nickm

Cc: erinn added

Adding 'erinn' to cc list of every ticket with 'helix' in its cc list -- erinn is helix's trac username.

comment:16 Changed 6 years ago by nickm

Cc: helix removed

Removing helix from cc lists -- helix is not erinn's trac username.

comment:17 Changed 5 years ago by Sebastian

Resolution: wontfix
Status: newclosed

Nobody has moved this forward in a long while, and magnet links wouldn't help with the automatic updater.

Note: See TracTickets for help on using tickets.