Opened 10 years ago

Closed 8 years ago

#1149 closed defect (wontfix)

Impossible to download files bigger then 32MB

Reported by: doda Owned by: loafier
Priority: High Milestone:
Component: Polipo Version: 0.2.1.20
Severity: Keywords:
Cc: doda, nickm, arma, loafier Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by phobos)

This is a bug in polipo not in tor, but since you included a 2 years abandoned app with knows bugs in a stable release of tor I will report it here in case you were unaware.

Bug #1
When downloading a file bigger then the chunkHighMark value in polipo.conf, currently set to 33554432 bytes, polipo stops sending output to the browser, but still continues to download the file from tor.
When this happens the browser is unable to disconnect the download or retry it because polipo blocks the requests until the connection is terminated by the server side or through the tor network map.

Bug #2
When downloading a file if the connection is closed by the server side or through tor, polipo automatically generates a background request to download the file again, however the way it does this contains a bug that instead of downloading from the part where the download was cut, it starts at the beginning of the file again, the result is a waste of bandwidth, and a long wait while polipo catches up to where the download stopped and starts outputting data to the browser again.

Bug #3
When using a download manager that supports resuming, the download manager will usually resume by sending a http header like "Range: bytes=1000-" indicating that the server should start sending from byte 1000.
However polipo strips this header from requests if the URL is one that has previously failed, meaning the download starts from 0 again.
This can be tested by use a download manager like getright, and pause the download and then try to resume it.

The combination of the 3 bugs above make it impossible to download a file larger then 32MB via tor, you cant download it, and cant resume it when it fails.

For speeding up testing, you can remove the "socksParentProxy" line from polipo.conf to make it connect to the internet directly, making it much faster to download 32mb and observe the bugs.

[Automatically added by flyspray2trac: Operating System: All]

Child Tickets

Change History (9)

comment:1 Changed 10 years ago by noino667

Polipo is not abandoned, it has a responsive developper and active mailing list(s) to whom you should report IMHO.

URL : http://sourceforge.net/mailarchive/forum.php?forum_name=polipo-users

Just a FYI & I'm a just plain user.

comment:2 Changed 10 years ago by phobos

re-assigned to the correct queue.

comment:3 Changed 10 years ago by xtoaster

upload many have similar problem. Some times, uploading always fail. Not sure how exactly it happened. Hm, the factors looks complex to me.

comment:4 Changed 10 years ago by arma

I just encountered this combination of bugs while testing a new feature in Tor.

I started a wget of a 300MB file through Polipo to Tor.

At about the 28MB mark (I guess that corresponds to 33 million bytes), my wget
stalled. But my Tor was still fetching stuff happily. Ok, so that's bug #1 above.

Then I killed my wget. And Tor kept fetching stuff! So it looks like if anybody
tries to download something huge over Tor via Polipo, even if they give up and
cancel the download, they still end up hosing the Tor network? This is really bad.

comment:5 Changed 10 years ago by loafier

Ugh, ok. I'll take a break from the DNS stuff and take a look at this. I'm
wondering if #1 happens all the time, or only some of the time... Hopefully
we'll grow some unit tests some day and this sort of thing will be caught
sooner.

comment:6 Changed 10 years ago by loafier

It looks like if the client end falls behind by a number of chunks,
the deadlock as described in #1 occurs when Polipo tries to reclaim
memory by nuking unlocked object chunks. Polipo continues to download
the object, but the client end can't catch up because of the missing
chunks, and so it stalls out.

comment:7 Changed 10 years ago by loafier

Polipo's design makes this bug a bit tricky to solve completely,
although hopefully commit 7a18baeca will make things a bit better.

I'm leaving this bug open for now.

comment:8 Changed 9 years ago by jch

Chris,

This is definitely a bug -- Polipo should be able to forward files larger than available memory fine. Your intuition that it is a locking bug definitely fits the symptoms.

Commit 7a18bae certainly looks good to me.

--Juliusz

comment:9 Changed 8 years ago by phobos

Description: modified (diff)
Resolution: Nonewontfix
Status: assignedclosed

Juliusz has taken over polipo maintenance again. He's asked the authors of existing bugs to email them to his mailing list at https://lists.sourceforge.net/lists/listinfo/polipo-users

Note: See TracTickets for help on using tickets.