Opened 7 years ago

Closed 5 years ago

Last modified 4 years ago

#9670 closed task (implemented)

Disable exploratory client circuit builds during botnet

Reported by: arma Owned by:
Priority: Medium Milestone:
Component: Core Tor/Tor Version:
Severity: Normal Keywords: tor-auth
Cc: mikeperry, sebastian, ln5, micah, weasel Actual Points:
Parent ID: #9657 Points:
Reviewer: Sponsor:

Description (last modified by arma)

When clients find that they're failing a lot of circuits, they back off from their computed cbt value, which puts them in the "launch a bunch of test circuits so we can get a better estimate of our cbt" phase. By default they launch 100 circuits, one per circuit_build_times_test_frequency(), from circuit_build_needed_circs() which calls circuit_predict_and_launch_new() which does

  if (num < MAX_UNUSED_OPEN_CIRCUITS-2 &&
      get_options()->LearnCircuitBuildTimeout &&
      circuit_build_times_needs_circuits_now(&circ_times)) {
    flags = CIRCLAUNCH_NEED_CAPACITY;
    log_info(LD_CIRC,
             "Have %d clean circs need another buildtime test circ.", num);
    circuit_launch(CIRCUIT_PURPOSE_C_GENERAL, flags);

I think while our network is overloaded (#9657), we should turn this behavior off.

To do it, we need to set

  cbtdisabled=1
  cbtmincircs=1

in our consensus params.

(We need to set both because there's a bug where it doesn't look at cbtdisabled when making the decision. But I'll open that as a separate ticket.)

Child Tickets

Change History (21)

comment:1 Changed 7 years ago by arma

Oops. Looks like this ticket can't work.

The lowest allowed value for cbtmincircs is 1, but circuit_build_times_enough_to_compute() checks

  return cbt->total_build_times >= circuit_build_times_min_circs_to_observe();

and if cbtdisabled is 1, then it resets total_build_times to 0.

Fuck.

comment:2 Changed 7 years ago by arma

Description: modified (diff)

comment:3 Changed 7 years ago by arma

Looks like we might be able to work around the bug by setting cbttestfreq to hours rather than 1 minute?

comment:4 Changed 7 years ago by arma

Summary: Disable exploratory client circuit builds (set cbtdisabled to 1 and cbtmincircs to 1)Disable exploratory client circuit builds during botnet

comment:5 Changed 7 years ago by mikeperry

I *think* if we set the following, it will cause CBT to very slowly compute a timeout and then ignore it:

  cbtmincircs=10 # I think 3 is the lowest that is safe, but some breathing room is nice
  cbtmintimeout=60000 # in ms
  cbttestfreq=1000000 # (1 meeelion seconds or so sounds about right)

We should test this on a couple real Tor clients before rolling it out as consensus parameters. It may still do strange things (like emitting a lot of warns or something, esp if cbtmincircs is low enough to produce weird statistics). The test clients should use a lower test frequency, so we can test what happens when they actually do learn the timeout.

Here's the equivalent #defines in or.h:

  CBT_DEFAULT_MIN_CIRCUITS_TO_OBSERVE
  CBT_DEFAULT_TIMEOUT_MIN_VALUE
  CBT_DEFAULT_TEST_FREQUENCY
Last edited 7 years ago by mikeperry (previous) (diff)

comment:6 Changed 7 years ago by mikeperry

One problem with this is that when we remove it, clients will all start building more test circuits again, once per minute. We'll have to crank up mincircs slowly, and turn down testfreq slowly for a while...

comment:7 Changed 7 years ago by mikeperry

Duh. Skruffy just caught that cbtmintimeout is ms. It needs to be 60000, not 60.

comment:8 Changed 7 years ago by mikeperry

Actually, maybe we want to let the test frequency be low enough to compute a discarded CBT sooner rather than later, otherwise we're left with a very low idle timeout in circuit_expire_old_circuits_clientside(), which may also increase the number of circuits that get built.

comment:9 Changed 7 years ago by mikeperry

Backing up a bit: Before we jump on disabling CBT entirely, do we have any evidence that CBT is actually causing more circuits to get built, other than during exploratory learning?

Has anyone gathered BUILDTIMEOUT_SET events from a few different clients to see if CBT is actually causing us to discard more than 20% of the circuits we attempt? My main Tor client is still timing out only 17.7% of its circuits, so CBT itself is working fine.

Last edited 7 years ago by mikeperry (previous) (diff)

comment:10 in reply to:  8 Changed 7 years ago by arma

Replying to mikeperry:

Actually, maybe we want to let the test frequency be low enough to compute a discarded CBT sooner rather than later, otherwise we're left with a very low idle timeout in circuit_expire_old_circuits_clientside(), which may also increase the number of circuits that get built.

The low value in circuit_expire_old_circuits_clientside() is 10 minutes, compared to 60 minutes normally.

I think there's no reason to induce any more circuits than we'll make naturally, by making circuits to handle port 80 on startup and to handle whatever actual use there is after that.

So in sum, I'm a fan of cbtmincircs=10 cbttestfreq=1000000.

As for setting cbtmintimeout, I'm not so clear on the expected benefit here. I guess it would make us build fewer circuits, since we're more willing to use crappy circuits. But there could be a steep cost in mean performance.

I'm inclined to start out trying just mincircs and testfreq.

comment:11 Changed 7 years ago by mikeperry

Setting cbtmincircs=10 may cause people to learn outlier timeouts. IIRC, the statistics from the initial experiments indicated that the learned timeout was unstable below 25, and still sometimes produced outliers at 25-50.

The experiment to run is to make a script to select N-subsets of the timeout counts from a Tor state file that has 1000 timeouts recorded already, and see for what values of N the new timeout Tor computes is not significantly different than the timeout computed from the full state file. I bet if you do N=10 you will see a ton of variance in the resulting timeout that Tor computes.

comment:12 Changed 7 years ago by arma

Ok. Then I am a fan of simply cbttestfreq=1000000. That should cut down the rate of circuit creation assuming most clients are often in exploratory mode, yes?

I have put that param in place on moria1.

comment:13 Changed 7 years ago by arma

It is now in place in the consensus. We'll see if anything changes.

comment:14 Changed 7 years ago by nickm

Milestone: Tor: 0.2.5.x-finalTor: 0.2.???

comment:15 Changed 6 years ago by arma

moria1 and tor26 cranked cbttestfreq down to 1000. We'll see if anything explodes.

comment:16 Changed 6 years ago by arma

Shall we let cbttestfreq go back to normal here? There are still a million or so bots (give or take a million), but they don't seem that dangerous here.

In retrospect, if we'd defined a new consensus param for ntor-using clients to look at, we could have left the old param really high but allowed newer clients to use cbt inhindered.

comment:17 Changed 6 years ago by arma

To be clear, I have put cbttestfreq down to 60 on moria1. That's its default:

#define CBT_DEFAULT_TEST_FREQUENCY 60

comment:18 Changed 5 years ago by arma

Cc: sebastian ln5 micah weasel added
Severity: Normal

I think we should resume doing what the code does, and stop overriding it with a consensus param.

In the short term, the simple way to make this happen is for some more directory authority operators to set cbttestfreq=60 in their consensus params.

Once the param is back to 60, we can then close this ticket.

comment:19 Changed 5 years ago by arma

Resolution: implemented
Status: newclosed

Ok! The param is back to 60. I wonder if the world will explode!

I am going to close this ticket optimistically in hopes that it doesn't explode.

comment:20 Changed 4 years ago by teor

Milestone: Tor: 0.2.???Tor: 0.3.???

Milestone renamed

comment:21 Changed 4 years ago by nickm

Milestone: Tor: 0.3.???

Milestone deleted

Note: See TracTickets for help on using tickets.