Opened 4 months ago

Last modified 4 days ago

#30716 assigned task

Improve the obfs4 obfuscation protocol

Reported by: phw Owned by: phw
Priority: High Milestone:
Component: Circumvention/Obfs4 Version:
Severity: Normal Keywords: sponsor28, anti-censorship-roadmap-october
Cc: arma, cohosh, gaba, phw, robgjansen, msherr, dcf Actual Points:
Parent ID: Points: 20
Reviewer: Sponsor: Sponsor28-must

Description (last modified by phw)

As part of our work for Sponsor 28, we will evaluate and improve the obfs4 obfuscation protocol, which may result in obfs5.

Roger started the discussion on our anti-censorship-team mailing list. Relevant reading is the CCS'15 paper Seeing through Network-Protocol Obfuscation and the S&P'16 paper SoK: Towards Grounding CensorshipCircumvention in Empiricism.

Let's use this ticket to keep track of this effort. Below is a list of ideas that we may or may not want to incorporate in obfs5.


Obfs4 already implements randomisation for packet lengths and inter-arrival times but there are other protocol aspects that we can randomise. Note that the adoption of these strategies may complicate censorship analysis: if obfs5 instance X looks very different from obfs5 instance Y, then X may end up getting blocked while Y still works. Instead of saying "obfs5 is blocked," one may then have to be more specific and say "the obfs5 instances that rely on UDP are blocked."

  • Payload: All bytes that obfs4 writes to the wire are randomly distributed. These high-entropy packets may or may not be common on the Internet. We could evade a "high-entropy filter" by having obfs4 servers derive a formal language from the shared secret. This language could, say, use dummy clear-text headers. The LibFTE library may be helpful here.
  • Cover traffic: dcf explains that obfs4 only sends data when it's given data to send. To improve on this, as dcf suggests, we could make obfs5 send data even when the application has nothing to send.
  • Packet directions: An obfs4 flow begins with the client sending data to the server. We could randomise packet directions and have, say, the server talk first with a server-specific probability.
  • Transport protocol: An obfs4 server could talk either TCP or UDP or SCTP. This may very well not be worth the effort.

Lessons learned from CCS'15 paper

  • DPI boxes tend to classify flows by only inspecting the first N packets of a flow. Keeping state is expensive, after all. We could exploit this by relaxing our obfuscation techniques after N packets to increase throughput.
  • The paper's data set may not be representative of what countries or ISPs would see:
    • It's "only" a university uplink. Universities typically have policies that prohibit file sharing such as BitTorrent. BitTorrent's "message stream encryption" may look similar to obfs3 and obfs4.
    • The data sets are from 2014, 2012, and 2010, respectively. That's a long time in Internet years.
    • The detectors' false positive rates are non-trivial and, as the authors point out themselves, would be problematic for a censor given that non-obfuscated traffic significantly outweighs obfuscated traffic.
    • Does the data set only contain one obfs4 server instance? This may have affected their results.


  • yawning writes that obfs4 doesn't easily support backward incompatible protocol alterations.
  • Crazy idea: Use a modified TCP stack that ignores RST and FIN segments, so the GFW's on-path devices cannot tear down the connection. Instead, the obfs5 protocol could signal the end of the connection in an authenticated control frame. We could ignore RST and FIN segments by using firewall rules, or to get more crazy, by shipping a user space TCP stack (this may be easy to fingerprint, though).

Child Tickets

#30986assignedphwUnderstand the "long tail" of unclassifiable network trafficCircumvention

Change History (14)

comment:1 Changed 4 months ago by yawning

One of the design deficiencies of the obfs4 protocol is that it doesn't easily/efficiently support backward incompatible protocol alterations.

There are ways around this, but at that point, people are better off writing a new different/backwards incompatible protocol entirely, that fixes a number of the design flaws in the underlying protocol.

comment:2 Changed 4 months ago by dcf

The obfs4 framing format is pretty nice, in that is allow arbitrary shaping: both client and server can send any amount of data, at any time. The only exception is at one point during the handshake: after the client has sent the MAC indicating the end of its padding, the client must remain silent until after the server has sent its part of the handshake. You can see the gap in the bottom two graphs at

So one desideratum from me is that the protocol should allow either side to send any amount of data at any time, and have it correctly interpreted as padding or meaningful data. Ideally it should even be possible for the server to send data before the client has sent anything.

comment:3 Changed 4 months ago by yawning

The framing could use better cryptography and a more sensible design overall, but there are larger deficiencies in the protocol.

So one desideratum from me is that the protocol should allow either side to send any amount of data at any time, and have it correctly interpreted as padding or meaningful data. Ideally it should even be possible for the server to send data before the client has sent anything.

At one point I had thoughts of how I would like to implement something like this, but it's been years since I gave serious thought about this problem. I personally would have felt uneasy about a responder-speaks-first design.

comment:4 Changed 3 months ago by phw

Description: modified (diff)

comment:5 Changed 3 months ago by phw

Description: modified (diff)

comment:6 Changed 3 months ago by robgjansen

Cc: robgjansen added

comment:7 Changed 2 months ago by phw

Cc: msherr added

comment:8 Changed 2 months ago by gaba

Keywords: anti-censorship-roadmap-august added; anti-censorship-roadmap removed

comment:9 Changed 4 weeks ago by phw

Description: modified (diff)

comment:10 Changed 3 weeks ago by phw

Cc: dcf added

Website fingerprinting attacks typically operate on traffic traces that are frequently encoded as sequences of the form:

<time>,+/-<packet length>

+<packet length> refers to packets going from the client to the server and -<packet length> refers to packets going from the server to the client. For example:


Interestingly, packet lengths may not even be necessary. In their CCS'18 paper, Sirinam et al. write in Section 5.1.1:

However, we performed preliminary evaluations to compare the WF attack performance between using packet lengths and without packet lengths, i.e., only packet direction, as feature representations. Our result showed that using packet lengths does not provide a noticeable improvement in the accuracy of the attack. Therefore, we follow Wang et al.’s methodology and consider only the direction of the packets.

The traffic trace above can therefore be reduced to:


Note that obfs4 makes no attempt to defend against website fingerprinting attacks. Its goal is to escape protocol classification but these two problems (and their respective attacks) overlap to some extent, which is why obfs4 would be better off with defences against such attacks.

As dcf already pointed out, obfs4 only sends data when the application (e.g., Tor) has data to send. Then, depending on what iatMode is used, obfs4 may append padding to the application's data and add inter-arrival delays. Coming back to the example above, obfs4 can only extend a packet burst but not break a burst. That is, obfs4 can turn the packet sequence


into the sequence

+1 (padding packet, which extends a burst)

but not into the sequence

-1 (padding packet, which breaks a burst)
+1 (padding packet, which breaks a burst)

I spent some time looking into ways to fix this issue. It turns out that we can add the ability to break packet bursts to obfs4 without losing backwards compatibility, allowing a brand-new, burst-breaking obfs4 client to talk to an old obfs4 server (however, see below for a caveat). I implemented a simple proof-of-concept, for now called sharknado, in my feature/30716 branch. The idea is simple: instead of having obfs4 write directly to its socket, it now writes to the SharknadoConn struct, which implements the net.Conn interface. After each call to Read, there's a 1 in 10 chance to send padding, regardless of if the application has data waiting or not.

There are several remaining challenges:

  • Effectively breaking bursts may require the client and the server to cooperate. For example, when the client receives the beginning of a burst, the adversary (who's somewhere between the client and the server) may already have seen the entire packet sequence, so we cannot break it anymore. We may be able to address this by having the server send only a few packets of its burst and then waiting until it received the client's burst-breaking packets.
  • We should find a way to make obfs4's packet sequences server-specific by incorporating the server's shared secret into the sequence generation process, just like it's done for packet lengths and inter-arrival times.
  • We need to build an evaluation framework to understand what works and what doesn't.

Any thoughts?

comment:11 Changed 3 weeks ago by cohosh

Nice! Thanks for sharing the update on this!

I have a few questions and comments about the website fingerprinting approach. The observation that these two problems have some overlap is a good one, but it's also worth noting that it's not necessarily the case that implementing website fingerprinting defences will be strictly better for a pluggable transport from an obfuscation point of view.

The main goal for a website fingerprinting defence is to prevent an attacker from learning which site you're visiting, not from learning that you are using Tor or even that you are using a website fingerprinting defence. And while the website fingerprinting portion is good to have, I think the current understanding of website fingerprinting of Tor traffic is that it makes more sense to apply to Onion Services at the moment and isn't urgent for the rest of Tor traffic yet? This is also being actively worked on I believe. Of course PTs don't have to only be used with Tor and this would make them better if used for other anti-censorship tools.

  • We need to build an evaluation framework to understand what works and what doesn't.

So my concern here is that even though the packet breaking is probabilistic (and eventually variable, etc.), can we do it without adding a fingerprintable feature to the obfuscation traffic? I suppose this is related to the "long tail" argument that we have, where hopefully by making the sharknado traffic look less and less like any regular thing, we're increasing the uncertainty a censor would have in blocking it.

And it brings us to your bullet point above. It would be an interesting research question perhaps to take a look at existing website fingerprinting work and see how identifiable the defences are as they are implemented in those works given that the adversary knows the client is using Tor.

  • We should find a way to make obfs4's packet sequences server-specific by incorporating the server's shared secret into the sequence generation process, just like it's done for packet lengths and inter-arrival times.

Was the purpose of this to maintain some consistency for censors who are observing all traffic to a specific bridge IP address (which might make it less suspicious)? Or to try to introduce some variability for different obfs bridges in order to add to the confusion?

comment:12 Changed 6 days ago by phw

Here's a brief overview of where we're at. We should improve both obfs4's payload and flow obfuscation. My above comment gave an overview of how we can improve flow obfuscation. To clarify: blindly incorporating WF defences is unlikely to succeed. We should instead extend obfs4's flow obfuscation and build an evaluation framework (which can be inspired by WF work).

The main issue with obfs4's payload obfuscation is that it consists entirely of uniformly distributed bytes for which Wang et al. built a reliable classifier in their CCS'15 paper. We should find a way to reduce obfs4's per-packet entropy – at least for the first n packets, assuming that our adversary classifies a flow in the first n packets. Keeping flow state is costly, after all.

We identified two approaches for reducing obfs4's per-packet entropy:

  1. Adopting LibFTE. We could derive regular expressions from obfs4's shared secret that we then feed into LibFTE. These regular expressions should make obfs4 look like "a protocol with structure."
  1. Building an entropy "modifier". A silly implementation of this idea could insert a 0-byte after every second obfs4 byte, which would reduce the stream's entropy. Again, the way this entropy modifier works should be derived from obfs4's shared secret. We should parameterise the modifier with an inflation factor i, which we can gradually scale back to 0, to improve goodput once our adversary classified the flow.

As cohosh pointed out, the challenge lies in making these improvements without having us stand out.

comment:13 Changed 5 days ago by phw

We briefly discussed the interaction between Tor's WF defences and obfs4's flow obfuscation in #tor-dev. Here's a summary:

  • We want a clear separation of responsibilities: Tor's circuit padding defends against WF attacks while obfs4 defends against traffic classification.
  • We want defence in depth. If an adversary breaks obfs4, she should not be able to fingerprint the encapsulated Tor stream, to learn what website the user is visiting.
  • That said, the defence in depth should come with as little overhead as possible. Every padding byte (be it from obfs4 or the application) is a byte that is no longer goodput. If performance suffers too much, users will turn to other circumvention tools.

Here are some additional research questions:

  • Both obfs4 and Tor add padding to defend against traffic classifiers and WF attacks, respectively. Can we combine these two systems in a smart way that minimises overhead while retaining our security properties? (Keep in mind that obfs4 is only present between client and bridge while Tor's WF defences may be present between client and middle relay.)
  • Can obfs4's padding negatively affect Tor's padding and vice versa?
  • How does the application data that goes into obfs4 affect our resistance to traffic classifiers? In other words: Is obfs4-transporting-tor more resistant to classifiers than obfs4-transporting-vpn?

comment:14 Changed 4 days ago by gaba

Keywords: anti-censorship-roadmap-october added; anti-censorship-roadmap-august removed
Note: See TracTickets for help on using tickets.