Monday, February 29, 2016, 14:30
Discussion on doing UX research yourself.
ASL19 does user studies, mainly on Psiphon.
Maybe 10% of users give feedback. But: there is some selection bias in the users who leave feedback: the users who don't leave feedback are different in other ways. ASL19 says they perceive the opposite, because the users who leave feedback are the ones who have trouble connecting, and may be less experienced. They leave feedback on e.g. Facebook. Either way, it's important to be aware of this effect.
The users have a strong incentive to give feedback, because they are so invested, it's a tool they use every day.
How do you incorporate the user feedback? What they say they want might differ from what they really want. What about sending out a survey?
Another example: when redesigning the Windows UI and UX, made a live mockup on the web site, got a lot of feedback, including stuff like fonts and colors. E.g., the Psiphon branding was red, but users associated red with blocking and censorship. Also, putting the server selection on the same screen as the connect button, users like feeling in control, even if they choose the default. Developers were convinced by user response in this experience to make different choices.
Results from the experiments were collected in many ways, e.g. through email, Facebook. They have a lot of users, enough to get good quantitative feedback, but hard to do a systematic study with distributed users. For example, if you could use remotely measured metrics, you could have a large, unbiased sample of users. Maybe e.g. total time to completion, success rate. But problematic in e.g. Iran, users don't want any kind of client-side analytics. So perhaps you can only instrument and measure your live web site mockup, before deploying it.
Could Tor Browser have an option after bootstrapping, like "do you want to send anonymous data"? Would that work? Could get at least a portion of users that way.
In Tails, when you start Tails and connect to Tor, it visits a web page to check if there is a security upgrade, and they do statistics on that web page to count the number of Tails boots per day. When users find out that these statistics exist, some of them freak out, even though it's not individual data. If you tell them, they may be irrationally afraid of not real problems. But if you don't tell them, and they find out, they can feel deceived.
Where are people getting their ideas about what Tor does? Where are people using instead of Tor? What do they think about how e.g. the network works?
Users feel motivated and valued when they talk to developers, get feedback, and feel like their voice is heard. Queries to the Tor help desk get help, but the help desk does not go out farther to other developers. At e.g. Google, the people who do user help, a big part of their job is distilling user reports and forwarding them to devs. There's a need for that.
Tor would also benefit from positive user stories. Beneficial uses of Tor in the public. Tor perhaps has more difficulty than e.g. Psiphon, with its focus on anonymity rather than pure access. But there are users who use it for access rather than anonymity. The tor-assistants list gets some positive "who uses Tor" stories, not sure what happens to those.
Some discussion on the redesign of the Tor web site.
From a large corporate perspective, with all the papers that come out, like "we broke Tor this way", there's a perception among people that Tor is unsafe. Sometimes the attacks are highly specific, or unrealistic or speculative. Tor is still safe or safer for people to use, against more realistic threat models.
The Tor blog reveals demand by users for a kind of feedbacck channel. Users leave feedback on the blog that they don't leave otherwise (on a mailing list or IRC, for example). Why not have a Facebook group, Slack, etc. How else are users finding out about releases than the blog? We should be responsive on those channels. On the other hand, if what we want to gather is user feedback, does it need to be in a public space, like a blog or IRC channel? Wonder if we can build ways of receiving feedback that are private. In Tails there is Whisperback, maybe not so well known, but it helps you send in an anonymous bug report. 8 to 10 reports per day. DOn't have to sort through spam, flames, etc. Psiphon has a built-in feedback UI, it's the main source of feedback. It sends feedback encrypted back to Psiphon. Maybe build in a feedback panel to Tor Browser? The channels serve different purposes: public places for positive image, private channels for good feedback. Could also offer a Tor Browser UI to get news and updates about Tor, two-way feedback.
Useful feedback that users provided: the Farsi version of Tor Browser had the UI flipped right to left. Users actually preferred the English Tor bundle, and said so via feedback.
Advice for Tor: be in public, that's where users are. Even if users are on unsafe Telegram, you have to be there in order to reach them. They'll keep using Telegram if you're not there.