Opened 4 weeks ago

Last modified 30 hours ago

#32090 new defect

Blog status and where to go

Reported by: hiro Owned by: tpa
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords:
Cc: isabela, anarcat, hiro Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by hiro)

We have a few issues with our blog.

  1. Our template is broken. Comments are displayed out of the intended layout.
  1. Our blog is generating a lot of page views and has become quite expensive (more below).

Our blog has become quite popular and has received around 300k monthly
"visitors" and above 1.5M "page loads".
This is bumping our expenses significantly and we are evaluating various options regarding caching.

  • Using a CDN like Fastly, Netlify, or Cloudflare
  • Using Varnish

Caching via Varnish could create a bottleneck for our blog and a single point of failure.

In the medium term we could also evaluate what we want to do with our current blog.

  • There is a Drupal static caching project called Tome that we could use together with drupal comments from pantheon.
  • We could migrate the blog to a static content generator and use a separate system for comments.

Child Tickets

TicketStatusOwnerSummaryComponent
#32239closedanarcatsetup a cache frontend for the blogInternal Services/Tor Sysadmin Team

Attachments (1)

snap-2019.11.11-10.45.54.png (64.4 KB) - added by anarcat 41 hours ago.

Download all attachments as: .zip

Change History (5)

comment:1 Changed 4 weeks ago by hiro

Description: modified (diff)
Summary: Caching for the blogBlog status and where to go

comment:2 Changed 4 weeks ago by anarcat

ah. i saw this ticket after replying in private by email, but i'll share that analysis here. ;)

TL;DR: I'd go with varnish still, and ask the next steps on that.

The single bottleneck issue for Varnish could be a problem, but we do
have multiple locations for our servers and would be able to providing
multiple redundant servers without too much problems if that becomes an
issue. I would certainly advocate towards creating at least two
frontends to start with.

As we discussed last week, we already have a (~free) contract with
Fastly, so if we want to go the "CDN" way, it would be a good
option. They say they don't log/track their users, but I'm not sure it
would be a great move in terms of "publicity". I'm also not quite sure I
trust Fastly with doing the right thing here, ultimately, nor do I feel
that the idea of putting all our eggs in the same basket to be safe. We
also run the chance of blowing our quota there eventually if we throw
everything in Fastly.

I would assume CF is out of the question, and I don't know enough about
Netlify to speak about it...

It would be useful to know a little more what "page loads" mean. The
300k "visitors" and 1.5M "pages" figures are similar to what we see in
the dashboard, but in terms of server resources, actual raw numbers
(megabits per second or total gigabytes, and "hits" per second, as
oposed to pages) would be more useful to evaluate our capacity. What's a
"page" for example? Is that one page load, with all extra resources like
CSS and images? While that's useful for them because it's their primary
driver (because it's drupal fighting with PHP and the database to create
the page on the fly), for us at the caching layer, we don't care about
the type of content as much. :)

Finally, I looked at Tome briefly. There were various modules like this
in Drupal's history, the one I knew about before today is called "boost"
but hasn't been ported to D8 it seems. Tome is interesting, as it does
allow the creation of a static site in front of drupal, and we could
then share it on the mirror system, but then it still means we need to
deal and pay with pantheon for the hosting, which still seems like an
expensive proposition for basically a glorified text editor. I'm not
sure how "just sending the comment links" would work in practice, but
maybe it can be done too.

Anyways, Tome would take time and effort to setup, and since we are
still considering our long-term options here, I wouldn't advise for that
solution just yet and just start working more concretely on how to setup
the varnish frontends, provided we have confirmation on the
numbers. With a rough guesstimate, 1.5M "pages" is about 23Mbit/s on
average during the month, something we could probably absorb in the
existing infrastructure without too much troubel. But that's assuming
just the 5MB frontpage, having better numbers would help here
tremendously.

Changed 41 hours ago by anarcat

comment:3 Changed 41 hours ago by anarcat

problem 2 should be solved:


we're taking about 88% of the traffic out of the blog, which should drastically reduce the costs. a 88% reduction should bump us from the peak 435000 visits (the 300k visits per month package, 1000$/mth or more), down to around 52k visits per month, which is about the metric for "medium" package (300$) or, worst case, the "large" package (150k, 600$).

we'll see the actual result at the end of november, i guess!

next up is the design issues and deeper underlying issue with the blog maintenance.

comment:4 Changed 30 hours ago by pili

We had a meeting in Stockholm about the blog, you may find the following notes interesting also: https://trac.torproject.org/projects/tor/wiki/org/meetings/2019Stockholm/Notes/Blog

It's not so much about the hosting but mainly about the broken template issues and whether we want to continue using Drupal

Note: See TracTickets for help on using tickets.