We need a new blogging system
The current blogging system is based on Drupal 5 and heavily hacked up to remove lots of surface area for classes of attacks. However, it doesn't work so much years later. The search functionality is broken. Lots of the admin functionality is broken as well. I've resorted to using raw SQL queries to manage the system. This is less than optimal.
Options I see are:
- Do nothing and let the blog further degrade.
- Migrate to a static blog generator like jekyll.
- Migrate to modern drupal in the debian repos.
- Use RedTeam's WordPress system for a more secure wordpress installation.
- Host it somewhere else and let them worry about it, so long as we can get our data out daily.
Activity
Like it or not, the blog comments have become our forum.
The mailing lists aren't it -- and many people rightly point out that they can use Tor to safely interact with the blog comments whereas getting a usable email address over Tor these days is becoming increasingly hard. And the stackexchange thing isn't a forum either.
So I think dropping the comment section, and not replacing it with something equivalent (even if somewhere else), would be a poor move.
Trac:
Cc: N/A to lunar@torproject.orgFWIW, I use pelican for my blog, I tried a whole bunch like a year and half ago. This one is in Python, and it didn't suck as much as the other Python ones, plus the Extension API for it is really simple (I was able to write a BibTeX to anonbib-ish-thing in a couple hours). There might be better ones now, but the process of searching and trying them was painful to me -- I wouldn't really want to repeat it.
Pelican has a Disqus plugin which comes with it automatically. I tested that out, but it was pretty unusable over Tor, required registration (with an email address) on their site (which was HTTP-only). It was gross, and scripts galore, so I ditched it.
I agree with arma's comment that the blog is the safest way for users to give us feedback, we should try to find something at least as safe/non-privacy-invading to replace it.
Trac:
Cc: lunar@torproject.org to lunar@torproject.org, isis-
All in for a static blog generator backed by a revision control system.
The blog is also doing the event calendar. Should that be kept?
What about migration? We need to keep content, but do we also want to migrate comments?
Regarding comments, the main think I can think of here is social: who's taking care of them? Moderation, answering the bulk… Roger is doing a good amount of that for the current blog, but we might want to have more formal roles or processes?
(This is actually the reason why I always disabled comments for TWN posts. I don't want to feel sole responsible with the comments there and piling unmoderated ones are bad from our users point of view.)
I don't think Disqus is an option, otherwise we are going to have the same problem we are currently having with Stack Exchange: we can't trust their data retention policy.
Discourse looks nice from several aspects. That's a Rails app, not the worst to administrate but it needs a maintainer on the sysadmin side. It also have an impressive feature list and so it also needs someone to decide about how to turn all the little knobs.
-
Replying to lunar:
All in for a static blog generator backed by a revision control system.
Sounds good to me (in theory).
The blog is also doing the event calendar. Should that be kept?
I think an event calendar could be quite useful if we keep it up to date and if we make it findable for our community. It could be something very simple, like a text file we update via git, and point to from the end of the 'upcoming events' section of TWN.
What about migration? We need to keep content, but do we also want to migrate comments?
Unfortunately, my vote is yes. It sure sounds like a pain, but many of the comment sections of more recent posts (where I've put a lot of effort in) are useful resources.
Maybe that means we don't want to migrate, and instead just take static html from the old posts-with-their-comments?
Another option is to go through and extract everything perfectly into stackexchange questions and answers. I'd like that to happen, but I think it needs to be done by the community at their own pace.
Regarding comments, the main think I can think of here is social: who's taking care of them? Moderation, answering the bulk… Roger is doing a good amount of that for the current blog, but we might want to have more formal roles or processes?
I'm basically the sole blog person at this point. And it is a bit weird that we have a blog, we have helpdesk, and we have stackexchange. It seems like about half of the blog things, and an unknown fraction of the helpdesk things, could be resolved by making a good stackexchange entry and just pointing to it each time the issue comes up. But there's remaining value in both even if we do that.
As for more formal roles / processes... that's a tough one. I'd love to have some more volunteers here. But it's not clear that it would be the best use of our (at this moment limited) funding. On the third hand, here I am not doing some of the other just-as-critical things that I could be doing.
(This is actually the reason why I always disabled comments for TWN posts. I don't want to feel sole responsible with the comments there and piling unmoderated ones are bad from our users point of view.)
Makes perfect sense.
I don't think Disqus is an option, otherwise we are going to have the same problem we are currently having with Stack Exchange: we can't trust their data retention policy.
Yeah.
Discourse looks nice from several aspects. That's a Rails app, not the worst to administrate but it needs a maintainer on the sysadmin side. It also have an impressive feature list and so it also needs someone to decide about how to turn all the little knobs.
Haven't looked at it, but am open to it.
Replying to arma:
I'm basically the sole blog person at this point.
I moderate comments and do direct sql to keep the spam down. If you mean by responding to comments, sure. In reality, ain't nobody got time for that. As we've seen with stack exchange, others in the community would be great at moderating and responding to comments.
Happy to report that I've scraped all blog posts and their comments (#10479 (moved)).
Trac:
Username: ultrasandwich-
Other free software alternatives to Disqus:
Trac:
Cc: lunar@torproject.org, isis to lunar@torproject.org, isis, phoulTrac:
Cc: lunar@torproject.org, isis, phoul to lunar@torproject.org, isis, phoul, jens@kubieziel.deAccording to Roger, our blog is even more broken now, see #12949 (moved) for fun.
Options are to either just migrate the blog to some 3rd party, or find someone to convert it to modern drupal 7.
-
Replying to phobos:
Step 1: Get our drupal blog working again. Step 2: migrate the content to jekyll with a nice design. Step 3: migrate the comments to juvia or something with a similarly nice design Step 4: stop worrying about the blog
I'm ready to try the Drupal migration whenever the current blog can be exported somewhere with new admin credentials. Has there been any movement on this? https://lists.torproject.org/pipermail/www-team/2014-August/000360.html
Trac:
Username: ultrasandwich -
Replying to phobos:
I also went one step further and imported these to a barebones Jekyll installation... http://tor-blog.deadhare.com
Trac:
Username: ultrasandwich Replying to ultrasandwich:
Replying to phobos:
I also went one step further and imported these to a barebones Jekyll installation... http://tor-blog.deadhare.com
Awesome.
Replying to ultrasandwich:
I'm ready to try the Drupal migration whenever the current blog can be exported somewhere with new admin credentials. Has there been any movement on this? https://lists.torproject.org/pipermail/www-team/2014-August/000360.html
Not much progress. I have the db export. Clearing the credentials breaks drupal it seems. I have to re-import the db export and then just reset passwords I think.
-
According to this LWN article, Discourse can import comments from Drupal:
A nice touch for sites with existing communities is that Discourse includes several utilities for importing existing user and comment databases from several other forum packages (including vBulletin, phpBB, Drupal, and BBpress).
Replying to ultrasandwich:
Not much progress. I have the db export. Clearing the credentials breaks drupal it seems. I have to re-import the db export and then just reset passwords I think.
Cool, whatever works. Thanks phobos! Just keep me posted on where to grab the export when that's done.
Ended up just dropping the user table. Everything else should be clean, if not, oh well, https://people.torproject.org/~andrew/2014-09-08-torblog-clean.dump.gz
-
In talks with ultrasandwich to convert everything to jekyll/juvia since drupal update isn't going so well. Indeed I'm very close to having this totally migrated, comments included. As briefly mentioned before, keep an eye on http://tor-blog.deadhare.com for progress. I expect the comment import into juvia to be complete later today.
[update 9/12]: Juvia is up and running with 10,000 imported comments. Heroku's free db won't let me store more than that. http://torblog-juvia-4525.herokuapp.com
Anyone keen on designing/reskinning can pull the most recent code/SASS here: https://github.com/eschaefer/tor-blog.
(phobos, I can update the tor-hosted one if you send me the credentials)
Trac:
Username: ultrasandwich Trac:
Cc: lunar@torproject.org, isis, phoul, jens@kubieziel.de to lunar@torproject.org, isis, phoul, jens@kubieziel.de, jmtodaro@gmail.com-
Trac:
Username: ultrasandwich
Cc: lunar@torproject.org, isis, phoul, jens@kubieziel.de, jmtodaro@gmail.com to lunar@torproject.org, isis, phoul, jens@kubieziel.de, jmtodaro@gmail.com, ultrasandwich How I wish trac could automatically create a gantt chart and dependencies based on child tickets, and turn the milestone page into the summary of all this.
I think the flow we want is:
- Get the content migration from drupal to jekyll working.
- Get the comment migration from drupal to jekyll working.
- Design and implement the html for jekyll.
- Design and implement the html for juvia.
- Test a complete content migration to our jekyll.
- Test a complete comment migration to our juvia.
- Pick a date, migrate content and comments to new systems.
- Document how to add posts to the blog for current authors.
- Setup blog.torproject.org vhost on static systems.
- Setup comment.torproject.org vhost in Tor infrastructure.
- On date, migrate content and comments to new systems.
- Shutdown drupal blog.
- A week later, decommission motor.torproject.org (current drupal blog machine).
Hi everyone! Here is the initial design implementation for review: http://jmtodaro.github.io/tor-blog
And the code: http://github.com/jmtodaro/tor-blog
A few things:
-
I took some liberties with hover link styling. Let me know if it could be better.
-
I didn't know what to put in the page footer so I've just left the base one.
-
Added the search box (present in the mockup) but not sure if/how we want to use it.
-
Currently both blogs and events are displayed on the review page, due to lack of support for jekyll plugins on github pages.
-
The tag cloud is ugly because the method I used to list the tags is weird and incredibly hard to style. We may need some sort of plugin to handle generating a tag cloud that works properly.
-
Archives page is a placeholder.
I based this code off of a clean jekyll base (jekyll new) and integrated components from ultrasandwich's repo as needed. I think I could merge the changes into ultra's codebase without too much trouble, if necessary.
-
When I last spoke with Ultrasandwich, it sounded like he would take care of the functionality (archives, tagging, etc.) and I just need to integrate the design into his codebase. Unfortunately I've been tied up with other projects and I haven't gotten to it yet. I planned to get it going this weekend.
Current Status
Here is an updated flow list to better reflect out current status:
-
Get the content migration from drupal to jekyll working.- Complete -
Get the comment migration from drupal to jekyll working.- Complete - Front-end development to implement tags, archives, events and search within jekyll - In progress
-
Create Juvia web app and integrate into jekyll for dev/testing- Complete - Jekyll and Juvia html styling. - In progress
- Setup transitional systems for testing complete data migration
- Test a complete content migration to our jekyll. - Ready
- Test a complete comment migration to our juvia. - Ready
- Pick a date, migrate content and comments to new systems.
- Document how to add posts to the blog for current authors.
- Setup blog.torproject.org vhost on static systems.
- Setup comment.torproject.org vhost in Tor infrastructure.
- On date, migrate content and comments to new systems.
- Shutdown drupal blog.
- A week later, decommission motor.torproject.org (current drupal blog machine).
I've set up a live working copy of the dev site in its current state here: [http://tor.jmtodaro.com/blog/] Search, tagging, and archive functions have been added. The archives will require a bit more work if we want a navigation system and not just a flat list. Also I fixed the pagination links at the bottom of the page and added the ability to disable comments on posts (such as the tor weekly news) by adding "comments: disabled" to post front matter.
Design-wise everything is still a bit rough around the edges. I will move on to polishing the css when the functionality is fully implemented.
The test site currently contains 40 of the most recent blog posts and their respective comments (~3000 comments) imported to a fully working copy of Juvia. Please feel free to test the commenting system out. New comments will only appear on the test site, so go wild. If Juvia doesn't load right away, give it a few seconds as the heroku app goes to sleep after an hour of inactivity and can take it a bit to "wake up".
A word on Juvia After implementing Juvia and understanding a bit more how it works, I must say that I have some serious concerns about whether or not this is the right comment system for us to migrate to. Here are a few highlights:
- Juvia doesn't support nested replies, and only does inline comments when "replying". There is not even a "jump" link.
- This completely breaks existing comment threads and make them very difficult to follow as Juvia sorts by date and there is no context to the comment it is replying to. Compare https://blog.torproject.org/blog/tor-browser-403-released#comments to http://tor.jmtodaro.com/blog/tor-browser-403-released/#comments . A crude workaround might be to programmatically insert a snippit of the parent comment into the replies or possibly a jump link during the scrape.
- There is no way to close commenting on an old thread.
- I'm not sure there is a way to differentiate between an actual author commenting or an imposter. I could be wrong about this if the user has moderator status, but I'm pretty sure juvia only supports one moderator per site.
- Juvia is currently unmaintained. ( https://github.com/phusion/juvia )
I will begin research on other solutions unless we are dead set on using Juvia.
Last but not least, I've created a repo of all the scraped and converted data from our current blog. The most recent full dump is here: [http://github.com/jmtodaro/tor-blog-data/] When using these, it will be necessary to update the urls relative to your own dev environment.
The development site repo can still be found here: http://github.com/jmtodaro/tor-blog
Todo
- Implement author blogs/categories
- Archive navigation
- Site Description and footer links
- Display start and end times on event posts
- Clean up Gemfile cruft
- Html styling
- tag cloud
- archive nav and results page
- search results page
- comments (if we switch systems)
- markdown anomalies
- events calendar and list
-
**Status Update:
I've just updated the development site: http://tor.jmtodaro.com/
Pretty much everything is completed. We just need to decide if we really want to use Juvia for the comments. If anyone cares to weigh in on this issue, I'd be glad to hear your thoughts. Here are the results of my research so far:
Unfortunately there really aren't too many options I found suitable. All said and done I only found two other acceptable candidates.
The first is Discourse, but here are my beefs;
- it looks like it works the same way as juvia i.e. a flat commenting system that is going to break existing nested comment threads (without some sort of kludge).
- Importing the comments does NOT look like it will be a trivial matter.
- Actually has the reverse problem of REQUIRING a sign-in to comment. No Anonymous/Guest comments.
The second and most promising option I have come across is Isso: https://github.com/posativ/isso I have not yet set up a local copy to evaluate, but when I do I will report back here with my findings.
'''Status Update:
I've done some digging into several commenting systems, and I'm afraid our options aren't great. Before I get started on the differences, I would just like to point out that over the history of the blog (2007- current), it has accumulated nearly 20000 individual comments. Roughly 6000 of those were made since January 2014. So what I'm trying to illustrate is that we have a fairly active comments section that will likely continue growing.
Unfortunately Juvia and Isso (I really like Isso btw) are designed with smaller blogs in mind and are missing many features that we will likely find necessary on a blog as active as ours, such as user authentication (for authors and moderating), ability to close comments, and moderation controls to name a few. I don't believe these are viable choices unless we are willing to further develop and contribute to these projects ourselves. Or are willing to sacrifice significant features that we currently have. If we do go this route, I think it might be a good idea to simply close comments on all existing threads and render them static with the blog posts, rather than importing them.
Hashover seems kind of interesting, but it is written in php and actually seems to require living on the same server as the blog. Bleh.
So, that really only leaves 2 viable choices (in my opinion), each with their own set of problems: Discourse or Third-party (such as Disqus).
Discourse is fully featured but the way it works is a bit strange. The main problem with this one is the fact that it requires a sign-up (with our instance of discourse, it doesn't leave our server) to comment, which in turn requires email validation. I saw no easy way around this. Switching to Discourse is also known for reducing comment activity in general due to the hurdles imposed, but according to the developers this is a //feature// designed to promote more thoughtful discussion.
Which leaves Disqus. I am actually somewhat sad to say that Disqus is nearly perfect in every way. I seriously have zero complaints regarding the functionality, and migrating would be a breeze. And yes it allows anonymous/guest comments without a sign in. The main drawback of course, is it being hosted by a third-party that does who knows what with the data it collects.
So I begrudgingly recommend Disqus based on featues alone. It would hands down be the most painless transition, for both us and our existing audience. Maybe we can add some sort of warning at the top of the comments section to encourage viewers to use tbb when commenting? I did not actually test it with tbb yet, I guess we would need to make sure it doesn't pull some cloudflare bs.
I've also put together a small chart to help compare the differences, available here. I look forward to hearing your feedback on this issue.
Trac:
Cc: lunar@torproject.org, isis, phoul, jens@kubieziel.de, jmtodaro@gmail.com, ultrasandwich to lunar@torproject.org, isis, phoul, jens@kubieziel.de, jmtodaro@gmail.com, ultrasandwich, mcs-
I would suggest to take a look at
- Ghost (Blogging Platform)
It has a really nice admin page (supports multiple users) However it doesn't support comments either.
For the comment stuff we need something we can host (Disqus is not an option, it tracks users, and it requires registration - even as a guest it requires you to put your email address)
So, stay with the solutions:
- Isso
- Juvia
- Hashover
- Commentar
- Commentics
Also, (if we are obligated to use a static generator (you don't like ghost)) We can take a look at
- Antwarjs
Trac:
Username: peluza -
Replying to peluza:
I would suggest to take a look at
- Ghost (Blogging Platform)
Correct me if I'm wrong, but jmtodaro already put a bunch of good work into a really nice Jekyll solution already in place for the blog content (after I dropped the ball).
Right now we just need a good commenting system figured out. These are crucial, and need to be migrated from the current blog as well. Isso seemed like the closest thing to workable, esp regarding anonymity and threading.
Let's keep our discussion focused on this commenting solution. I'm willing to further explore the Isso solution with another person. I need to be accountable to someone else to actually get this shit done, haha.
Trac:
Username: ultrasandwich Reassigning/recomponentising phobos' tickets, since we deleted phobos' password (#15896 (moved)). Please feel free to reassign/recomponentise again if you feel there's a better category.
Trac:
Type: enhancement to project
Milestone: 2014 Tor Blog Replacement to N/A
Owner: phobos to N/A
Status: accepted to assigned-
The author of Isso seems to have made it a lot easier to set up in the last few months... I am working on a comment migration script to see if I can get those at least set up.
[UPDATE] Actually, yeah I got a migration successfully run from the JSON blob of scraped comments. Only took some tweaking of the existing migration script that the Isso author has in place for Wordpress/Disqus import.
Trac:
Username: ultrasandwich
