Create a sitemap for current torproject.org website

changed milestone to %WebsiteV3

added component::webpages/website defer-new-website easy milestone::WebsiteV3 owner::isabela parent::18243 priority::medium resolution::fixed severity::normal status::closed type::task www-team labels

Trac:
tor-project-ia-proposed-030110.pdf

proposed IA from 2010 redesign

Here is the list of every HTML files created by a build of the current website (SVN r26526):

index.html.en
getinvolved/relays.html.en
getinvolved/translation-overview.html.en
getinvolved/volunteer.html.en
getinvolved/open-positions.html.en
getinvolved/mirrors.html.en
getinvolved/tshirt.html.en
getinvolved/translation.html.en
docs/tor-manual-dev.html.en
docs/tor-relay-debian.html.en
docs/tor-doc-windows.html.en
docs/running-a-mirror.html.en
docs/documentation.html.en
docs/hidden-services.html.en
docs/manual.html.en
docs/bridges.html.en
docs/pluggable-transports.html.en
docs/tor-manual.html.en
docs/trademark-faq.html.en
docs/android.html.en
docs/tor-doc-relay.html.en
docs/debian.html.en
docs/installguide.html.en
docs/short-user-manual.html.en
docs/tor-doc-unix.html.en
docs/debian-vidalia.html.en
docs/tor-hidden-service.html.en
docs/faq-abuse.html.en
docs/signing-keys.html.en
docs/tor-doc-win32.html.en
docs/faq.html.en
docs/tor-doc-osx.html.en
docs/rpms.html.en
docs/verifying-signatures.html.en
docs/proxychain.html.en
docs/N900.html.en
projects/obfsproxy-debian-instructions.html.en
projects/arm.html.en
projects/onionoo.html.en
projects/torweather.html.en
projects/projects.html.en
projects/tordnsel.html.en
projects/torbrowser-details.html.en
projects/obfsproxy.html.en
projects/obfsproxy-instructions.html.en
projects/gettor.html.en
projects/sampleproject.html.en
projects/torbrowser.html.en
projects/vidalia.html.en
press/inthemedia.html.en
press/2008-12-19-roadmap-press-release.html.en
press/2011-08-28-tor-022-stable.html.en
press/2010-03-25-tor-store-press-release.html.en
press/2010-09-16-ten-things-circumvention-tools.html.en
press/press.html.en
press/2009-03-12-performance-roadmap-press-release.html.en
torbutton/torbutton-faq.html.en
torbutton/index.html.en
torbutton/torbutton-options.html.en
eff/tor-legal-faq.html.en
eff/tor-dmca-response.html.en
download/download-easy.html.en
download/download.html.en
download/download-unix.html.en
about/torusers.html.en
about/sponsors.html.en
about/board.html.en
about/jobs-extdev.html.en
about/contact.html.en
about/gsoc.html.en
about/jobs-browserhacker.html.en
about/jobs.html.en
about/contributors.html.en
about/corepeople.html.en
about/jobs-projectcoordinator.html.en
about/volunteers.html.en
about/overview.html.en
about/financials.html.en
about/jobs-pluggabletransport.html.en
about/translators.html.en
about/jobs-lead-automation.html.en
about/jobs-translatorsupport.html.en
about/jobs-coredev.html.en
donate/donate-hardware.html.en
donate/donate-service.html.en
donate/donate.html.en
donate/become-sponsor.html.en
donate/matching-program.html.en
donate/thankyou.html.en

The following two are HTML pages that are generated from DocBook:

torbutton/en/design/index.html.en
projects/torbrowser/design/index.html

I used http://www.xml-sitemaps.com/ to automatically generate a sitemap, and I added the xml file to the www root directory. I also added the line

Sitemap: https://www.torproject.org/sitemap.xml

to robots.txt. What else needs to be done now?

We had one of these in the past. We learned the hardway if it's not 100% accurate (updated when pages are updated), the result ranking demotion in google/bing/etc are fairly draconian. How can we automatically keep this sitemap 100% accurate? Preferably, it's created at build time during "publish" of the website.

Please note this is also advertising a 3rd party service, we don't want to do that.

Also, this has the various apache dir listings in it? Seems silly to include these. See these lines for example:

1463 <url>
1464   <loc>https://www.torproject.org/dist/misc/ttdnsd/?C=M;O=D</loc>
1465   <changefreq>daily</changefreq>
1466   <priority>0.21</priority>
1467 </url>

The original point of this ticket is not to create a sitemap xml file for search engines, but to look at the information architecture of the site as a whole. We could use this IA to design a newer IA or rework the current IA for easier usability.

Replying to phobos:

The original point of this ticket is not to create a sitemap xml file for search engines, but to look at the information architecture of the site as a whole. We could use this IA to design a newer IA or rework the current IA for easier usability.

I don't want to put the website at risk of being search engine punished. I'll just remove the xml file, as it appears I added it based on misunderstanding.

This should be an easy script to write which traverses the directories, looks for .html.en files, puts them into a data structure and then outputs them to either dot format or html or something.

Trac:
Keywords: www-team deleted, www-team easy added

Replying to Sebastian:

This should be an easy script to write which traverses the directories, looks for .html.en files, puts them into a data structure and then outputs them to either dot format or html or something.

Hello, could you (or please let me know if there's a better person to ask) give me an example of what the output should look like?

Trac:
Username: wilbns

I'm thinking just graphviz DOT files (https://en.wikipedia.org/wiki/DOT_(graph_description_language))

Trac:
Severity: N/A to Normal
Keywords: www-team easy deleted, www-team easy defer-new-website added
Sponsor: N/A to N/A

Trac:
Parent: N/A to #18243 (moved)

Trac:
Status: new to assigned
Owner: N/A to isabela

Trac:
Description: Create a sitemap for current torproject.org website.

We need a current sitemap to let people understand our current information architecture.

to

During the Tor Dev Meeting in Berlin 2015, we decided on new information structure for torproject.org website: https://trac.torproject.org/projects/tor/wiki/Website/MainSiteRedesign

We also had some suggestions on what to do with the content that we were removing from the homepage.

The goal of this task is to map all the current content on torproject.org so we can make sure we are not forgetting any page and we have a plan for everything we are moving out of it.