#32627 closed enhancement (implemented)

deploy torspec as HTML to GitLab Pages

Reported by: eighthave Owned by:
Priority: Medium Milestone: Tor: 0.4.3.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: torspec, doc, extra-review, 043-can
Cc: nickm Actual Points:
Parent ID: Points:
Reviewer: ahf, nickm Sponsor:


https://github.com/torproject/torspec/pull/96 will deploy torspec in HTML to any GitLab fork setup with CI and Pages (default on gitlab.com). it only adds one file: .gitlab-ci.yml

Once merged, the site will show up automatically on https://torproject.gitlab.io/torspec, and it'll sync every commit from the canonical repo and automatically rebuild the HTML.

The sed regexps in .gitlab-ci.yml could be used as the beginnings of a conversion to Markdown format, as needed.

Child Tickets

Change History (16)

comment:1 Changed 11 months ago by eighthave

Status: newneeds_review

comment:2 Changed 11 months ago by eighthave

Summary: deploy torspec as HTML todeploy torspec as HTML to GitLab Pages

comment:3 Changed 11 months ago by eighthave

Here's a Python version of the same script, for anyone who wants it.

#!/usr/bin/env python3

import glob
import os
import re

if not os.path.exists('public'):

titles = dict()
for f in glob.glob('*.txt'):
    with open(f) as fp:
        contents = fp.read()
    contents = re.sub(r'^\n +', r'# ', contents)
    contents = re.sub(r'\n {1,3}([^ ])', r'\n\1', contents)
    contents = re.sub(r'\n([0-9]+\. (?!http))', r'\n## \1', contents)
    contents = re.sub(r'\n([0-9]+\.[0-9]+\. )', r'\n### \1', contents)
    contents = re.sub(r'\n([0-9]+\.[0-9]+\.[0-9]+\. )', r'\n#### \1', contents)
    contents = re.sub(r'\n([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\. )', r'\n##### \1', contents)
    with open(f, 'w') as fp:
    m = re.match(r'\s*# (.*)\n', contents)
    titles[re.sub(r'\.txt$', r'', f)] = m.group(1)

with open('public/index.html', 'w') as fp:
    fp.write('<!DOCTYPE html>\n\n<html><body><h1>%s</h1><ul>' % os.getenv('CI_PROJECT_PATH'))
    for k, v in sorted(titles.items()):
        fp.write('<li><a href="%s.html"><tt>%s</tt>: %s</a></li>' % (k, k, v))

comment:4 Changed 11 months ago by teor

Component: Core TorCore Tor/Tor
Keywords: doc added
Milestone: Tor: 0.4.3.x-final
Type: defectenhancement

Thanks for this code!

I think we'd prefer python to sed, but I'll leave that to the reviewer to decide.

comment:5 Changed 11 months ago by eighthave

Having it in sed makes the whole GitLab CI process a single file, that's working. The Python script is there for anyone who wants to improve the automated conversion, or use it for doing a final conversion to a standard format.

comment:6 Changed 11 months ago by asn

Reviewer: ahf

comment:7 Changed 11 months ago by ahf

Status: needs_reviewmerge_ready

I think we should merge this for the following reasons:

  1. The output is a bit rough in the edges, but we can fix the formatting over time.
  2. We can extend it to include proposals over time and also fix their formatting over time.
  3. It will integrate well if/when we move to Gitlab ourselves.
  4. It's only a single additional file in the repository.

comment:8 Changed 11 months ago by teor

Cc: nickm added
Keywords: extra-review added

Since this is a new feature, I just want to check with nickm before merging it?

comment:9 Changed 11 months ago by eighthave

I updated the merge request to fix all blockquote issues I could find, only by adding blank newlines. And a second commit that makes the generated HTML complete with a header and footer.

comment:10 Changed 11 months ago by eighthave

The current output is visible here: https://eighthave.gitlab.io/torspec/

comment:11 Changed 11 months ago by teor

Status: merge_readyneeds_review

Putting back in needs_review, because the PR changed.

comment:12 Changed 10 months ago by ahf

Reviewer: ahfahf, nickm

I think it looks good. Teor wanted Nick to give some input here too.

Please avoid changing things in the pull-request while we are doing reviews. This makes it very hard to review anything :-) We can always do changes once things have been merged.

comment:13 Changed 10 months ago by nickm

Status: needs_reviewmerge_ready

I think this is okay; we might decide to do it differently later on, but for now, this is far better than nothing.

(In the long term, maybe we should just have our specs be in markdown directly, so they won't need a preprocessor? I'm not sure there.)

comment:14 Changed 10 months ago by eighthave

I only included this preprocessor as a temporary measure to help with migrating to Markdown (or whatever format). I don't think it makes sense to keep it around beyond its use in migrating to Markdown. I think one workable policy here would be to just say all future edits must be Markdown, then the rest of the documents can be migrated as the opportunity arises, either piecemeal or all at once.

comment:15 Changed 10 months ago by ahf

Keywords: 043-can added

comment:16 Changed 10 months ago by nickm

Resolution: implemented
Status: merge_readyclosed


Note: See TracTickets for help on using tickets.