Opened 6 months ago

Closed 5 months ago

#32627 closed enhancement (implemented)

deploy torspec as HTML to GitLab Pages

Reported by: eighthave Owned by:
Priority: Medium Milestone: Tor: 0.4.3.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: torspec, doc, extra-review, 043-can
Cc: nickm Actual Points:
Parent ID: Points:
Reviewer: ahf, nickm Sponsor:

Description will deploy torspec in HTML to any GitLab fork setup with CI and Pages (default on it only adds one file: .gitlab-ci.yml

Once merged, the site will show up automatically on, and it'll sync every commit from the canonical repo and automatically rebuild the HTML.

The sed regexps in .gitlab-ci.yml could be used as the beginnings of a conversion to Markdown format, as needed.

Child Tickets

Change History (16)

comment:1 Changed 6 months ago by eighthave

Status: newneeds_review

comment:2 Changed 6 months ago by eighthave

Summary: deploy torspec as HTML todeploy torspec as HTML to GitLab Pages

comment:3 Changed 6 months ago by eighthave

Here's a Python version of the same script, for anyone who wants it.

#!/usr/bin/env python3

import glob
import os
import re

if not os.path.exists('public'):

titles = dict()
for f in glob.glob('*.txt'):
    with open(f) as fp:
        contents =
    contents = re.sub(r'^\n +', r'# ', contents)
    contents = re.sub(r'\n {1,3}([^ ])', r'\n\1', contents)
    contents = re.sub(r'\n([0-9]+\. (?!http))', r'\n## \1', contents)
    contents = re.sub(r'\n([0-9]+\.[0-9]+\. )', r'\n### \1', contents)
    contents = re.sub(r'\n([0-9]+\.[0-9]+\.[0-9]+\. )', r'\n#### \1', contents)
    contents = re.sub(r'\n([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+\. )', r'\n##### \1', contents)
    with open(f, 'w') as fp:
    m = re.match(r'\s*# (.*)\n', contents)
    titles[re.sub(r'\.txt$', r'', f)] =

with open('public/index.html', 'w') as fp:
    fp.write('<!DOCTYPE html>\n\n<html><body><h1>%s</h1><ul>' % os.getenv('CI_PROJECT_PATH'))
    for k, v in sorted(titles.items()):
        fp.write('<li><a href="%s.html"><tt>%s</tt>: %s</a></li>' % (k, k, v))

comment:4 Changed 6 months ago by teor

Component: Core TorCore Tor/Tor
Keywords: doc added
Milestone: Tor: 0.4.3.x-final
Type: defectenhancement

Thanks for this code!

I think we'd prefer python to sed, but I'll leave that to the reviewer to decide.

comment:5 Changed 6 months ago by eighthave

Having it in sed makes the whole GitLab CI process a single file, that's working. The Python script is there for anyone who wants to improve the automated conversion, or use it for doing a final conversion to a standard format.

comment:6 Changed 6 months ago by asn

Reviewer: ahf

comment:7 Changed 6 months ago by ahf

Status: needs_reviewmerge_ready

I think we should merge this for the following reasons:

  1. The output is a bit rough in the edges, but we can fix the formatting over time.
  2. We can extend it to include proposals over time and also fix their formatting over time.
  3. It will integrate well if/when we move to Gitlab ourselves.
  4. It's only a single additional file in the repository.

comment:8 Changed 6 months ago by teor

Cc: nickm added
Keywords: extra-review added

Since this is a new feature, I just want to check with nickm before merging it?

comment:9 Changed 6 months ago by eighthave

I updated the merge request to fix all blockquote issues I could find, only by adding blank newlines. And a second commit that makes the generated HTML complete with a header and footer.

comment:10 Changed 6 months ago by eighthave

The current output is visible here:

comment:11 Changed 6 months ago by teor

Status: merge_readyneeds_review

Putting back in needs_review, because the PR changed.

comment:12 Changed 6 months ago by ahf

Reviewer: ahfahf, nickm

I think it looks good. Teor wanted Nick to give some input here too.

Please avoid changing things in the pull-request while we are doing reviews. This makes it very hard to review anything :-) We can always do changes once things have been merged.

comment:13 Changed 6 months ago by nickm

Status: needs_reviewmerge_ready

I think this is okay; we might decide to do it differently later on, but for now, this is far better than nothing.

(In the long term, maybe we should just have our specs be in markdown directly, so they won't need a preprocessor? I'm not sure there.)

comment:14 Changed 6 months ago by eighthave

I only included this preprocessor as a temporary measure to help with migrating to Markdown (or whatever format). I don't think it makes sense to keep it around beyond its use in migrating to Markdown. I think one workable policy here would be to just say all future edits must be Markdown, then the rest of the documents can be migrated as the opportunity arises, either piecemeal or all at once.

comment:15 Changed 5 months ago by ahf

Keywords: 043-can added

comment:16 Changed 5 months ago by nickm

Resolution: implemented
Status: merge_readyclosed


Note: See TracTickets for help on using tickets.