UP | HOME

Table of Contents

Site management

Here is documented how my projects pages sites collection is maintained, mostly so I don't have to remember it later.

Source of truth

My top-level projects site is currently hosted at Sourcehut, with its top at scwork.srht.site. AWS hosts a custom domain ste.venclare.work pointing here.

Some of my projects' pages are included in this top-level documentation; others have their own sites, and when these are also hosted at Sourcehut they have their own custom domain names like https://boat-helmer.ste.venclare.work.

Nothing detects these separate sites; they'll be mentioned in the frontpage top-level site only if I make an explicit link. See Sourcehut's documentation for site hosting details.

Sourcehut Sites publish HTTPS and Gemini document trees separately, so you are accessing separate non-overlapping subtrees of content files depending on which protocol is delivering your requests. It's on you to keep them consistent.

Both content trees are mirrored (by crawling, as described below) to public.local at my home. (Test that this is operating <2026-11-17 Tue>.) Other projects' sites can do the same thing (using the same code, or variations of it) and all should. (Test this too <2026-11-17 Tue>.)

The ultimate source of truth for the content in each of these sites is the project's main source code repository. If any project's site accumulates content (like a history of CI build products) things get complicated. Because srht.site has no auto-indexing and allows only full-subtree-replacement edits, there is no reliable way to add files to a canonical history here. When needed we make a best effort to maintain the site automatically, and rely on correctness of updates WRT the tree scheme described below and on backups of the mirror site (and maybe on project-specific mechanisms) to accumulate canonical history.

Site folder tree organization

srht.site hosts static pages and assets (mostly). It imposes these limits, among others:

  • A storage size limit, currently 1GB (<2026-09-03 Thu>): source. The wording of that text suggests the size limit applies to each particular upload, not to the size of the site in aggregage, but I haven't tested this.
  • Whole-subtree updates only - no blind additions to existing subtrees, or individual page updates within a subtree. The subtree's parent does not need to exist already - the "hut pages publish" -s option does something like rm -rf then mkdir -p on its argument.
  • No auto-index is provided - if you don't know a page or directory exists I've found no way to discover it. Sort-of-exception: links that point to a directory, with or without a trailing slash on the path, will try to retrieve index.html (or index.gmi) from that dir.
  • Mime-type in Gemini is determined entirely at the server, and Sourcehut provides no way to configure the type of a link - a response for anything.gmi is annotated Content-type: text/gemini, and for anything.jpg Content-type: image/jpeg. There are probably other file endings in the map.

The Gemini protocol has limits and restrictions of its own, particularly:

  • Reliability goes down with increasing file size.

Working around these limits (when needed) involves some hacks.

  1. Large files should be HTTPS links in the .org source, with MIME types annotated explicitly. (Why? And: How is this done? <2026-11-17 Tue>)
  2. Adding content within an existing subtree, like accumulating automated build products, requires specific footwork to retrieve and update at least one link to the new content - and this doc containing the link should exist elsewhere (SCM) as it is canonical.

So here are the tactics I abide by. I trust they'll continue to be useful if I move public hosting elsewhere in the future.

Reserve filename .subtree-contents.

These are created during each site's publishing process, only where needed. There's probably a way or two for a subproject to include a file of this name, but at best it would be confusing. See below, and just don't.

Similarly, don't mention a file of this name in a link in any document in the project. Again, it's probably possible to do so without breaking stuff, but don't.

Each place a project uses .subtree-contents it's done explicitly: the processing that uses such a file doesn't search for it, instead it hard-codes its path. These files are only needed to find files that a web-crawl would NOT find, but they can list anything - including redundant files, even files that are not under its directory.

Every subtree upload can also upload listings of its directories' contents.

Every directory in a published site can contain, in addition to its content, a file listing named .subtree-contents:

.../basename/
             index.html
             more/
                  index.html
             .subtree-contents

Here the text of the file .subtree-contents listed above:

index.html
more/
more/index.html

To replace a subtree, first prepare its files under a local directory basename. Next make the .subtree-contents listing and the upload tarball, then do the upload.

$ (cd ./basename && ls .) > ./.subtree-contents && mv ./.subtree-contents ./basename
$ tar -c basename -czf ./upload.tar.gz .
$ hut pages publish ./upload.tar.gz -d mydomain.srht.site -s wherever/basename
$ rm ./basename/.subtree-contents ./upload.tar.gz
  • Managing the site root

    If a project might need to update it's top index.html without rebuilding the entire site (because of incremental additions like CI build products), the root index.html (and index.gmi and whatever other content scheme needs a root document) can be invariant, providing only a protocol-specific (or file-type-specific) redirect to the corresponding file /root/index.html (or /root/index.gmi or whatever). These can be changed by subtree uploads (hut pages publish -d sitename -s root). Updating these redirect files requires rebuilding the entire domain.srht.site contents - and there should be a way to do that, rebuilding the incrementally-added sections from canonical locations.

To crawl the site

With no auto-indexing, a srht site crawl will miss any files that aren't mentioned in links or .subtree-contents files. Notice that any link naming a directory is actually a link to a file named index.* in that directory (where the extension is html or gmi depending on protocol).

Here's a generic approach, starting at path /; an empty items list, and an empty working list; and a pointer to the beginning of the working list:

Retrieve file ./.subtree-contents if it exists and append its contents to the working list.
Retrieve file ./index.html if it exists and append it to the working list.
While the pointer is not at end of the working list, examine the corresponding item:
    If the item's absolute path, considering recursion, is not in the items list already:
        If the item is a file (entry does not end with a slash):
            Add the absolute path to the items list.
            For each site-local link in the file:
                If the link as an absolute path is not already in the items list:
                    Append the absolute path to both lists.
        If the item is a directory (entry ends with a slash):
            Recursively apply the algorithm to the directory.
    Increment the working list pointer.

Because Sourcehut keeps Gemini and HTTP documents in separate trees, this crawl must be done separately for each (using index.gmi where appropriate). Links in .org files to anything except another .org file are rendered as explicit HTTPS external links, so the Gemini tree only contains .gmi files with external links to the non-Org-sourced content.

Author: Steve Work

Created: 2026-11-17 Tue 21:49

Validate