Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions docs/concepts/changelog_templates.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1035,6 +1035,101 @@ the remote VCS for each commit. Both of these are injected into the template env
by PSR.


.. _changelog-templates-remote:

Remote Template Directories
---------------------------

*Introduced in v10.6.0*

As of v10.6.0, PSR supports loading changelog templates from remote filesystems such as
GitHub repositories, S3 buckets, HTTP servers, and other locations supported by `fsspec`_.
This enables you to centralize your changelog templates in a shared location and reuse
them across multiple projects.

.. _fsspec: https://filesystem-spec.readthedocs.io/

To use a remote template directory, set the :ref:`template_dir <config-changelog-template_dir>`
setting to a URL supported by fsspec.

Most remote protocols require additional packages. Install the appropriate
fsspec extra for your protocol (e.g., ``fsspec[s3]`` for S3).
Refer to the `fsspec extras documentation`_ for the full list of available protocols.

.. _fsspec extras documentation: https://filesystem-spec.readthedocs.io/en/latest/#installation


GitHub Repository (Public)
^^^^^^^^^^^^^^^^^^^^^^^^^^

For a public GitHub repository, use the ``github://`` protocol:

.. code-block:: toml

[tool.semantic_release.changelog]
template_dir = "github://myorg:shared-templates@main/changelog-templates"

The URL format is ``github://org:repo@ref/path`` where:

* ``org`` is the GitHub organization or username
* ``repo`` is the repository name
* ``ref`` is the branch name, tag, or commit SHA
* ``path`` is the path to the template directory within the repository


GitHub Repository (Private)
^^^^^^^^^^^^^^^^^^^^^^^^^^^

For private repositories, you need to provide authentication via the
:ref:`storage_options <config-changelog-storage_options>` setting:

.. code-block:: toml

[tool.semantic_release.changelog]
template_dir = "github://myorg:private-templates@main/changelog-templates"

[tool.semantic_release.changelog.storage_options]
username = { env = "GITHUB_TOKEN" }
token = { env = "GITHUB_TOKEN" }

Refer to the `fsspec GithubFileSystem documentation`_ for details on supported
authentication methods.

.. _fsspec GithubFileSystem documentation: https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.implementations.github.GithubFileSystem


Other Supported Protocols
^^^^^^^^^^^^^^^^^^^^^^^^^

Through fsspec, PSR supports many other remote filesystems including:

* **HTTP/HTTPS**: ``https://example.com/templates``
* **S3**: ``s3://bucket-name/templates``
* **Google Cloud Storage**: ``gcs://bucket-name/templates``
* **Azure Blob Storage**: ``az://container-name/templates``

Refer to the `fsspec documentation`_ for the full list of supported filesystems and
their respective configuration options.

.. _fsspec documentation: https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations


Security Considerations
^^^^^^^^^^^^^^^^^^^^^^^

.. warning::
Chained protocols (e.g., ``simplecache::s3://bucket/path``) are not supported for
security reasons.

When using remote templates, be aware that:

* Templates are fetched from the remote location each time the changelog is generated
* Local path traversal protections do not apply to remote URLs (remote templates are
by definition external to your repository)
* Ensure your remote template source is trusted, as templates have access to your
project's release history and are rendered using Jinja2


.. _changelog-templates-custom_release_notes:

Custom Release Notes
Expand Down
46 changes: 46 additions & 0 deletions docs/configuration/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -727,6 +727,8 @@ reStructuredText ``..\n version list``
``template_dir``
****************

*Remote filesystem support introduced in v10.6.0*

**Type:** ``str``

When files exist within the specified directory, they will be used as templates for
Expand All @@ -738,12 +740,56 @@ No default changelog template or release notes template will be used when this d
exists and the directory is not empty. If the directory is empty, the default changelog
template will be used.

As of v10.6.0, ``template_dir`` supports remote filesystem URLs in addition to local
paths. This allows you to store your changelog templates in a central location such
as a GitHub repository, S3 bucket, or any other filesystem supported by `fsspec`_.
See :ref:`changelog-templates-remote` for usage examples.

.. _fsspec: https://filesystem-spec.readthedocs.io/

.. warning::
Chained protocols (e.g., ``simplecache::s3://...``) are not supported for security
reasons.

This option is discussed in more detail at :ref:`changelog-templates`

**Default:** ``"templates"``

----

.. _config-changelog-storage_options:

``storage_options``
*******************

*Introduced in v10.6.0*

**Type:** ``dict[str, str | EnvConfigVar]``

Authentication and configuration options for remote template directories. This setting
is only relevant when using a remote URL for :ref:`template_dir <config-changelog-template_dir>`.

Values can be plain strings or environment variable references using the ``{ env = "VAR_NAME" }``
syntax.

This setting is passed to the underlying `fsspec`_ filesystem. Refer to the fsspec documentation
for available options per protocol.

**Example:** GitHub private repository authentication

.. code-block:: toml

[tool.semantic_release.changelog]
template_dir = "github://myorg:private-repo@main/templates"

[tool.semantic_release.changelog.storage_options]
username = { env = "GITHUB_TOKEN" }
token = { env = "GITHUB_TOKEN" }

**Default:** ``{}`` (empty)

----

.. _config-commit_author:

``commit_author``
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ dependencies = [
"rich ~= 14.0",
"shellingham ~= 1.5",
"Deprecated ~= 1.2", # Backport of deprecated decorator for python 3.8
"universal-pathlib ~= 0.2.0",
]

[project.scripts]
Expand Down
118 changes: 79 additions & 39 deletions src/semantic_release/changelog/template.py
Original file line number Diff line number Diff line change
@@ -1,25 +1,59 @@
from __future__ import annotations

import os
import shutil
from pathlib import Path, PurePosixPath
from typing import TYPE_CHECKING

from jinja2 import FileSystemLoader
from jinja2 import BaseLoader, TemplateNotFound
from jinja2.sandbox import SandboxedEnvironment
from upath import UPath

from semantic_release.globals import logger
from semantic_release.helpers import dynamic_import

if TYPE_CHECKING: # pragma: no cover
import os
from typing import Callable, Iterable, Literal

from jinja2 import Environment


class UPathLoader(BaseLoader):
"""
Jinja2 loader using UPath for universal filesystem abstraction.

This loader enables loading templates from any filesystem supported by fsspec/UPath,
including local files, git repositories, HTTP URLs, S3 buckets, etc.
"""

def __init__(
self,
searchpath: UPath,
encoding: str = "utf-8",
) -> None:
self.searchpath = searchpath
self.encoding = encoding

def get_source(
self, _environment: Environment, template: str
) -> tuple[str, str, Callable[[], bool]]:
path = self.searchpath / template
if not path.exists():
raise TemplateNotFound(template)
source = path.read_text(encoding=self.encoding)
return source, str(path), lambda: True

def list_templates(self) -> list[str]:
templates: list[str] = []
for f in self.searchpath.rglob("*"):
if f.is_file():
rel_path = PurePosixPath(f.path).relative_to(self.searchpath.path)
templates.append(str(rel_path))
return templates


# pylint: disable=too-many-arguments,too-many-locals
def environment(
template_dir: Path | str = ".",
template_dir: Path | UPath | str = ".",
block_start_string: str = "{%",
block_end_string: str = "%}",
variable_start_string: str = "{{",
Expand All @@ -36,10 +70,10 @@ def environment(
autoescape: bool | str = True,
) -> SandboxedEnvironment:
"""
Create a jinja2.sandbox.SandboxedEnvironment with certain parameter resrictions.
Create a jinja2.sandbox.SandboxedEnvironment with certain parameter restrictions.

For example the Loader is fixed to FileSystemLoader, although the searchpath
is configurable.
Uses UPathLoader which supports both local and remote template directories
(git repositories, HTTP URLs, S3 buckets, etc.) via fsspec/UPath.

``autoescape`` can be a string in which case it should follow the convention
``module:attr``, in this instance it will be dynamically imported.
Expand All @@ -52,6 +86,11 @@ def environment(
else:
autoescape_value = autoescape

if not isinstance(template_dir, UPath):
template_dir = UPath(template_dir)

loader = UPathLoader(template_dir, encoding="utf-8")

return ComplexDirectorySandboxedEnvironment(
block_start_string=block_start_string,
block_end_string=block_end_string,
Expand All @@ -67,7 +106,7 @@ def environment(
keep_trailing_newline=keep_trailing_newline,
extensions=extensions,
autoescape=autoescape_value,
loader=FileSystemLoader(template_dir, encoding="utf-8"),
loader=loader,
)


Expand All @@ -89,50 +128,51 @@ def join_path(self, template: str, parent: str) -> str:


def recursive_render(
template_dir: Path,
template_dir: Path | UPath | str,
environment: Environment,
_root_dir: str | os.PathLike[str] = ".",
) -> list[str]:
rendered_paths: list[str] = []
for root, file in (
(Path(root), file)
for root, _, files in os.walk(template_dir)
for file in files
if not any(
elem.startswith(".") for elem in Path(root).relative_to(template_dir).parts
)
and not file.startswith(".")
):
output_path = (_root_dir / root.relative_to(template_dir)).resolve()
logger.info("Rendering templates from %s to %s", root, output_path)
root_dir = Path(_root_dir)

if not isinstance(template_dir, UPath):
template_dir = UPath(template_dir)

for src_file in template_dir.rglob("*"):
if not src_file.is_file():
continue

# Convert to PurePosixPath for local path operations.
# PurePosixPath is correct because remote filesystems always use forward slashes
rel_path = PurePosixPath(src_file.path).relative_to(template_dir.path)

if any(part.startswith(".") for part in rel_path.parts):
continue

output_path = (root_dir / rel_path.parent).resolve()
logger.info("Rendering templates from %s to %s", src_file.parent, output_path)
output_path.mkdir(parents=True, exist_ok=True)
if file.endswith(".j2"):
# We know the file ends with .j2 by the filter in the for-loop
output_filename = file[:-3]
# Strip off the template directory from the front of the root path -
# that's the output location relative to the repo root
src_file_path = str((root / file).relative_to(template_dir))
output_file_path = str((output_path / output_filename).resolve())

if rel_path.suffix == ".j2":
src_file_rel = str(rel_path)
output_file_path = output_path / rel_path.stem

# Although, file stream rendering is possible and preferred in most
# situations, here it is not desired as you cannot read the previous
# contents of a file during the rendering of the template. This mechanism
# is used for inserting into a current changelog. When using stream rendering
# of the same file, it always came back empty
logger.debug("rendering %s to %s", src_file_path, output_file_path)
rendered_file = environment.get_template(src_file_path).render().rstrip()
with open(output_file_path, "w", encoding="utf-8") as output_file:
output_file.write(f"{rendered_file}\n")
logger.debug("rendering %s to %s", src_file_rel, output_file_path)
rendered_file = environment.get_template(src_file_rel).render().rstrip()
output_file_path.write_text(f"{rendered_file}\n", encoding="utf-8")

rendered_paths.append(output_file_path)
rendered_paths.append(str(output_file_path))

else:
src_file = str((root / file).resolve())
target_file = str((output_path / file).resolve())
logger.debug(
"source file %s is not a template, copying to %s", src_file, target_file
)
shutil.copyfile(src_file, target_file)
rendered_paths.append(target_file)
# Copy non-template file
target_file = output_path / rel_path.name
logger.debug("copying %s to %s", src_file, target_file)
target_file.write_bytes(src_file.read_bytes())
rendered_paths.append(str(target_file))

return rendered_paths
Loading
Loading