Developer documentation

This page is for whoever has to change DDD rather than use it: to add an output format, to add a check, or to understand why the code is arranged the way it is before moving something in it.

DDD is a front end and a set of backends with one contract between them. The front end reads the description files, resolves the project and reports every disagreement it finds; each backend turns the resolved result into files of one output format. The front end never mentions c or a2l, and a backend never touches the loader or the checks. That is not a style preference: a code generator whose layers leak becomes a generator in which nobody can change the c output without wondering what it does to the a2l, and in which a rule about ownership ends up being re-implemented, slightly differently, in a jinja template.

Layers

layer	knows about	does not know about
src/ddd/models/	the json file formats, storage sizes, value ranges	c, a2l
src/ddd/loading.py	files, includes, globs, encodings	what the data means
src/ddd/analysis.py	ownership, agreement between components, references	any output format
src/ddd/ir.py	the contract: the resolved data dictionary	how it is rendered
src/ddd/backends/c/	uint16_t, literals, include guards, rendering the project’s templates	a2l, the loader, the checks, what the generated files are called
src/ddd/backends/a2l/	UWORD, compu methods, record layouts, its own templates	c, the loader, the checks

The last two rows are deliberately not symmetric. The a2l backend carries its own templates, because ASAP2 is a format ASAM defines and a project has nothing to decide about it; the c backend carries none, because what generated c looks like is a house style. It is constructed with the template directory --template-dir names, works out what to render from the file names it finds there, and therefore does not know before a run which files that run produces. Those naming rules are part of the interface a project depends on: Templates documents them, and the module docstring of src/ddd/backends/c/backend.py states them again next to the code that implements them.

Three smaller modules sit beside them. diagnostics.py holds the severity policy and the registry of every check, and is what both the loader and the analysis report through. compare.py answers the directional question of whether one dictionary may replace another, and is the second consumer of the contract next to the backends. cli.py is the only module that knows about argument parsing, exit codes and where output goes; it is also where the backends a ddd generate run uses are assembled.

The two contract pages describe the data that travels between the layers: the input file formats under data contracts, and the resolved form under data dictionary.

The split is enforced by a test

A layering that lives only in the documentation rots the first time somebody is in a hurry, so DDD asserts it. tests/test_backends.py parses every module under src/ddd with ast, collects the ddd.* modules each one imports, and fails if the import graph disagrees with the table above:

• loading.py, analysis.py, ir.py and diagnostics.py import no backend,

• nothing under backends/ imports ddd.loading or ddd.analysis,

• the c backend does not import the a2l backend, and the a2l backend does not import the c one,

• both backends satisfy the ddd.backends.base.Backend protocol, and every datatype is spelled by both of them, so a new datatype cannot be added to the contract while one output format silently has no name for it.

A second test in the same file reads the text of src/ddd/models/ and fails if a spelling that belongs to a single output format - uint16_t, UWORD, COMPU_, AXIS_PTS, stdint - has leaked into the input contract. The one documented exception is src/ddd/models/reserved.py, which lists the c keywords and the names <stdint.h> claims: which identifiers a c compiler takes for itself is a property of the input format, because generating c is not optional in DDD. It is a file of its own precisely so that the guard can stay strict about everything else.

Note

The word measurement is deliberately not on that list. It is DDD’s own term for an online value and part of the input file format; that the a2l keyword MEASUREMENT happens to coincide with it is not a leak.

What a backend is

A backend is anything with a name and a generate(dictionary, output_dir) method, as described by the protocol in src/ddd/backends/base.py:

@runtime_checkable
class Backend(Protocol):
    """Turns a data dictionary into files."""

    name: str

    def generate(self, dictionary: DataDictionary, output_dir: Path) -> list[GeneratedFile]:
        """Render every artefact of this backend; nothing is written to disk."""
        ...

That signature carries two decisions worth knowing. First, generate receives the resolved ddd.ir.DataDictionary and nothing else: no workspace, no diagnostic bag, no command line arguments. Anything it needs about the project has to be a field of the contract, which is what stops one output format from acquiring its own private view of what a project means. Second, a backend renders but does not write. It returns fully rendered GeneratedFile objects, and the driver decides what happens to them, which is what makes three things possible in one place rather than in every backend:

• two artefacts claiming the same path are refused with a message naming the backends involved, instead of one silently overwriting the other,

• a file whose content has not changed is left alone, so that a regeneration does not trigger a rebuild of everything downstream,

• --dry-run is a parameter of the writing step rather than a flag every backend has to honour correctly.

What a backend needs about the run rather than about the project is settled when it is constructed, and the constructor is a backend’s own business. That is where the two differ:

class CBackend:
    def __init__(
        self, template_dir: Path, options: COptions | None = None, generator: str = "ddd"
    ) -> None: ...

class A2lBackend:
    def __init__(self, options: A2lOptions | None = None, generator: str = "ddd") -> None: ...

The template directory comes first for the c backend because it is the one argument that has no default: a run without --template-dir is a usage error of ddd generate, never a fallback to a built-in set of templates, and there is nothing for the constructor to fall back to either. The a2l backend takes no such argument at all, since its templates are part of the package.

Adding an output format

Adding one - a header for another language, a csv, an ARXML - means adding a package next to the existing two and touching nothing else:

1. Create src/ddd/backends/<format>/ with a class exposing name and generate. Follow the shape of the existing two: a model.py that turns the dictionary into whatever the templates want to iterate over, a types.py mapping every ddd.models.Datatype to the spelling of that format, and an options.py for what the command line can tune. Then decide where the templates come from, which is the one design question a new backend has to answer for itself: ship a templates/ directory inside the package if the format is specified elsewhere, as a2l is, or take a template directory as the first constructor argument if the format is a house style, as c is. Use ddd.backends.base.make_environment and render_template either way, so the jinja settings - in particular StrictUndefined, which turns a typo in a template into an error rather than into an empty string - are the same as everywhere else.

2. Export it from src/ddd/backends/__init__.py.

3. Add it to the list of backends that _command_generate in src/ddd/cli.py builds, together with the option that selects or configures it.

4. Add it to the protocol assertion in tests/test_backends.py. The import graph tests pick the new package up on their own, so the first thing the suite will tell you is whether the new backend reached into the front end.

Nothing in the front end changes, and neither of the existing backends is touched.

Diagnostics never raise

The loader and the analysis report through a ddd.diagnostics.DiagnosticBag and do not raise. An exception escaping from the middle of a run would end it with a bare python message and throw away everything already collected, and the author would fix one problem, run again, and be told about the next one. Collecting instead means a run reports as much as it can: a file that cannot be read, a file that is not utf-8, json nested deeper than python can parse, a path the operating system cannot represent - each comes back as a located finding, and the rest of the project is still checked.

Two consequences for anyone adding a check. A check is an entry in the CHECKS registry in src/ddd/diagnostics.py - an identifier, a default severity and a one line description - plus the code that adds the finding; the identifier is part of the public interface of the tool, since it is what -W and --strict address and what a ci job matches on, and ddd checks prints the registry as it stands (see consistency checks). And a check that cannot be relaxed has to say so: the five entries marked overridable=False are the ones after which nothing further can be said about the file at all, and every other check has to survive being set to ignore.

What the calibration tools actually implement

Structured data can be written into an a2l two ways, and the choice is not a matter of taste. ASAP2 has a typedef family - TYPEDEF_STRUCTURE, STRUCTURE_COMPONENT, INSTANCE - that describes a structure once and instantiates it, which is the obvious fit for a project with twenty instances of one type. The alternative is to flatten: one ordinary object per leaf, named after the path to it, at the address of the instance plus the offset of the member.

The native form was tested against CANape 15 before any of it was built, with hand written a2l files, and it is not usable there:

• ASAP2_VERSION 1 71 is refused outright - unknown ASAP2 version 1.71 - so the version DDD declares cannot simply be raised. 1 70 is accepted.

• at 1 61 and 1 70 a file containing a TYPEDEF_STRUCTURE and two INSTANCE of it loads without a warning and contains no objects at all. Loading is not evidence of support.

• the grammar does know the keyword: a SYMBOL_TYPE_LINK inside the structure is a syntax error on its own line, not on the enclosing block, so the body is being parsed and validated. It accepts the shape and exposes nothing for it.

• a GROUP referencing members through their instance loads, and is empty.

So the a2l flattens, and two further constraints come out of the same exercise. Do not emit ASAP2_VERSION 1 71, since 1 61 carries everything a flattened structure needs. And do not rely on a tool to report a bad reference: CANape silently dropped one it could not resolve, which means a mistake in a generated name costs an object with no diagnostic anywhere, and the burden of catching it sits here.

INCA has not been tested. Since it is generally the more conservative of the two, flattening is what a project targeting both can rely on.

The coverage gate

Coverage runs with every test run, over statements and branches, and a gap fails the run:

[tool.pytest.ini_options]
addopts = "-q --cov --cov-report=term-missing --cov-fail-under=100"

[tool.coverage.run]
source = ["ddd"]
branch = true

The reasoning is that a line nobody executes is a line nobody has ever seen behave - and in a code generator, an unexercised branch means an output nobody has ever looked at. The two consequences are worth stating plainly, because they change how the gate is met. The gate is what found the dead code this project used to carry, in the shape of unused properties on the analysis and contract types; the fix was deleting them, not writing tests for them. And the paths that only a coverage run reaches - unreadable files, malformed json, relaxed severities, odd float literals - are collected in tests/test_edge_cases.py rather than being scattered through the suite that describes behaviour.

Three suites guard things a type checker cannot. tests/test_backends.py walks the import graph, as described above. tests/test_hardening.py holds one test per defect that once reached a customer-facing artefact or verdict - a transposed a2l array, a header that does not compile, a legal name rejected, a description file that ended the run with a python traceback - grouped by what was at stake rather than by module. tests/test_documentation.py asserts that every check identifier, every command, every object kind and every datatype is named in README.md and in SPEC.md, that the README invents no check that is not registered, and that no link in either points at a file that no longer exists.

Running the checks

python -m pytest              # the suite, the coverage gate and the documentation checks
python -m pytest --no-cov     # quicker, while working on a single test
python -m ruff check .
python -m ruff format .
python -m mypy

mypy runs in strict mode over src/ddd with the pydantic plugin; ruff lints the sources, the tests and the documentation configuration with a line length of 100. The suite runs in a few seconds, so there is no reason to run anything less than all of it.

Nothing in the suite skips. A test that skips when a tool is absent reports success without having run, and the one place that used to do it - validating the examples against the committed schemas, which needs jsonschema - was skipping everywhere except on the machine of whoever happened to have it installed. It is a development dependency instead.

The repository also ships a small linux image, which is what the generated c code is actually compiled with - a generator whose output no compiler has ever accepted is a generator with no evidence behind it. Run it from a WSL shell, where docker speaks linux containers:

docker compose run --rm test        # pytest with the coverage gate
docker compose run --rm coverage    # same, plus build/htmlcov/index.html
docker compose run --rm lint        # ruff check, ruff format --check and mypy
docker compose run --rm compile     # generate, compile, link and verify the symbols
docker compose run --rm cmake       # build examples/cmake through cmake/Ddd.cmake
docker compose run --rm docs        # build this documentation

The compile service is the one that keeps the c backend honest. It generates the demo project, writes one translation unit per generated header that includes it twice - which proves that every header is self contained and that its include guard works - compiles everything with -std=c11 -Wall -Wextra -Wpedantic -Werror -Wconversion -Wshadow -Wcast-qual -Wstrict-prototypes, links all objects into one binary, and finally compares nm against ddd list --format json so that every variable DDD promised is defined exactly once and nothing else is. The last three steps run twice, once plain and once with the conditional declarations enabled, so both states of a #if guarded variable are covered. It renders the example templates, which is what makes them evidence rather than a sketch: the set a project starts from is the set a compiler has accepted. TEMPLATES points the service at any other directory, so a project can put its own templates through the same treatment.

Continuous integration

.github/workflows/ci.yml runs exactly the commands above - the suite with its coverage gate, ruff twice and mypy - on every push to master and every pull request.

The suite runs across a matrix of ubuntu and windows on python 3.12 and 3.13, which is the four combinations the classifiers in pyproject.toml advertise. That is not thoroughness for its own sake: a path handling defect that only appeared on linux has already reached a user of this project, having passed the whole suite on windows first. A test in tests/test_documentation.py keeps the matrix and those classifiers in agreement, so advertising a new interpreter without testing it fails.

Each job installs the project with pip install -e ".[dev]" rather than running it out of src. That is deliberate too, and it is the cheapest check in the file: it exercises the packaging metadata, which the tests themselves never touch, so a dependency list that no longer builds fails here rather than for whoever installs the distribution.

Style and types are checked once rather than per platform, since neither varies by platform. publish.yml runs the suite again before it builds a release, which is not redundant: a release can be cut from a commit this workflow never saw, and an upload to an index is permanent.

Building this documentation

pip install -e ".[docs]"
sphinx-build -b html docs build/docs/html -W --keep-going

Warnings are errors, which matters more here than in most projects: the reference sections are generated from the sources - autoprogram renders the command line from the argument parser itself, autodoc_pydantic renders the file formats from the contracts - so a renamed option or a changed field cannot leave its documentation behind, and a reference that no longer resolves fails the build instead of quietly disappearing from the page.

Two programs have to be on the path as well, and each fails visibly rather than silently dropping a figure: dot from graphviz draws the entity relationship diagram of every model on the file format pages, and plantuml draws the .. uml:: diagrams. Without a plantuml installation, docs/conf.py still names one, so the build reports a warning per diagram - which under -W is a failure. Both are apt packages, and both are in the image behind docker compose run --rm docs, which is the way to build the documentation without installing either.

Publishing this documentation

.github/workflows/docs.yml builds the html and publishes it to GitHub Pages. It runs on every pull request, and deploys on two events: a push to master, and a published release. The deployment authenticates the same way the release upload does, with a short lived OIDC token rather than a stored secret.

The site keeps one directory per version, and a menu in the bottom left corner of every page switches between them:

Directory	Written by	Rewritten
latest	a push to master	every push
v0.5.0, v0.6.0, …	publishing that release	never

Old documentation describes old code, so nothing rebuilds a released version. That is not only a saving: a tag from two years ago would have to keep building under the sphinx of the day, and the run that failed would be the one publishing the current release.

Since a deployment replaces the whole site, the versions that are not being built have to come from somewhere. The gh-pages branch is that archive - storage rather than the published thing, restored at the start of the job and pushed back at the end of it. Pages still serves the artifact the workflow uploads, so its Source setting stays on GitHub Actions. The branch is created by the first deployment; there is nothing to set up.

The menu cannot be baked into a page at build time, or a version released today would be missing from the menu of every page built before it - which is the menu somebody reading an old page is looking at. So docs/_templates/versions.html renders an empty menu and docs/_static/js/versions.js fills it in on load from versions.json at the root of the site, which the workflow rewrites from what is on disk on every deployment. A build with no such file above it - a local one, a pull request one - shows no menu at all, which is honest: there is nothing to switch to.

The root of the site redirects to the newest release rather than to latest. Somebody arriving without a version in the url wants the documentation of what they can install, not master’s account of features that are not released yet.

The workflow installs graphviz and plantuml from apt, so publishing needs nothing but a stock runner: there is no prepared image to keep in step with the sources. Only html is built. A pdf would want a LaTeX distribution, roughly a gigabyte of packages, and nothing asks for one - docs/conf.py still carries the LaTeX settings, so sphinx-build -M latexpdf docs output produces one for whoever does.

Three things are worth knowing before the first run.

Pages has to be switched on by hand, once. In Settings → Pages, set Source to GitHub Actions. The workflow’s token may deploy to a site but may not create one, so until that setting is made the deployment step fails with a message about a missing Pages site. There is nothing to change in the repository to fix it: correct the setting and re-run.

The ``github-pages`` environment has to allow tags. GitHub creates that environment with its deployments restricted to the default branch, which is the whole of what a release deployment is not: the run is triggered by a tag. It fails with not allowed to deploy to github-pages due to environment protection rules, after a green build, on the release. In Settings → Environments → github-pages → Deployment branches and tags, choose Selected branches and tags and add a branch rule for master and a tag rule for v*. This is a setting rather than a file, so it survives nothing in the repository and has to be made once per repository.

A pull request builds but never deploys. The deploy job names the branch and the release event it publishes, rather than resting on the event alone, because a pull request from a fork proposes arbitrary content: without that condition, opening one would be enough to publish somebody else’s revision as the product’s documentation.

Publishing a release

.github/workflows/publish.yml builds, checks and uploads the distribution. It never holds an API token: the upload authenticates with trusted publishing, where GitHub mints a short lived OIDC token and the index decides whether the claims in it match a publisher somebody registered.

Two things follow from that, and both have bitten this project.

The two indices are separate registrations. TestPyPI and PyPI are different services with different accounts and different publisher configurations. A publisher registered on pypi.org has no effect whatsoever on test.pypi.org, even though the pages look identical. The workflow uploads to whichever the job names, so each needs its own:

field	test.pypi.org registration	pypi.org registration
PyPI project name	ddd-tool	ddd-tool
owner	Sauci	Sauci
repository	ddd	ddd
workflow name	publish.yml	publish.yml
environment name	testpypi	pypi

The environment name is the field most easily got wrong, because it is the GitHub deployment environment of the job rather than anything about the index: the job publishing to TestPyPI declares environment: name: testpypi, so the registration has to say testpypi too.

Until the project exists on an index, its registration is a pending publisher, which is also what creates the project on first upload. Note what the page itself warns: a pending publisher does not reserve the name, so anybody may take it first. Once the project exists the registration becomes an ordinary publisher, and a project that already exists needs the publisher configured on the project rather than as a pending one.

The configuration is read at upload time, not at commit time. A run that failed with invalid-publisher will succeed on a plain re-run once the registration is corrected - there is nothing to change in the repository and no new commit to push.

A workflow_dispatch run with target: testpypi is the dry run; publishing to PyPI happens on a published GitHub release tagged v<version>, and the build refuses to go on unless that tag is exactly v followed by the version in pyproject.toml. The prefix is checked rather than stripped, because the documentation site publishes a release under a directory named after its tag and lists only the ones beginning with v.

The publishing jobs name a deployment environment - pypi, testpypi - and so is subject to the same trap as github-pages above: an environment whose Deployment branches and tags is left at the default rejects a release, because a release is triggered by a tag and the default permits only the default branch. It fails after a green build, with not allowed to deploy … due to environment protection rules, and is fixed in the settings rather than in the repository.

Publishing the editor extension

The same workflow publishes editors/vscode to the Visual Studio Marketplace and attaches the .vsix to the release. Both come from one npm run package: the marketplace step publishes that file with --packagePath rather than packaging a second time, so what the marketplace serves and what the release carries are the same bytes.

The marketplace has no equivalent of trusted publishing, so unlike the index upload this one holds a credential. Three facts about it are worth having written down before it is needed.

The publisher lives in Azure DevOps, not on GitHub. A marketplace publisher is created once, at https://marketplace.visualstudio.com/manage, against an Azure DevOps organisation. Its id has to be the publisher field of editors/vscode/package.json - sauci - and the extension is then identified everywhere as sauci.ddd.

The token is a personal access token with one scope. From that organisation, a token with Marketplace → Manage, issued against All accessible organizations rather than a single one: a token scoped to one organisation authenticates and is then refused when it publishes, which reads as a wrong password rather than a wrong scope. It is stored as the repository secret VSCE_PAT.

It expires, and the failure lands on a release. Azure DevOps allows a year at most. When it lapses, the extension job fails on a release that has already uploaded to PyPI, which cannot be taken back and does not need to be: only the marketplace step failed. Issue a new token, update the secret, and re-run the job. The step checks the secret is present before it does anything and says all of this in the error, rather than failing inside vsce.

The name is claimed at the first publish. Until then the marketplace link above is a 404, and the release still attaches its .vsix.