manifest

manifest.yml is the file at a repository’s root that says what the repository holds: what it is called, what version it is at, which modules it contains, where each is rooted, and what each depends on. dev reads it for everything it does by module – building, vendoring, selecting the tests a change reaches, and checking that what the code imports is what the manifest declares – so a manifest drawn at the right grain is where most of the tools’ value comes from. This package, manifest:manifest, owns the document’s schema, manifest:manifest, and reads a manifest into typed, normalized, validated models, so every program reads the same declaration the same way.

The library validates the document’s structure. What the declared dependency graph means – cycles, layering, which module may reach which – is decided by the program that reads the graph, and dev’s checks are the ones described below.

Writing one

This takes a repository with no manifest to one whose modules and dependencies dev accepts. It assumes the repository’s adapter is chosen and its adapter catalog wired, which repo-adapter-api’s “Setting one up” does; its step 5 writes a first manifest, and this walkthrough is that step done in full. Every command is run from the repository’s root. Where a command refuses, its message names a code and the field or edge at fault; repair what it names and run the step again.

  1. Start the file. Create manifest.yml with the repository’s own facts:

    schemaVersion: '0.0.0-dev'
    project: acme
    version: 0.1.0
    repoAdapter: language:python
    catalogs:
      repo-adapter:
        path: repo-adapter
        pkg: repo-adapter
    modules: []
    • schemaVersion is the version of the manifest’s schema these tools read, manifest.SCHEMA_VERSION, written whole and quoted. A file stating another is refused as it loads, as MANIFEST_VALIDATION_FAILED, naming the version to write.
    • project is the repository’s name. Write it in lowercase letters, digits and hyphens: it is also the namespace every module is published under, the first part of each module’s coordinate <namespace>:<name>. State namespace beside it only where the packages are to be published under another name than the repository’s.
    • version is major.minor.patch, the version the repository is at, and every module’s. Take it from the repository’s packaging metadata, respelled where that spells it otherwise (2.3.0.dev is written 2.3.0-dev). Nothing reads one from the other, so a release from here on changes the version in both files.
    • repoAdapter is the adapter’s coordinate, and catalogs the catalog it resolves against, both as the adapter’s setup gave them.

    Done when dev module list answers Project: acme and Targets (0). A field the schema does not hold, or one missing or misspelt, is refused as MANIFEST_VALIDATION_FAILED, naming the field: (project): Field required, (projct): Extra inputs are not permitted, (version): Input should be a valid dictionary or instance of Version for a version that is not three numbers.

  2. List the candidates. Write down every package the repository holds that something imports or runs: each directory the language treats as an importable package at the top of the source tree, each application with an entry point, and each library of test helpers that tests import. Leave out tests, generated output and scripts nothing imports. An adapter document the repository keeps for itself is already a module, declared by the adapter’s setup with its runtimeCoordinate; keep that entry as it is. Done when you have the list and, for each, the directory it is rooted at and the file a consumer enters it by – __init__.py for a Python package, the file that starts it for an application.

  3. Declare one module per candidate. Each is an entry under modules:

    modules:
    - name: acme-cli
      rootDir: src/acme_cli
      entryPoint: __main__.py
      repoAdapter: language:python
      kind: executable
    - name: acme-core
      rootDir: src/acme_core
      entryPoint: __init__.py
      repoAdapter: language:python
      kind: library
    • name is the module’s name, the second part of its coordinate: lowercase and hyphenated, whatever the directory is called. Prefix each with the project’s name (acme-core, not core): a name has to be unique across every manifest whose modules are ever read together, and the prefix is what keeps it so.
    • rootDir is the directory, relative to the repository’s root, spelled as it is on disk. entryPoint is relative to rootDir.
    • repoAdapter is the top-level one again, or another where the module is in another language.
    • kind is library for a package that is imported, executable for an application that is run, and test-library for helpers only tests import.
    • Keep modules sorted by name; the file is read back sorted either way.

    Done when dev module list lists every module under Targets (<count>), one line each, as - acme-core: src/acme_core/__init__.py (language:python, library). A name declared twice is refused as it loads, as MANIFEST_VALIDATION_FAILED with duplicate 'name' value 'acme-core' in tree.

  4. Choose the granularity. A module is a unit something answers for: it has dependencies of its own, a place in the graph, and tests a change to it selects. Draw a module boundary wherever one of these holds, and nowhere else:

    • The parts have different dependencies. A package whose half imports a database driver and half does not is two modules, so the half that does not is not rebuilt and retested when the driver moves.
    • The parts are consumed or released separately. Something imports one part without the other: a client library beside the service that implements it, a data model several applications share.
    • The parts have different audiences. One part is released to clients and the other kept in-house (step 7): audience is per module, so a boundary is needed for the two to differ.
    • One part is an application and another is what other code imports of it. An application is a leaf: nothing may depend on an executable module. Code another module imports goes in a library of its own, and where that library is the application’s generated API, it states apiOf naming the application.

    Do not split where none of these holds:

    • No consumer would take the part alone. Two directories that always change together and are always imported together are one unit with two folders.
    • The split only mirrors folders. A directory is not a reason. Declaring each one, or wrapping each loose file in a directory to declare it, produces a graph as large as the tree that says nothing the tree did not.
    • The code cannot have the boundary. Two files that import each other are two halves of one thing; a declared graph refuses the cycle between them, so they stay one module.

    Decide each candidate by these tests: keep it whole, merge it with another, or mark it to be split in step 6. A first manifest of one module per package, as step 3 wrote, is the usual answer, and a coarse manifest that passes the checks is a better start than a fine one that does not; a finer division can always be made later. Done when every candidate is decided; step 5 ends by checking the decision against the graph the code actually has.

  5. Declare the dependencies, and let detection say what was missed. Each module lists what it depends on, in four lists:

    • dependencies – what it imports at run time.
    • devDependencies – what only static analysis needs: an import made for type annotations alone, in Python one under if TYPE_CHECKING:.
    • testDependencies – what its tests import that the module itself does not.
    • buildDependencies – what has to be built or generated before it, such as a schema its types are generated from.

    An entry is the name of a module this manifest declares, or the coordinate <namespace>:<name> of a package another repository declares. Third-party packages from the language’s own index are not listed: they are the packaging metadata’s.

    Write what you know, then run dev module dependency detect --validate. It reads what each module’s source actually imports and compares it with what the manifest declares:

    [FAIL] Detected dependencies differ from declared dependencies
    Modules analyzed: 2/2
    Detected dependencies:
      - acme-cli -> acme-core
    Undeclared detected dependencies:
      - acme-cli -> acme-core (runtime)

    Each line under Undeclared detected dependencies: is one edge to add, in the list its bracket names – (runtime) is dependencies, (dev) is devDependencies and (test) is testDependencies, so here acme-core goes under acme-cli’s dependencies – and each under Unused declared dependencies: is one to remove. Imports in test files are read as (test) edges, and imports of third-party packages are not reported. The run fails as DETECTED_DEPENDENCY_VALIDATION_FAILED until the two agree. The manifest is what is corrected, never the code: an edge you did not expect is the code’s real shape, and a boundary it crosses is a granularity question for step 4. Done when it prints [PASS] Detected dependencies match declared dependencies.

    Then run dev module dependency list --validate, which judges the declared graph itself, refusing as DEPENDENCY_VALIDATION_FAILED and listing each: a name nothing declares (Unknown dependencies:), a cycle (Dependency cycles found:), and a dependency on an application or on a test library from non-test code. Done when it prints [PASS] Declared dependencies valid.

    Last, look at the result. dev module graph lists the edges between modules, one per line, as acme-cli -> acme-core [runtime] (declared+detected). Draw them, and ask whether the drawing is how the people who work on the repository would draw the system. A package that is really three subsystems becomes three modules (step 6); three packages nobody thinks of apart become one, by step 4’s tests. Go round steps 4 and 5 again until the drawing matches and both validators pass.

  6. Nest the modules a package is made of, where step 4 split one. A module inside another is declared under its parent’s submodules, with a rootDir inside the parent’s:

    - name: acme-core
      rootDir: src/acme_core
      entryPoint: __init__.py
      repoAdapter: language:python
      kind: library
      submodules:
      - name: acme-core-pricing
        rootDir: src/acme_core/pricing.py
        entryPoint: pricing.py
        repoAdapter: language:python
        kind: library

    A single file can be a module: rootDir names the file and entryPoint is that file’s name. A nested module travels inside its parent’s package and is never obtained alone. The parent depends on what it holds without saying so, so its lists never name its own submodules; a nested module’s own lists name what it imports, siblings included. It reaches another top-level module of the same manifest only through its parent, which declares that dependency and hands it down – in Python, through the parent’s submodule_api.py. A package of another repository it declares and imports itself.

    dev module dependency list --validate refuses a module rooted inside another but declared beside it (Nested modules must be declared as submodules instead of independent modules:), a parent listing its own submodule (Explicit parent-to-submodule dependency declarations are not allowed:), a nested module’s rootDir outside its parent’s, and a nested module depending directly on another top-level module (Submodules must not directly depend on modules outside their top-level module:). A nested name used twice anywhere in the manifest is refused as it loads, and as DUPLICATE_MODULE_NAMES across manifests read together; prefixing each nested name with its parent’s (acme-core-pricing) keeps it unique.

    A nested module declared beside its parent instead fails both validators: dev module dependency list --validate as above, and dev module dependency detect --validate with the parent’s import of it undeclared (acme-core -> acme-core-pricing (runtime)), since the parent’s implicit edge holds only for its own submodules. Declared as a submodule, the edge is read as the parent’s own: dev module graph shows it as acme-core -> acme-core-pricing [runtime] (detected), detected and never declared, which is right.

    Then run dev module export list --validate. It reads each parent’s export surface, the file the adapter names for one – in Python external_api.py at the module’s root, its exports listed in __all__ – and a parent keeping none is not asked anything. A name the surface imports from a nested module and re-exports straight through is a second door onto that module, and is refused as PARENT_EXPORT_VALIDATION_FAILED, each as acme-core/external_api.py:1 [price] imported from acme-core-pricing and re-exported. Either the nested module is the consumer’s to import, and the parent stops re-exporting it, or the parent’s surface is the consumer’s door and the boundary is in the wrong place: the part is no unit of its own and is left undeclared. Done when steps 5 and this one pass.

  7. Say who each module is released to. audience on a top-level module is internal where it is not stated: reached only by the publisher’s own repositories, which is right for every module until someone outside is meant to obtain it. public releases it to clients and documents it for them; hidden releases it the same way and documents it nowhere, for a library a public one needs that nobody should be pointed at. What a released library depends on at run and build time is released with it, so dev module dependency list --validate refuses a public or hidden library depending on an internal module (A library released to a client, public or hidden, must not depend on an internal module:), and an application declared hidden (An application is public or internal, never hidden:). An application may be public whatever its libraries are, since what it is built from is inside it. A public module’s README.md, at its rootDir, is its client documentation, so a module declared public owes one: it travels in the module’s archive and is published as written. Done when every module that leaves the repository says so, and both validators pass.

  8. Check the whole file. Each of these reads the manifest whole and exits non-zero on what it refuses; the four that validate are checks of the gate, which a plain dev gate leaves to CI and dev gate --all runs:

    dev module list
    dev module dependency list --validate
    dev module dependency detect --validate
    dev module export list --validate
    dev project list --validate

    dev project list --validate reads the project’s line, acme 0.1.0 (acme@0.1.0) untagged before any release, and holds the version to the repository’s release tags, <project>@<version>: once a release is tagged, content that moves after it with the version unchanged is refused as PROJECT_UNMARKED until the version is marked -dev, and before any release a dependency pinned at a version is refused as DEVELOPMENT_REFERENCE_PINNED, since a development reference floats – drop the @<version> and let a release write one. Done when all five pass. Commit the manifest; from here on a new package is declared in the change that adds it.

Reference

The manifest

A module

The fields’ own help in manifest:manifest is the full statement of each, and a tool reading the document shows that help. Strings are trimmed, paths normalized, repoAdapter and runtimeCoordinate lower-cased and held to the coordinate pattern, and module lists sorted, by the rules the schema declares in type’s rule vocabulary.

Reading a manifest from code

The library is named by the coordinate manifest:manifest and imported as manifest. It depends on type:type, which arrives with it. Everything is imported from the package root, which is the whole surface:

The models are loaded with type’s loaders, which run the normalization and validation the schema declares:

from pathlib import Path

from manifest import MANIFEST_DEFAULT, Manifest, module_coordinate
from type import load_data_file

loaded = load_data_file(schema_type=Manifest, path=Path(MANIFEST_DEFAULT))
for module in loaded.modules:
    print(module_coordinate(manifest=loaded, module=module, version="1.0.0"))