SourceProvider Abstraction for the Module Loader

Summary

Replace direct fs::read_to_string calls in module_loader with a SourceProvider trait so that the module loader can serve source from multiple backends: the filesystem (current CLI behaviour), embedded compiled-in data (stdlib), and in-memory overlays (LSP unsaved buffers).

Motivation

The module loader currently calls fs::read_to_string at every import site. That assumption is violated by two planned workstreams in Sprint 22:

Embedded stdlib (METEL-181). Stdlib modules will be compiled into the binary as &'static str data via include_str! / build.rs. They have no on-disk path at run-time; the loader must be able to serve them directly from a static map keyed by canonical module path.

LSP in-memory overlays (LSP bootstrapping report §4). The language server holds unsaved buffer contents for open documents. When the user edits foo.mln without saving, the LSP must inject the buffer content rather than reading the stale on-disk version. Without this abstraction the LSP is forced to write temporary files or patch the loader in an ad-hoc way.

These two needs are structurally identical: both want to intercept the read step and supply an alternative source string for a given path.

Design

Trait definition

As-built note (METEL-183). The shipped trait also receives the resolved filesystem path: fn read(&self, module_path: &[String], file_path: &Path) -> Result<String, MetelError>. The loader discovers files by multi-candidate probing (parser.mtl vs parser/ast.mtl, longest matching prefix wins), which a pure segment→path join cannot reproduce — so it passes the already-resolved file_path alongside the logical module_path. FsSourceProvider reads file_path; EmbeddedStdlibProvider keys on module_path and falls through to file_path. The "logical path only" design below is the original proposal; the extra parameter is the only deviation.

pub trait SourceProvider {
    fn read(&self, module_path: &[String]) -> Result<String, MetelError>;
}

A single method: given a logical module path (the same Vec<String> segment representation the loader already uses internally — e.g. ["std", "core"] or ["my_app", "utils"]), return the source string or a MetelError.

The key is the logical path, not a filesystem path. This eliminates the need for a fake URL scheme for embedded stdlib paths and keeps the abstraction at the right level: above path materialisation. FsSourceProvider performs the same segment-to-filesystem conversion the loader currently does inline; EmbeddedStdlibProvider matches against a HashMap<Vec<String>, &'static str> keyed on the same segments. No std:// or other synthetic scheme is needed.

Default implementation

pub struct FsSourceProvider {
    root: PathBuf,
}

impl SourceProvider for FsSourceProvider {
    fn read(&self, module_path: &[String]) -> Result<String, MetelError> {
        let mut path = self.root.clone();
        for segment in module_path {
            path.push(segment);
        }
        path.set_extension("mln");
        fs::read_to_string(&path).map_err(|e| /* existing error mapping */)
    }
}

FsSourceProvider uses the same segment-to-filesystem conversion the loader currently performs inline. It is the default for all existing call sites (pipeline::run_file, load_program, etc.). No behaviour changes for the CLI.

Embedded stdlib implementation

pub struct EmbeddedStdlibProvider {
    inner: FsSourceProvider,
}

impl SourceProvider for EmbeddedStdlibProvider {
    fn read(&self, module_path: &[String]) -> Result<String, MetelError> {
        if let Some(src) = stdlib::lookup(module_path) {
            return Ok(src.to_owned());
        }
        self.inner.read(module_path)
    }
}

stdlib::lookup is generated by build.rs from the stdlib/ directory; it holds a HashMap<&'static [&'static str], &'static str> (or equivalent phf::Map) keyed on path segment slices. When module_path matches a known stdlib key the compiled-in source is returned; otherwise the call falls through to the filesystem. This is the provider used by the default CLI pipeline once stdlib embedding lands.

LSP overlay implementation

pub struct OverlaySourceProvider<P: SourceProvider> {
    open_documents: Arc<DashMap<Vec<String>, String>>,
    inner: P,
}

impl<P: SourceProvider> SourceProvider for OverlaySourceProvider<P> {
    fn read(&self, module_path: &[String]) -> Result<String, MetelError> {
        if let Some(src) = self.open_documents.get(module_path) {
            return Ok(src.clone());
        }
        self.inner.read(module_path)
    }
}

Defined in metel-lsp, not in metel-core. The LSP document store keys open buffers by logical module path (derived from the document URI) rather than by filesystem path. The LSP wraps EmbeddedStdlibProvider (or FsSourceProvider in tests) with its document map. No changes to metel-lsp's re-analysis loop other than passing the overlay provider to load_root.

Changes to load_root

pub fn load_root<P: SourceProvider>(
    path: impl AsRef<Path>,
    provider: &P,
) -> Result<ModuleGraph, MetelError>

load_root is generic over P rather than taking &dyn SourceProvider. Call sites always have a concrete type known at compile time (EmbeddedStdlibProvider for the CLI, a concrete OverlaySourceProvider<…> for the LSP), so dynamic dispatch provides no benefit and adds a vtable cost.

All internal fs::read_to_string calls in module_loader.rs are replaced with provider.read(module_path) where module_path is the Vec<String> the loader has already computed. The pipeline::run_file call site passes &EmbeddedStdlibProvider::new(root) (or &FsSourceProvider::new(root) in tests that do not need embedded stdlib).

load_program (the single-file parse helper used by the test harness) is updated to accept the same generic provider parameter.

std:: path handling

The module loader currently returns Ok(None) for std:: imports and relies on StdPrelude for them. Once stdlib modules are embedded files, EmbeddedStdlibProvider will recognise paths beginning with ["std", …] and return their compiled-in source — no special URL scheme required, since the key is already a Vec<String>. The Ok(None) bypass is removed as part of METEL-181. This RFC only introduces the abstraction; the bypass removal is METEL-181's concern.

Protected std namespace

std is a reserved keyword in Metel — it cannot appear as an identifier, so import std::foo always produces PathRoot::Std and routes to the stdlib. However, a root-relative import such as import ::std::foo uses PathRoot::Root and goes through find_module_file, which would discover a user-created std.mln at the project root. Once EmbeddedStdlibProvider is in place, the provider's embedded-map check runs before the filesystem fallthrough — meaning a stdlib path silently wins over the user's file, or the user's file silently wins over an absent stdlib path, depending on the specific name.

To eliminate this ambiguity, the loader must treat std as a protected namespace at the file-discovery layer. Add a validation analogous to the existing validate_super_root check:

fn validate_std_namespace(module_path: &[String], file_path: &Path) -> Result<(), MetelError> {
    if module_path.first().map(|s| s == "std").unwrap_or(false) {
        return Err(module_error(
            "module path `std::…` is reserved for the standard library",
            file_path,
        ));
    }
    Ok(())
}

This is called for every user-supplied module path before loading proceeds. Stdlib modules served by EmbeddedStdlibProvider are never assigned to loader.modules and never pass through this check; only modules resolved from the filesystem do. The result is a clear error rather than silent shadowing in either direction.

Alternatives Considered

A — Thread-local source registry

A thread_local! static map of path → source replaces the trait. Simpler call sites (no parameter threading) but global mutable state, hard to test in parallel, and wrong for LSP where multiple analysis tasks for different roots may coexist.

Rejected. The explicit trait is testable and composable.

B — Callback closure instead of trait

fn load_root(path, read_fn: impl Fn(&Path) -> Result<String, MetelError>)

Equivalent in expressiveness. A named trait communicates intent better and allows OverlaySourceProvider to be a first-class, documented type in metel-lsp.

Rejected in favour of trait.

C — Separate loader entry point for LSP only

Add load_root_with_overlay(path, overlay: &HashMap) as a second entry point. Avoids touching existing callers.

Rejected. It duplicates the loader logic and prevents the embedded-stdlib use case from sharing the same clean abstraction.

Implementation Notes

• Touching load_root's signature is a single-file change in module_loader.rs plus call-site updates in pipeline.rs and metel-bench.rs.
• FsSourceProvider and the updated load_root signature are the only additions to metel-core. The overlay and embedded variants follow in their respective workstreams.
• This RFC does not change parsing, name resolution, or any downstream stage. It is a read-layer abstraction only.
• The std:: path bypass (Ok(None)) is intentionally left in place until METEL-181 is ready to replace it. The two changes are independent and can land in different commits.
• Add validate_std_namespace alongside the existing validate_super_root call in load_module. This is the only place needed — it guards every filesystem-resolved module path before it enters loader.modules.

References

• LSP bootstrapping report: metel-lsp/docs/reports/lsp-bootstrapping-analysis.md §4
• METEL-181: Unify builtin and std::core modeling with the normal module pipeline
• METEL-182: Design stdlib-only native declarations for host-backed implementations
• RFC-0057: Standard Library Layering and Host Module Boundary
• module_loader.rs load_root and load_program entry points
• pipeline.rs run_file entry point