rsconstruct 0.3.5

Rust based fast build system
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
# Suggestions

Ideas for future improvements, organized by category.
Completed items have been moved to [suggestions-done.md](suggestions-done.md).

Grades:
- **Urgency**: `high` (users need this), `medium` (nice to have), `low` (speculative/future)
- **Complexity**: `low` (hours), `medium` (days), `high` (weeks+)

## Test Coverage

### Add tests for untested generators
- 14 out of 17 generator processors have no integration tests: a2x, drawio, gem, libreoffice, markdown, marp, mermaid, npm, pandoc, pdflatex, pdfunite, pip, sphinx.
- The test pattern is well-established in `tests/processors/` — each test creates a temp project, writes source files, runs `rsconstruct build`, and verifies outputs.
- **Urgency**: high | **Complexity**: low (per processor)

### Add tests for untested checkers
- 5 checkers have no integration tests: ascii, aspell, markdownlint, mdbook, mdl.
- **Urgency**: medium | **Complexity**: low (per processor)

## New Processors

### Linting / Checking (stub-based)

#### yamllint
- Lint YAML files (`.yml`, `.yaml`) using `yamllint`.
- Catches syntax errors and style violations.
- Config: `linter` (default `"yamllint"`), `args`, `extra_inputs`, `scan`.
- **Urgency**: medium | **Complexity**: low

#### jsonlint
- Validate JSON files (`.json`) for syntax errors.
- Could use `python3 -m json.tool` or a dedicated tool like `jsonlint`.
- Config: `linter`, `args`, `extra_inputs`, `scan`.
- **Urgency**: medium | **Complexity**: low

#### toml-lint
- Validate TOML files (`.toml`) for syntax errors.
- Could use `taplo check` or a built-in Rust parser.
- Config: `linter` (default `"taplo"`), `args`, `extra_inputs`, `scan`.
- **Urgency**: low | **Complexity**: low

#### markdownlint
- Lint Markdown files (`.md`) for structural issues (complements zspell which only checks spelling).
- Uses `mdl` or `markdownlint-cli`.
- Config: `linter` (default `"mdl"`), `args`, `extra_inputs`, `scan`.
- **Urgency**: low | **Complexity**: low

#### black-check
- Python formatting verification using `black --check`.
- Verifies files are formatted without modifying them.
- Config: `args`, `extra_inputs`, `scan`.
- **Urgency**: low | **Complexity**: low

### Compilation / Generation

#### rust_single_file
- Compile single-file Rust programs (`.rs`) to executables, like cc_single_file but for Rust.
- Useful for exercise/example repositories.
- Config: `rustc` (default `"rustc"`), `flags`, `output_suffix`, `extra_inputs`, `scan`.
- **Urgency**: medium | **Complexity**: medium

#### sass
- Compile `.scss`/`.sass` files to `.css`.
- Single-file transformation using `sass` or `dart-sass`.
- Config: `compiler` (default `"sass"`), `args`, `extra_inputs`, `scan`.
- **Urgency**: low | **Complexity**: low

#### protobuf
- Compile `.proto` files to generated code using `protoc`.
- Config: `protoc` (default `"protoc"`), `args`, `language` (default `"cpp"`), `extra_inputs`, `scan`.
- **Urgency**: low | **Complexity**: medium

#### pandoc
- Convert Markdown (`.md`) to other formats (PDF, HTML, EPUB) using `pandoc`.
- Single-file transformation.
- Config: `output_format` (default `"html"`), `args`, `extra_inputs`, `scan`.
- **Urgency**: low | **Complexity**: low

#### jinja2
- Render Jinja2 templates (`.j2`, `.jinja2`) via `python3 -c` using the `jinja2` library.
- Similar to the mako and tera processors but using Jinja2 syntax.
- Scan directory: `templates.jinja2/`, strips prefix and extension for output path.
- Config: `extra_inputs`, `scan`.
- **Urgency**: medium | **Complexity**: low

### Testing

#### pytest
- Run Python test files and produce pass/fail stubs.
- Each `test_*.py` file becomes a product.
- Config: `runner` (default `"pytest"`), `args`, `extra_inputs`, `scan` (default extensions `["test_*.py"]`).
- **Urgency**: medium | **Complexity**: medium

#### doctest
- Run Python doctests and produce stubs.
- Each `.py` file with doctests produces a stub.
- Config: `args`, `extra_inputs`, `scan`.
- **Urgency**: low | **Complexity**: medium

## Build Execution

### Distributed builds
- Run builds across multiple machines, similar to distcc or icecream for C/C++.
- A coordinator node distributes work to worker nodes, each running rsconstruct in worker mode.
- Workers execute products and return outputs to the coordinator, which caches them locally.
- Challenges: network overhead for small products, identical tool versions across workers, local filesystem access.
- **Urgency**: low | **Complexity**: high

### Sandboxed execution
- Run each processor in an isolated environment where it can only access its declared inputs.
- Prevents accidental undeclared dependencies.
- On Linux, namespaces can provide lightweight sandboxing.
- **Urgency**: low | **Complexity**: high

### Content-addressable outputs (unchanged output pruning)
- Hash outputs too to skip downstream rebuilds when an input changes but produces identical output.
- Bazel calls this "unchanged output pruning."
- **Urgency**: medium | **Complexity**: medium

### Persistent daemon mode
- Keep rsconstruct running as a background daemon to avoid startup overhead.
- Benefits: instant file index via inotify, warm Lua VMs, connection pooling, faster incremental builds.
- Daemon listens on Unix socket (`.rsconstruct/daemon.sock`).
- `rsconstruct watch` becomes a client that triggers rebuilds on file events.
- **Urgency**: low | **Complexity**: high

### Persistent workers
- Keep long-running tool processes alive to avoid startup overhead.
- Instead of spawning `ruff` or `pylint` per invocation, keep one process alive and feed it files.
- Bazel gets 2-4x speedup for Java this way. Could benefit pylint/mypy which have heavy startup.
- Multiplex variant: multiple requests to a single worker process via threads.
- **Urgency**: medium | **Complexity**: high

### Dynamic execution (race local vs remote)
- Start both local and remote execution of the same product; use whichever finishes first and cancel the other.
- Useful when remote cache is slow or flaky.
- Configurable per-processor via execution strategy.
- **Urgency**: low | **Complexity**: high

### Execution strategies per processor
- Map each processor to an execution strategy: local, remote, sandboxed, or dynamic.
- Different processors may benefit from different execution models.
- Config: `[processor.ruff] execution = "remote"`, `[processor.cc_single_file] execution = "sandboxed"`.
- **Urgency**: low | **Complexity**: medium

### Build profiles
- Named configuration sets for different build scenarios (ci, dev, release).
- Profiles inherit from base configuration and override specific values.
- Usage: `rsconstruct build --profile=ci`
- **Urgency**: medium | **Complexity**: medium

### Conditional processors
- Enable or disable processors based on conditions (environment variables, file existence, git branch, custom commands).
- Multiple conditions can be combined with `all`/`any` logic.
- **Urgency**: low | **Complexity**: medium

### Target aliases
- Define named groups of processors for easy invocation.
- Usage: `rsconstruct build @lint`, `rsconstruct build @test`
- Special aliases: `@all`, `@changed`, `@failed`
- File-based targeting: `rsconstruct build src/main.c`
- **Urgency**: medium | **Complexity**: medium

## Graph & Query

### Build graph query language
- Support queries like `rsconstruct query deps out/foo`, `rsconstruct query rdeps src/main.c`, `rsconstruct query processor:ruff`.
- Useful for debugging builds and CI systems that want to build only affected targets.
- **Urgency**: low | **Complexity**: medium

### Affected analysis
- Given changed files (from `git diff`), determine which products are affected and only build those.
- Useful for large projects where a full build is expensive.
- **Urgency**: medium | **Complexity**: medium

### Critical path analysis
- Identify the longest sequential chain of actions in a build.
- Helps users optimize their slowest builds by showing what's actually on the critical path.
- Display with `rsconstruct build --critical-path` or include in `--timings` output.
- **Urgency**: medium | **Complexity**: medium

## Extensibility

### Plugin registry
- A central repository of community-contributed Lua plugins.
- Install with `rsconstruct plugin install eslint`.
- Registry could be a GitHub repository with a JSON index.
- Version pinning in `rsconstruct.toml`.
- **Urgency**: low | **Complexity**: high

### Project templates
- Initialize new projects with pre-configured processors and directory structure.
- `rsconstruct init --template=python`, `rsconstruct init --template=cpp`, etc.
- Custom templates from local directories or URLs.
- **Urgency**: low | **Complexity**: medium

### Rule composition / aspects
- Attach cross-cutting behavior to all targets of a certain type (e.g., "add coverage analysis to every C++ compile").
- **Urgency**: low | **Complexity**: high

### Output groups / subtargets
- Named subsets of a target's outputs that can be requested selectively.
- E.g., `rsconstruct build --output-group=debug` or per-product subtarget selection.
- Useful for targets that produce multiple output types (headers, binaries, docs).
- **Urgency**: low | **Complexity**: medium

### Visibility / access control
- Restrict which processors can consume which files or directories.
- Prevents accidental cross-boundary dependencies in large repos.
- Config: per-processor `visibility` rules or directory-level `.rsconstruct-visibility` files.
- **Urgency**: low | **Complexity**: medium

## Developer Experience

### Build profiling / tracing
- Generate Chrome trace format or flamegraph SVG showing what ran when, including parallel lanes.
- Include critical path highlighting, CPU usage, and idle time analysis.
- Usage: `rsconstruct build --trace=build.json`
- Viewable in `chrome://tracing` or Perfetto UI.
- **Urgency**: medium | **Complexity**: medium

### Build Event Protocol / structured event stream
- rsconstruct has `--json` on stdout, but a proper Build Event Protocol (file or gRPC stream) enables external dashboards, CI integrations, and build analytics services.
- Write events to a file (`--build-event-log=events.pb`) or stream to a remote service.
- Richer event types than current JSON Lines: action graph, configuration, progress, test results.
- **Urgency**: medium | **Complexity**: medium

### Build notifications
- Desktop notifications when builds complete, especially for long builds.
- Platform-specific: `notify-send` (Linux), `osascript` (macOS).
- Config: `notify = true`, `notify_on_success = false`.
- **Urgency**: low | **Complexity**: low

### `rsconstruct build <target>` — Build specific targets
- Build only specific targets by name or pattern:
  `rsconstruct build src/main.c`, `rsconstruct build out/cc_single_file/`, `rsconstruct build "*.py"`
- **Urgency**: medium | **Complexity**: medium

### Parallel dependency analysis
- The cpp analyzer scans files sequentially, which can be slow for large codebases.
- Parallelize header scanning using rayon or tokio.
- **Urgency**: low | **Complexity**: medium

### IDE / LSP integration
- Language Server Protocol server for IDE integration.
- Features: diagnostics, code actions, hover info, file decorations.
- Plugins for VS Code, Neovim, Emacs.
- **Urgency**: low | **Complexity**: high

### Build log capture
- Save stdout/stderr from each product execution to a log file.
- Config: `log_dir = ".rsconstruct/logs"`, `log_retention = 10`.
- `rsconstruct log ruff:main.py` to view logs.
- **Urgency**: low | **Complexity**: medium

### Build timing history
- Store timing data to `.rsconstruct/timings.json` after each build.
- `rsconstruct timings` shows slowest products, trends, time per processor.
- **Urgency**: low | **Complexity**: medium

### Remote cache authentication
- Support authenticated remote caches: S3 (AWS credentials), HTTP (bearer tokens), GCS.
- Variable substitution from environment for secrets.
- **Urgency**: medium | **Complexity**: medium

### `rsconstruct fmt` — Auto-format source files
- Run formatters (black, isort, clang-format, rustfmt) that modify files in-place.
- Distinct from checkers which only verify — formatters actually fix formatting.
- Could be a new processor type (`Formatter`) or a convenience command that runs formatter processors.
- **Urgency**: medium | **Complexity**: medium

### `rsconstruct why <file>` — Explain why a file is built
- Show which processors consume a given file, what products it belongs to, and what triggered a rebuild.
- Useful for debugging unexpected rebuilds or understanding the build graph.
- **Urgency**: medium | **Complexity**: low

### `rsconstruct doctor` — Diagnose build environment
- Check for common issues: missing tools, misconfigured processors, stale cache, version mismatches.
- Report warnings and suggestions for fixing problems.
- **Urgency**: medium | **Complexity**: low

### `rsconstruct lint` — Run only checkers
- Convenience command to run only checker processors.
- Equivalent to `rsconstruct build -p ruff,pylint,...` but shorter.
- **Urgency**: low | **Complexity**: low

### `rsconstruct sloc` — Source lines of code statistics
- Count source lines of code across the project, broken down by language/extension.
- Leverage rsconstruct's existing file index and extension-to-language mapping from processor configs.
- Show: files, blank lines, comment lines, code lines per language. Total summary.
- Optional COCOMO-style effort/cost estimation (person-months, schedule, cost at configurable salary).
- Usage: `rsconstruct sloc`, `rsconstruct sloc --json`, `rsconstruct sloc --cocomo --salary 100000`
- Similar to external tools: `sloccount`, `cloc`, `tokei`.
- **Urgency**: low | **Complexity**: medium

### Watch mode keyboard commands
- During `rsconstruct watch`, support `r` (rebuild), `c` (clean), `q` (quit), `Enter` (rebuild now), `s` (status).
- Only activate when stdin is a TTY.
- **Urgency**: low | **Complexity**: medium

### Layered config files
- Support config file layering: system (`/etc/rsconstruct/config.toml`), user (`~/.config/rsconstruct/config.toml`), project (`rsconstruct.toml`).
- Lower layers provide defaults, higher layers override.
- Per-command overrides via `[build]`, `[watch]` sections.
- Similar to Bazel's `.bazelrc` layering.
- **Urgency**: low | **Complexity**: low

### Test sharding
- Split large test targets across multiple parallel shards.
- Set `TEST_TOTAL_SHARDS` and `TEST_SHARD_INDEX` environment variables for test runners.
- Config: `shard_count = 4` per processor or product.
- Useful for pytest/doctest processors when added.
- **Urgency**: low | **Complexity**: medium

### Runfiles / runtime dependency trees
- Track runtime dependencies (shared libs, config files, data files) separately from build dependencies.
- Generate a runfiles directory per executable with symlinks to all transitive runtime deps.
- Useful for deployment, packaging, and containerization.
- **Urgency**: low | **Complexity**: high

## Caching & Performance

### Deferred materialization
- Don't write cached outputs to disk until they're actually needed by a downstream product.
- **Urgency**: low | **Complexity**: high

### Garbage collection policy
- Time-based or size-based cache policies: "keep cache under 1GB" or "evict entries older than 30 days."
- Config: `max_size = "1GB"`, `max_age = "30d"`, `gc_policy = "lru"`.
- `rsconstruct cache gc` for manual garbage collection.
- **Urgency**: low | **Complexity**: medium

### Shared cache across branches
- Surface in `rsconstruct status` when products are restorable from another branch.
- Already works implicitly via input hash matching.
- **Urgency**: low | **Complexity**: low

### Merkle tree input hashing
- Hash inputs as a Merkle tree rather than flat concatenation.
- More efficient for large input sets — changing one file only rehashes its branch, not all inputs.
- Also enables efficient transfer of input trees to remote execution workers.
- **Urgency**: low | **Complexity**: medium

## Reproducibility

### Hermetic builds
- Control all inputs beyond tool versions: isolate env vars, control timestamps, sandbox network, pin system libraries.
- Config: `hermetic = true`, `allowed_env = ["HOME", "PATH"]`.
- Verification: `rsconstruct build --verify` builds twice and compares outputs.
- **Urgency**: low | **Complexity**: high

### Determinism verification
- `rsconstruct build --verify` mode that builds each product twice and compares outputs.
- **Urgency**: low | **Complexity**: medium

## CI & Reporting

### CI config generator
- `rsconstruct ci generate` outputs a GitHub Actions or GitLab CI config that runs the build.
- Detects enabled processors and required tools, generates install steps and build commands.
- Supports `--format=github|gitlab|circleci`.
- **Urgency**: medium | **Complexity**: medium

### HTML build report
- Generate a visual HTML dashboard of build times, cache hit rates, and processor statistics.
- `rsconstruct build --report=build.html` or `rsconstruct report`.
- Include charts for timing trends, per-processor breakdown, cache efficiency.
- **Urgency**: low | **Complexity**: medium

### PR comment bot
- Post build results (pass/fail, timing, warnings) as a GitHub PR comment.
- `rsconstruct ci comment` reads build output and posts via GitHub API.
- **Urgency**: low | **Complexity**: medium

## Content & Documentation

### `rsconstruct init --detect`
- Scan the project and generate a complete `rsconstruct.toml` with all detected processors enabled and configured.
- Goes beyond `smart auto` by also setting processor-specific config (scan_dirs, extensions, tool paths).
- **Urgency**: medium | **Complexity**: medium

### `rsconstruct fmt`  — Auto-format rsconstruct.toml
- Sort `[processor.*]` sections alphabetically, align values, remove redundant defaults.
- Distinct from the existing `rsconstruct fmt` suggestion about formatting source files.
- **Urgency**: low | **Complexity**: low

### Cross-project term sync
- Automatically keep terms directories in sync across multiple repos.
- Could run as a daemon or a periodic CI job.
- `rsconstruct terms sync --repos=repo1,repo2` or config-driven.
- **Urgency**: low | **Complexity**: medium

### Glossary generator
- `rsconstruct terms glossary` generates a markdown glossary from the terms directory.
- Optionally pulls definitions from context in the markdown files where terms are used.
- **Urgency**: low | **Complexity**: medium

### Link checker processor
- Validate that URLs in markdown files are not broken (HTTP HEAD requests).
- Configurable timeout, retry, and allow/blocklist patterns.
- Cache results to avoid re-checking unchanged URLs.
- **Urgency**: medium | **Complexity**: medium

### Image optimizer processor
- Compress and resize images referenced in markdown files.
- Uses tools like `optipng`, `jpegoptim`, `svgo`.
- Config: quality levels, max dimensions, output format.
- **Urgency**: low | **Complexity**: medium

### HTML+JS compression and packaging
- Minify and bundle HTML, CSS, and JavaScript files for deployment.
- Could use tools like `terser` (JS), `csso` (CSS), `html-minifier` (HTML).
- Bundle multiple JS/CSS files into single outputs, generate source maps.
- Integrate with existing eslint/stylelint processors for a full web frontend pipeline.
- **Urgency**: medium | **Complexity**: medium

## Processor Ecosystem

### WASM processor plugins
- Beyond Lua, allow processors written in any language compiled to WebAssembly.
- Provides sandboxing, portability, and language flexibility.
- WASI for filesystem access within the sandbox.
- **Urgency**: low | **Complexity**: high

### Processor marketplace / registry
- A central repository of community-contributed processor configs and Lua plugins.
- Install with `rsconstruct plugin install prettier`.
- Registry as a GitHub repository with a JSON index. Version pinning in `rsconstruct.toml`.
- **Urgency**: low | **Complexity**: high

## Security

### Shell command execution from source file comments
- `EXTRA_*_SHELL` directives execute arbitrary shell commands parsed from source file comments.
- Document the security implications clearly.
- **Urgency**: medium | **Complexity**: low