Runtime configuration

Padas Core reads TOML from $PADAS_HOME/etc/. PADAS_HOME is the resolved Core home (for example /opt/padas/core when installed under /opt/padas).

Configuration files and load order

File	Role
padas.default.toml	Shipped baseline: full key set, defaults, comments. Do not edit—upgrades may replace it.
padas.toml	Operator-owned overrides; pass to --config. Only keys you set override defaults; everything else resolves from padas.default.toml.

Load order:

1. Defaults — padas.default.toml (or build-equivalent next to your operator file).
2. Overrides — padas.toml at the --config path.
3. Environment — container paths, PADAS_HOME, and variables documented in the default file.

Keep one operator file per environment (dev / staging / production). In Core’s source tree, the template may appear as configs/padas.default.toml; packages place the pair under $PADAS_HOME/etc/.

Lab bring-up: minimal padas.toml

For trusted localhost or doc-style curl only, you can override API transport while inheriting all other defaults. Not for untrusted networks—TLS and API auth are disabled.

# Lab / trusted localhost only
[api]
host = "127.0.0.1"

[api.tls]
enabled = false

[api.auth]
enabled = false

Point --config at your file (for example /opt/padas/core/etc/padas.toml with PADAS_HOME=/opt/padas/core), with padas.default.toml in the same directory.

Production baselines: first deltas from defaults

Beyond lab, keep padas.default.toml read-only and usually touch at least:

Area	Action
TLS	[api.tls].enabled = true with real cert_file / key_file.
Auth	[api.auth].enabled = true; protect service_account_token_file and your rotation process.
CORS	Replace allowed_origins = "*" with explicit UI or gateway origins.
Paths	Prefer absolute core.data_dir, WAL path, registry config_dir, logs—on durable disks.
Logs	e.g. [observability.logging].format = "Json" for centralized logging.
Metrics	[observability.metrics_collection].level = 2 is a common production balance unless comments say otherwise.

Verify HTTP API reachability

After Core starts, confirm transport matches [api.tls]:

• Status: https://<CORE_IP>:8999/api/v1/status
• Metrics: https://<CORE_IP>:8999/api/v1/metrics

Use curl --insecure for self-signed TLS, or http only when TLS is explicitly off in padas.toml.

Padas UI pairing: Core writes UI registration material under $PADAS_HOME/data/security/service-account.token (for example /opt/padas/core/data/security/service-account.token when PADAS_HOME=/opt/padas/core). Steps: Quickstart: Core + UI.

API and registry ([api])

Topic	What to set
Listen address	host (all interfaces vs localhost-only).
Port	port — default 8999.
Base path	prefix — default /api/v1.
TLS	[api.tls] — enabled, certificate and key paths.
Auth	[api.auth] — service-account tokens, paths, rotation.
CORS	[api.cors] — when browsers call the API directly.
Registry on disk	[api.persistence] — config_dir and filenames for streams, tasks, connectors the API manages.

Engine ([core])

Topic	Notes
Concurrency	workers — 0 often means auto from CPU.
Sharding	virtual_shards — distribution semantics.
Data root	data_dir — base for WAL, state, and related paths unless overridden.
Startup / shutdown	startup_timeout_secs, shutdown_timeout_secs.

Streams, WAL, and backpressure

Stream defaults ([core.stream] and nested tables):

• Retries — retry_attempts, retry_backoff_ms.
• Per-stream buffers — [core.stream.buffer] — max_events, timeout_ms, mode (timeout / drop / block), cleanup thresholds. Drives latency vs drops under load.
• Per-stream WAL — [core.stream.wal] plus [core.wal.batch] (max_bytes, max_events, max_timeout_ms). WAL off favors throughput; WAL on favors durability and catch-up.

Global WAL ([core.wal]):

• path — often under data_dir.
• Retention — retention_ms, retention_bytes, segment caps.
• Durability vs speed — sync_writes, batch and channel buffers ([core.wal.batch], [core.wal.buffer]).
• Compression — [core.wal.compression].

Subscribers and lag

[core.subscriber], [core.subscriber.lag], and batch tables control StreamRouter + WAL behaviour, lag thresholds, and read batch sizes when consumers fall behind.

Tasks, state, and aggregations

• [core.state] — Storage engine (rocksdb vs memory), paths, checkpointing, window retention, late-event behaviour.
• [core.task] — Thread limits, per-task workers, [core.task.buffer], [core.task.batch], [core.task.aggregation.cleanup] for windowed PDL.
• [core.task.aggregation.watermark] — Idle/shutdown flushing for tumbling windows; defaults usually suffice until you tighten late or idle behaviour.

Tune when you run heavy aggregation or need stricter recovery checkpoints.

Observability ([observability])

Topic	Purpose
Master switch	enabled — metrics, logging, system streams.
System streams	[observability.system_streams.*] — internal and metrics streams (names, retention, compression).
Metrics collection	[observability.metrics_collection] — enable, level, category flags.
Metrics churn	coalesce_unchanged_stream_subscriber_metrics — fewer duplicate rows in _padas_metrics when drops unchanged (default true).
Metrics HTTP	[observability.metrics_api] — GET /api/v1/metrics; optional aggregation_cache_ttl_ms.
Logging	[observability.logging] — level, format (Text / Json), path, rotation, retention.

Connectors subsystem ([connectors])

• enabled — Subsystem on/off.
• auto_start — Auto-start discovered connectors.
• [connectors.discovery] — search_paths, extensions for plugin binaries.

Concrete connector rows live in the registry (see [api.persistence]) or your packaging layout.

Default capacity assumptions

Numeric defaults in padas.default.toml (stream buffers, task workers, WAL batching, …) target a typical single-node host in the 8–12 vCPU, 8–32 GiB RAM, roughly 10–20 K events/s per pipeline with ~512–1024 byte events—unless a specific comment overrides that story. For higher sustained throughput, scale buffers and task settings using the same file’s comments as guidance, not arbitrary copies.

Conventions and naming

Convention	Rule
Naming	snake_case keys.
Units	Prefer *_ms, *_bytes in key names.
Overrides only	Edit padas.toml beside padas.default.toml; override only what you need.

Registry object IDs and patterns: Naming conventions.

Tuning matrix (goals → tables)

Goal	Where to look first
Higher throughput (more memory OK)	[core.stream.buffer], [core.wal.buffer], [core.task.batch].
Lower latency	Smaller batch max_events / max_timeout_ms, tighter buffer timeouts.
Stronger durability	WAL on critical streams; sync_writes; TLS + auth on [api].
Less disk	WAL and stream retention, compression, aggregation cleanup intervals.
Connector stability	Class-specific retry/backpressure (see connector docs bundled with Core).

End-to-end checklist

1. Set [api] host / port, TLS, and auth for your network model.
2. Point core.data_dir (and custom WAL path if any) at writable storage sized for retention.
3. Choose per-stream WAL vs global [core.wal] retention for durability vs disk.
4. Configure [observability.logging] and [observability.metrics_*] for your stack.
5. Confirm /api/v1/status and /api/v1/metrics (scheme matches [api.tls]).

Where to go next

• Naming conventions
• System requirements
• Installation — Linux