weefnn/full-stack-fastapi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add mqtt raw ingestion implementation plan

Browse Source
This commit is contained in:
魏风
2026-03-17 12:08:05 +08:00
parent e0bc70d568
commit b92b1f75f0
1 changed files with 581 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

581
docs/superpowers/plans/2026-03-17-mqtt-raw-ingestion-implementation.md Normal file
View File

@@ -0,0 +1,581 @@
# MQTT Raw Ingestion Pipeline Implementation Plan
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build a production-safe MQTT ingestion pipeline that subscribes to `terminal/{device_id}/raw`, stores payload bytes losslessly, parses NMEA + RTCM 3.2 type summaries asynchronously, and supports 90-day hot data plus local cold archive.
**Architecture:** Use a DB-queue two-stage pipeline: `mqtt-ingestor` writes raw payloads to PostgreSQL immediately, and `parser-worker` asynchronously consumes pending rows to produce event/state tables. A separate archive job writes local cold files and manifest records, then prunes hot data beyond retention.
**Tech Stack:** FastAPI/SQLModel/Alembic/PostgreSQL, paho-mqtt, pytest, Docker Compose.
---
## Scope Check
This plan covers one cohesive subsystem (MQTT ingest + parse + archive pipeline). It intentionally excludes frontend UI and realtime push implementation, but produces backend data structures required for both.
## Execution Skills
- `@superpowers/test-driven-development`
- `@superpowers/systematic-debugging`
- `@superpowers/verification-before-completion`
## File Structure Map
### Create
- `backend/app/mqtt/__init__.py` — MQTT package marker.
- `backend/app/mqtt/topic.py` — topic parsing and validation (`device_id` extraction).
- `backend/app/mqtt/repository.py` — DB helpers for raw insert, pending fetch, status update, event/state write, archive manifest.
- `backend/app/mqtt/parsers/__init__.py` — parser package exports.
- `backend/app/mqtt/parsers/nmea.py` — NMEA sentence extraction + structured fields.
- `backend/app/mqtt/parsers/rtcm.py` — RTCM 3.x frame scan + type summary.
- `backend/app/mqtt/parsers/mixed.py` — mixed payload scanner orchestrating NMEA + RTCM parsers.
- `backend/app/mqtt/ingestor.py` — MQTT client callbacks + buffered DB flush.
- `backend/app/mqtt/parser_worker.py` — pending job polling loop (`FOR UPDATE SKIP LOCKED`).
- `backend/app/mqtt/archive_job.py` — local archive writer and retention cleanup.
- `backend/app/alembic/versions/a1b2c3d4e5f6_add_mqtt_pipeline_tables.py` — migration for raw/event/state/archive tables.
- `backend/tests/mqtt/test_topic.py` — unit tests for topic parser.
- `backend/tests/mqtt/test_mixed_parser.py` — unit tests for mixed payload parse behavior.
- `backend/tests/mqtt/test_parser_worker.py` — DB-backed tests for pending->parsed transitions.
- `backend/tests/mqtt/test_archive_job.py` — archive/manifest/cleanup tests.
### Modify
- `backend/pyproject.toml` — add MQTT dependency (`paho-mqtt`).
- `backend/app/core/config.py` — add MQTT and pipeline configuration env vars.
- `backend/app/models.py` — add `TerminalMessageRaw`, `TerminalObservationEvent`, `TerminalDeviceState`, `TerminalRawArchiveManifest` models.
- `compose.yml` — add production `mqtt-ingestor` and `mqtt-parser-worker` services.
- `compose.override.yml` — add local dev services and optional archive runner profile.
- `.env.production.example` — add MQTT and archive env examples.
- `backend/README.md` — operational docs for running ingestor/worker/archive and verification commands.
### Optional (if command wiring is needed)
- `backend/pyproject.toml` entry points for CLI commands (`mqtt-ingestor`, `mqtt-parser-worker`, `mqtt-archive`).
---
### Task 1: Configuration + Topic Parsing Foundation
**Files:**
- Create: `backend/app/mqtt/__init__.py`
- Create: `backend/app/mqtt/topic.py`
- Test: `backend/tests/mqtt/test_topic.py`
- Modify: `backend/app/core/config.py`
- Modify: `backend/pyproject.toml`
- [ ] **Step 1: Write failing tests for topic parsing**
```python
# backend/tests/mqtt/test_topic.py
import pytest
from app.mqtt.topic import extract_device_id
def test_extract_device_id_success() -> None:
assert extract_device_id("terminal/device-001/raw") == "device-001"
def test_extract_device_id_rejects_invalid_topic() -> None:
with pytest.raises(ValueError, match="terminal/.+/raw"):
extract_device_id("terminal/device-001/nmea")
```
- [ ] **Step 2: Run test to verify it fails**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_topic.py -v`
Expected: FAIL with `ModuleNotFoundError: No module named 'app.mqtt.topic'`
- [ ] **Step 3: Implement minimal topic parser + config fields**
```python
# backend/app/mqtt/topic.py
import re
TOPIC_RE = re.compile(r"^terminal/(?P<device_id>[^/]+)/raw$")
def extract_device_id(topic: str) -> str:
match = TOPIC_RE.match(topic)
if not match:
raise ValueError("topic must match terminal/{device_id}/raw")
return match.group("device_id")
```
```python
# backend/app/core/config.py (additions)
MQTT_HOST: str = "101.35.119.226"
MQTT_PORT: int = 1883
MQTT_USERNAME: str = ""
MQTT_PASSWORD: str = ""
MQTT_TOPIC_PATTERN: str = "terminal/+/raw"
MQTT_CLIENT_ID: str = "spatialhub-mqtt-ingestor"
RAW_HOT_RETENTION_DAYS: int = 90
RAW_ARCHIVE_BASE_DIR: str = "/data/archive/gnss"
PARSER_BATCH_SIZE: int = 200
PARSER_POLL_INTERVAL_MS: int = 300
```
- [ ] **Step 4: Run tests to verify pass**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_topic.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add backend/app/mqtt/__init__.py backend/app/mqtt/topic.py backend/tests/mqtt/test_topic.py backend/app/core/config.py backend/pyproject.toml
git commit -m "feat: add MQTT topic parsing and pipeline config"
```
### Task 2: Add DB Schema for Raw/Event/State/Archive
**Files:**
- Modify: `backend/app/models.py`
- Create: `backend/app/alembic/versions/a1b2c3d4e5f6_add_mqtt_pipeline_tables.py`
- Test: `backend/tests/mqtt/test_parser_worker.py`
- [ ] **Step 1: Write failing DB test for raw row persistence**
```python
# backend/tests/mqtt/test_parser_worker.py
from sqlmodel import Session
from app.models import TerminalMessageRaw
def test_terminal_message_raw_model_can_persist(db: Session) -> None:
row = TerminalMessageRaw(
device_id="device-001",
topic="terminal/device-001/raw",
payload_bytes=b"$GPGGA,123519,3723.2475,N,12158.3416,W,1,12,0.8,10.0,M,-25.0,M,,*65\r\n",
)
db.add(row)
db.commit()
db.refresh(row)
assert row.id is not None
```
- [ ] **Step 2: Run test to verify it fails**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_parser_worker.py::test_terminal_message_raw_model_can_persist -v`
Expected: FAIL with `ImportError`/`AttributeError` for missing `TerminalMessageRaw`
- [ ] **Step 3: Implement models and migration**
```python
# backend/app/models.py (new table skeleton example)
class TerminalMessageRaw(SQLModel, table=True):
id: uuid.UUID = Field(default_factory=uuid.uuid4, primary_key=True)
device_id: str = Field(index=True)
topic: str
received_at: datetime = Field(default_factory=get_datetime_utc, sa_type=DateTime(timezone=True))
payload_bytes: bytes
payload_size: int
payload_sha256: str = Field(max_length=64, index=True)
parse_status: str = Field(default="pending", index=True)
```
```python
# alembic upgrade should create:
# terminal_message_raw, terminal_observation_event, terminal_device_state, terminal_raw_archive_manifest
```
```bash
# if creating from CLI
cd backend && ../.venv/bin/alembic revision --autogenerate -m "add mqtt pipeline tables" --rev-id a1b2c3d4e5f6
```
- [ ] **Step 4: Run migration + targeted test**
Run: `cd backend && ../.venv/bin/alembic upgrade head`
Expected: migration applies without errors
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_parser_worker.py::test_terminal_message_raw_model_can_persist -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add backend/app/models.py backend/app/alembic/versions/*.py backend/tests/mqtt/test_parser_worker.py
git commit -m "feat: add MQTT pipeline database schema"
```
### Task 3: Implement Raw Ingestion Repository + MQTT Ingestor
**Files:**
- Create: `backend/app/mqtt/repository.py`
- Create: `backend/app/mqtt/ingestor.py`
- Test: `backend/tests/mqtt/test_parser_worker.py`
- [ ] **Step 1: Write failing test for raw insert helper**
```python
def test_insert_raw_message_sets_pending_status(db: Session) -> None:
row = insert_raw_message(
session=db,
device_id="device-001",
topic="terminal/device-001/raw",
payload=b"$GPRMC,123520,A,3723.2475,N,12158.3416,W,0.13,309.62,120598,,,A*74\r\n",
qos=1,
retain=False,
)
assert row.parse_status == "pending"
assert row.payload_size > 0
```
- [ ] **Step 2: Run test to verify failure**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_parser_worker.py::test_insert_raw_message_sets_pending_status -v`
Expected: FAIL with `NameError`/missing repository function
- [ ] **Step 3: Implement repository + ingestor callback path**
```python
# backend/app/mqtt/repository.py
def insert_raw_message(*, session: Session, device_id: str, topic: str, payload: bytes, qos: int, retain: bool) -> TerminalMessageRaw:
row = TerminalMessageRaw(
device_id=device_id,
topic=topic,
payload_bytes=payload,
payload_size=len(payload),
payload_sha256=hashlib.sha256(payload).hexdigest(),
qos=qos,
retain=retain,
parse_status="pending",
)
session.add(row)
session.commit()
session.refresh(row)
return row
```
```python
# backend/app/mqtt/ingestor.py (callback shape)
def on_message(client: Client, userdata: object, message: MQTTMessage) -> None:
device_id = extract_device_id(message.topic)
with Session(engine) as session:
insert_raw_message(
session=session,
device_id=device_id,
topic=message.topic,
payload=message.payload,
qos=message.qos,
retain=message.retain,
)
```
- [ ] **Step 4: Run tests for repository path**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_parser_worker.py -k insert_raw_message -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add backend/app/mqtt/repository.py backend/app/mqtt/ingestor.py backend/tests/mqtt/test_parser_worker.py
git commit -m "feat: persist MQTT raw messages via ingestor repository"
```
### Task 4: Build NMEA Parser for GGA/RMC/GSV/GSA/GST
**Files:**
- Create: `backend/app/mqtt/parsers/__init__.py`
- Create: `backend/app/mqtt/parsers/nmea.py`
- Test: `backend/tests/mqtt/test_mixed_parser.py`
- [ ] **Step 1: Write failing tests for NMEA extraction**
```python
def test_parse_nmea_sentences_extracts_gga_and_rmc() -> None:
payload = b"$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47\r\n$GPRMC,123520,A,4807.038,N,01131.000,E,0.13,309.62,120598,,,A*10\r\n"
events = parse_nmea_sentences(payload)
assert {e["event_type"] for e in events} >= {"nmea_gga", "nmea_rmc"}
```
- [ ] **Step 2: Run test to verify failure**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_mixed_parser.py -k nmea -v`
Expected: FAIL with missing parser function
- [ ] **Step 3: Implement minimal NMEA parser**
```python
# backend/app/mqtt/parsers/nmea.py
def parse_nmea_sentences(payload: bytes) -> list[dict[str, Any]]:
events: list[dict[str, Any]] = []
for raw_line in payload.splitlines():
if not raw_line.startswith(b"$"):
continue
line = raw_line.decode("ascii", errors="ignore")
talker_and_type = line[3:6].lower()
if talker_and_type in {"gga", "rmc", "gsv", "gsa", "gst"}:
events.append({"event_type": f"nmea_{talker_and_type}", "raw": line})
return events
```
- [ ] **Step 4: Run NMEA tests**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_mixed_parser.py -k nmea -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add backend/app/mqtt/parsers/__init__.py backend/app/mqtt/parsers/nmea.py backend/tests/mqtt/test_mixed_parser.py
git commit -m "feat: parse NMEA sentences into observation events"
```
### Task 5: Build RTCM Summary + Mixed Payload Scanner
**Files:**
- Create: `backend/app/mqtt/parsers/rtcm.py`
- Create: `backend/app/mqtt/parsers/mixed.py`
- Test: `backend/tests/mqtt/test_mixed_parser.py`
- [ ] **Step 1: Write failing test for RTCM type summary in mixed payload**
```python
def test_parse_mixed_payload_returns_rtcm_summary() -> None:
payload = (
b"$GPGSV,1,1,04,02,17,213,40,05,29,121,37,12,66,314,44,25,14,048,39*71\r\n"
+ b"\xd3\x00\x13"
+ b"\x43\x50\x00"
+ b"\x00" * 16
)
result = parse_mixed_payload(payload)
assert "rtcm_summary" in {e["event_type"] for e in result}
```
- [ ] **Step 2: Run test to verify failure**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_mixed_parser.py -k rtcm -v`
Expected: FAIL with missing mixed parser
- [ ] **Step 3: Implement RTCM scanner and orchestrator**
```python
# backend/app/mqtt/parsers/rtcm.py
def summarize_rtcm_types(payload: bytes) -> dict[int, int]:
counts: dict[int, int] = {}
i = 0
while i + 5 < len(payload):
if payload[i] != 0xD3:
i += 1
continue
length = ((payload[i + 1] & 0x03) << 8) | payload[i + 2]
end = i + 3 + length + 3
if end > len(payload):
break
msg_type = ((payload[i + 3] << 4) | (payload[i + 4] >> 4)) & 0x0FFF
counts[msg_type] = counts.get(msg_type, 0) + 1
i = end
return counts
```
```python
# backend/app/mqtt/parsers/mixed.py
def parse_mixed_payload(payload: bytes) -> list[dict[str, Any]]:
events = parse_nmea_sentences(payload)
rtcm_counts = summarize_rtcm_types(payload)
if rtcm_counts:
events.append({"event_type": "rtcm_summary", "rtcm_types": rtcm_counts})
return events
```
- [ ] **Step 4: Run mixed parser tests**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_mixed_parser.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add backend/app/mqtt/parsers/rtcm.py backend/app/mqtt/parsers/mixed.py backend/tests/mqtt/test_mixed_parser.py
git commit -m "feat: add RTCM type summary and mixed payload parser"
```
### Task 6: Implement Parser Worker (Pending Queue -> Event/State)
**Files:**
- Create: `backend/app/mqtt/parser_worker.py`
- Modify: `backend/app/mqtt/repository.py`
- Test: `backend/tests/mqtt/test_parser_worker.py`
- [ ] **Step 1: Write failing worker test for status transitions and event writes**
```python
def test_worker_marks_success_and_writes_events(db: Session) -> None:
raw = insert_raw_message(
session=db,
device_id="device-001",
topic="terminal/device-001/raw",
payload=b"$GPGGA,123519,3723.2475,N,12158.3416,W,1,12,0.8,10.0,M,-25.0,M,,*65\r\n",
qos=1,
retain=False,
)
processed = process_pending_batch(session=db, batch_size=10)
assert processed == 1
db.refresh(raw)
assert raw.parse_status == "succeeded"
```
- [ ] **Step 2: Run test to verify failure**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_parser_worker.py -k worker -v`
Expected: FAIL with missing `process_pending_batch`
- [ ] **Step 3: Implement pending fetch with row locking and worker loop**
```python
# repository fetch query shape
select(TerminalMessageRaw).where(
TerminalMessageRaw.parse_status == "pending"
).order_by(TerminalMessageRaw.received_at).with_for_update(
skip_locked=True
).limit(batch_size)
```
```python
# parser_worker.py
def process_pending_batch(*, session: Session, batch_size: int) -> int:
rows = fetch_pending_rows_for_update(session=session, batch_size=batch_size)
for row in rows:
mark_processing(session=session, row=row)
events = parse_mixed_payload(row.payload_bytes)
write_events_and_update_state(session=session, row=row, events=events)
mark_succeeded(session=session, row=row)
return len(rows)
```
- [ ] **Step 4: Run parser worker tests**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_parser_worker.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add backend/app/mqtt/parser_worker.py backend/app/mqtt/repository.py backend/tests/mqtt/test_parser_worker.py
git commit -m "feat: add async parser worker for pending raw messages"
```
### Task 7: Implement Local Archive Job + 90-Day Hot Retention Cleanup
**Files:**
- Create: `backend/app/mqtt/archive_job.py`
- Test: `backend/tests/mqtt/test_archive_job.py`
- Modify: `backend/app/mqtt/repository.py`
- [ ] **Step 1: Write failing archive test for byte-accurate output and manifest**
```python
def test_archive_writes_raw_bytes_and_manifest(tmp_path: Path, db: Session) -> None:
seed_old_raw_rows(db, days_ago=91, payloads=[b"abc", b"\x01\x02"])
output = run_archive_once(session=db, archive_base_dir=tmp_path)
archived_file = Path(output[0].file_path)
assert archived_file.read_bytes() == b"abc\x01\x02"
```
- [ ] **Step 2: Run test to verify failure**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_archive_job.py -v`
Expected: FAIL with missing archive job implementation
- [ ] **Step 3: Implement archive writer and retention cleanup**
```python
# backend/app/mqtt/archive_job.py
def archive_day(*, session: Session, device_id: str, day_start: datetime, output_path: Path) -> None:
with output_path.open("wb") as f:
rows = fetch_rows_for_archive(session=session, device_id=device_id, day_start=day_start)
for row in rows:
f.write(row.payload_bytes)
write_manifest(session=session, device_id=device_id, day_start=day_start, output_path=output_path, rows=rows)
delete_archived_hot_rows(session=session, archived_rows=rows)
```
- [ ] **Step 4: Run archive tests**
Run: `cd backend && ../.venv/bin/pytest tests/mqtt/test_archive_job.py -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add backend/app/mqtt/archive_job.py backend/app/mqtt/repository.py backend/tests/mqtt/test_archive_job.py
git commit -m "feat: add local archive job with hot retention cleanup"
```
### Task 8: Runtime Wiring, Compose Services, and Operational Docs
**Files:**
- Modify: `compose.yml`
- Modify: `compose.override.yml`
- Modify: `.env.production.example`
- Modify: `backend/README.md`
- Modify: `backend/pyproject.toml` (CLI scripts if used)
- [ ] **Step 1: Write failing smoke checklist (documented commands) for ingestor/worker startup**
```text
# backend/README.md verification block should include:
1) start db/backend/mqtt services
2) publish sample payload to terminal/device-001/raw
3) verify raw/event/state rows in postgres
```
- [ ] **Step 2: Run local startup command and capture expected failure before wiring**
Run: `docker compose up -d mqtt-ingestor mqtt-parser-worker`
Expected: FAIL with `no such service` before compose changes
- [ ] **Step 3: Add compose services + env vars + runbook commands**
```yaml
# compose service shape
mqtt-ingestor:
command: ["python", "-m", "app.mqtt.ingestor"]
mqtt-parser-worker:
command: ["python", "-m", "app.mqtt.parser_worker"]
```
- [ ] **Step 4: Run end-to-end verification commands**
Run:
- `cd backend && ../.venv/bin/pytest tests/mqtt -v`
- `cd backend && ../.venv/bin/pytest tests/api/routes -v`
- `cd backend && ../.venv/bin/mypy app`
- `cd backend && ../.venv/bin/ruff check app`
Expected: all PASS
- [ ] **Step 5: Commit**
```bash
git add compose.yml compose.override.yml .env.production.example backend/README.md backend/pyproject.toml
git commit -m "chore: wire mqtt ingestor and parser worker runtime configuration"
```
## Final Verification Gate
- [ ] Run: `cd backend && ../.venv/bin/pytest tests/mqtt tests/api/routes -v`
- [ ] Run: `cd backend && ../.venv/bin/mypy app`
- [ ] Run: `cd backend && ../.venv/bin/ruff check app && ../.venv/bin/ruff format app --check`
- [ ] Run a manual MQTT publish and verify DB rows (`raw`, `event`, `state`) update.
- [ ] Confirm archive output filename format: `device_<id>_<start>_<end>.txt` and content is byte-identical concatenation.
## Implementation Notes
- Keep ingestor lightweight; no heavy parsing inside MQTT callbacks.
- Any parse failure must retain raw row and set retry metadata.
- Do not add metadata into archive download file bytes; only payload concatenation is allowed.
- Use UTC timestamps everywhere for consistent retention and download range slicing.