178 lines
5.3 KiB
Markdown
178 lines
5.3 KiB
Markdown
# Real CV prototype — RealSense D415 + OpenCV
|
||
|
||
**Status:** WORKING_PROTOTYPE
|
||
**Production integrated:** NO (https://arhipovdan.ru does **not** consume this pipeline)
|
||
**Source:** consolidated from branch `drho1y-mvp_1` (`vision_classifier/`) into `cv/`
|
||
|
||
Same Track 3 B/C/D rules as the web twin; different input path (real depth camera vs simulated sensor).
|
||
|
||
## Purpose
|
||
|
||
Measure parcels on a conveyor with an Intel RealSense D415 (depth + color), estimate L×W×H and circularity, classify into zones **B / C / D**, optionally publish results over MQTT for hardware routing.
|
||
|
||
## Data flow
|
||
|
||
```
|
||
RealSense D415 (V4L2 depth + color)
|
||
→ OpenCV segmentation on depth (optional RGB flat detect)
|
||
→ measure L×W×H + circle_ratio
|
||
→ stabilize (median / vote → LOCK)
|
||
→ classify B/C/D
|
||
→ optional MQTT (category, dimensions, servo/motor topics)
|
||
```
|
||
|
||
## Entrypoints (start here)
|
||
|
||
| Command | Role |
|
||
|---|---|
|
||
| `./demo.sh` | Browser HUD demo on `:8080` (needs camera for live view) |
|
||
| `./run.sh --preview` | Live pipeline with JPEG preview frames |
|
||
| `./run.sh --once --no-mqtt --no-motor` | Single-shot / dry hardware |
|
||
| `.venv/bin/python test_classify.py` | Classifier unit checks **without camera** |
|
||
| `.venv/bin/python test_geometry.py` | Geometry helpers **without camera** |
|
||
|
||
Primary modules: `main.py` (live), `demo.py` (HUD), `classify.py` (rules), `measure.py` (depth metrics), `camera.py` (V4L2 RealSense).
|
||
|
||
## Classification rules (Track 3)
|
||
|
||
1. Dimensions must be strictly **> 10×10×10 mm** and **< 450×320×320 mm** → else **C**
|
||
2. Else if `circle_ratio > 0.8` → **D**
|
||
(`K == 0.8` is **not** circular — same strict rule as web `classifier.ts`)
|
||
3. Else → **B**
|
||
|
||
Stabilization: median window + vote → **LOCK**. Uncertain cases fall back to zone **C** after N frames.
|
||
|
||
## File structure
|
||
|
||
```
|
||
cv/
|
||
main.py # live pipeline entry
|
||
demo.py / demo.sh # browser demo
|
||
run.sh # venv bootstrap + main.py
|
||
camera.py # RealSense via V4L2 + ffmpeg depth
|
||
measure.py # segmentation + dimensions
|
||
classify.py # B/C/D rules
|
||
stabilize.py # temporal LOCK
|
||
mqtt_bridge.py # optional MQTT (disabled by default)
|
||
calibrate.py # fx/fy + belt height calibration
|
||
align_color.py # RGB↔depth alignment helper
|
||
tracker.py # multi-object tracking assist
|
||
journal.py # decisions JSONL writer
|
||
demo_hud.py # HUD rendering
|
||
collect_log.py # log helper
|
||
test_classify.py # no-camera tests
|
||
test_geometry.py # no-camera tests
|
||
config.example.yaml # safe defaults (commit)
|
||
config.yaml # local only (gitignored)
|
||
requirements.txt
|
||
Dockerfile / docker-compose.yml
|
||
```
|
||
|
||
## Dependencies
|
||
|
||
**Software**
|
||
|
||
- Python **3.10+** (3.11 recommended; Docker image uses 3.11)
|
||
- `opencv-python-headless`, `numpy`, `PyYAML`, `pillow`, `paho-mqtt` — see `requirements.txt`
|
||
- System: **ffmpeg**, V4L2 (`v4l-utils` useful)
|
||
|
||
**Hardware (live mode)**
|
||
|
||
- Intel **RealSense D415** on USB3
|
||
- Linux host with `/dev/video*` depth+color nodes (Orange PI / x86)
|
||
|
||
`npm` / Node packages are **not** used here.
|
||
|
||
## Installation
|
||
|
||
```bash
|
||
cd cv
|
||
python3 -m venv .venv
|
||
source .venv/bin/activate
|
||
pip install -U pip
|
||
pip install -r requirements.txt
|
||
cp config.example.yaml config.yaml # optional; scripts auto-copy
|
||
```
|
||
|
||
Or simply:
|
||
|
||
```bash
|
||
cd cv
|
||
./demo.sh # creates .venv and config.yaml on first run
|
||
```
|
||
|
||
## Demo / tests without claiming live camera
|
||
|
||
Classifier and geometry (no RealSense required):
|
||
|
||
```bash
|
||
cd cv
|
||
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
|
||
.venv/bin/python test_classify.py
|
||
.venv/bin/python test_geometry.py
|
||
python3 -m compileall .
|
||
```
|
||
|
||
Live HUD (requires D415):
|
||
|
||
```bash
|
||
./demo.sh
|
||
# open http://127.0.0.1:8080/
|
||
```
|
||
|
||
Live pipeline:
|
||
|
||
```bash
|
||
./run.sh --preview --no-mqtt --no-motor
|
||
# or full hardware once MQTT/routing configured in local config.yaml:
|
||
./run.sh --preview
|
||
```
|
||
|
||
## Configuration
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `config.example.yaml` | Committed safe defaults; **MQTT/motor/routing disabled** |
|
||
| `config.yaml` | Local overrides — **gitignored**; never commit credentials |
|
||
|
||
Optional MQTT (enable only locally):
|
||
|
||
```yaml
|
||
mqtt:
|
||
enabled: true
|
||
broker: "127.0.0.1"
|
||
port: 1883
|
||
user: "<your-user>"
|
||
password: "<your-password>"
|
||
```
|
||
|
||
CLI overrides: `--no-mqtt`, `--no-motor`, `--dry-route`, `--once`, `--preview`.
|
||
|
||
## Output schema (LOCK)
|
||
|
||
- Zone: `B` | `C` | `D`
|
||
- Dimensions mm: L×W×H
|
||
- `circle_ratio`
|
||
- Optional MQTT topics (when enabled): `vision/feedback/category`, `…/dimensions`, `…/circle_ratio`
|
||
- Optional JSONL: `logs/decisions.jsonl` (local, gitignored)
|
||
|
||
## Limitations
|
||
|
||
- Not connected to the web digital twin runtime.
|
||
- Requires calibrated intrinsics / belt height for accurate mm.
|
||
- Live demo needs a physical D415; CI hosts usually lack it.
|
||
- MQTT/servo/motor path is optional and site-specific.
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Check |
|
||
|---|---|
|
||
| No `/dev/video*` | USB3, `lsusb`, `v4l2-ctl --list-devices` |
|
||
| Depth empty | ffmpeg installed; correct `depth_device` |
|
||
| Wrong sizes | run `calibrate.py --length … --width …` |
|
||
| MQTT offline | expected when `mqtt.enabled: false` |
|
||
|
||
## Relation to web twin
|
||
|
||
Web (`src/domain/classifier.ts`) and CV (`classify.py`) implement the **same official bounds**. The public site uses a **digital sensor simulation**; this folder is the **hardware prototype** for future integration.
|