| @@ -1,5 +1,30 @@ | |||||
| # Changelog | # Changelog | ||||
| ## 1.0.7 — 2026-09-22 — Non-blocking panic32 rescue path | |||||
| - Reordenada toda acción real a validación O(1), señal, actualización en memoria | |||||
| y sólo después enqueue de observabilidad. Ningún `guard.log` ni syslog precede | |||||
| a SIGTERM/SIGKILL. | |||||
| - Añadido logger independiente con ring preasignado de 512 eventos y enqueue | |||||
| mediante `trylock`: cola llena/ocupada descarta telemetría y cuenta drops. | |||||
| - Movidos escritura, rotación y syslog al worker; añadidos `logging_delivery` y | |||||
| `logging_stall` con latencias separadas de guard log y syslog. | |||||
| - Movido el scan completo `/proc` a un worker independiente. El hilo de muestreo | |||||
| sólo solicita scans y conserva la última caché completa si el scanner se para. | |||||
| - Ampliada la caché global interna de 3 a 16 candidatos; sólo se siguen mostrando | |||||
| los tres primeros. Permite las 8 víctimas EMERGENCY y 12 floor sin nuevo scan. | |||||
| - En CRITICAL/EMERGENCY/FLOOR, una entrada antigua ya no se excluye sólo por | |||||
| edad: se usa como `stale_cache_revalidated` tras validar identidad y política. | |||||
| - La caché útil sobrevive EMERGENCY → RECOVERY → EMERGENCY hasta que un scan | |||||
| completo sano publique su reemplazo. | |||||
| - Añadidos timestamps `decision_mono`, `kill_attempt_mono`, `kill_return_mono` | |||||
| y métricas de validación, dispatch, syscall, enqueue, guard log y syslog. | |||||
| - Añadidos fixture/informe panic32 y fault injection para logger/syslog detenidos | |||||
| 180 s, log no disponible, scanner parado, cola ocupada, caché antigua y 8/12 | |||||
| víctimas sin scan; dispatch simulado permanece por debajo de 100 ms. | |||||
| - Conservados thresholds 1/2/2, gracias 5/1/0, budgets 4/8/12, ranking global, | |||||
| syslog local5, warmup y dry-run distribuido. No hubo despliegue ni señales reales. | |||||
| ## 1.0.6 — 2026-09-21 — Monotonic overrun handling and selective syslog | ## 1.0.6 — 2026-09-21 — Monotonic overrun handling and selective syslog | ||||
| - Las transiciones PRE-PRESSURE/PRESSURE se basan en tiempo monotónico real y | - Las transiciones PRE-PRESSURE/PRESSURE se basan en tiempo monotónico real y | ||||
| @@ -771,3 +771,21 @@ no deben depender de ticks exactos de un segundo. v1.0.6 conserva tiempo | |||||
| monotónico a través de `sampling_overrun`, registra saltos de umbral y añade | monotónico a través de `sampling_overrun`, registra saltos de umbral y añade | ||||
| syslog nativo selectivo. La evidencia y sus límites causales están en | syslog nativo selectivo. La evidencia y sus límites causales están en | ||||
| `INCIDENT_PAGEOUT_DEADMAN_AVOIDED_2026_09_21.md`. | `INCIDENT_PAGEOUT_DEADMAN_AVOIDED_2026_09_21.md`. | ||||
| ## 20. Addendum v1.0.7 — panic32 y ruta crítica no bloqueante | |||||
| El 2026-09-22 se observó un nuevo `pageout_deadman` con `freemem=424`. El dump | |||||
| mostró pageout detenido en `zio_wait()`/`dbuf_read()` sobre la ruta de swap zvol | |||||
| y numerosas esperas `page_create_*`. El guard había recuperado memoria dos veces, | |||||
| pero en una tercera emergencia transcurrieron unos 145 s entre la decisión | |||||
| SIGKILL y `signal_sent`; después recayó con la caché caducada. | |||||
| v1.0.7 elimina log, syslog y scan completo del hilo de rescate. La señal precede | |||||
| a todo enqueue; logger y scanner son workers separados, la cola descarta antes | |||||
| que esperar y la caché global interna crece a 16 candidatos. En rescate, una | |||||
| entrada antigua se revalida por PID/inicio/UUID/generación/política en vez de | |||||
| descartarse sólo por edad. Se conservan thresholds, ranking y budgets. | |||||
| La evidencia confirmada, las hipótesis aún abiertas y el fixture están en | |||||
| `INCIDENT_PAGEOUT_DEADMAN_PANIC32_2026_09_22.md` y | |||||
| `tests/fixtures/panic32_daemon_stall_2026_09_22.csv`. | |||||
| @@ -0,0 +1,100 @@ | |||||
| # panic32: `pageout_deadman` y stall del daemon — 2026-09-22 | |||||
| ## Resumen | |||||
| El nodo terminó con: | |||||
| ```text | |||||
| pageout_deadman: stuck pushing the same page for 90 seconds (freemem is 424) | |||||
| ``` | |||||
| Con páginas de 4096 bytes, `freemem` era aproximadamente 1,66 MiB. Antes del | |||||
| panic, `o6h-memory-guard` había resuelto dos emergencias y recuperado memoria, | |||||
| pero durante una tercera quedó sin progresar unos 145 segundos entre la decisión | |||||
| de escalar y el envío efectivo de SIGKILL. Ese intervalo supera los 90 segundos | |||||
| del deadman y motivó el rediseño de la ruta crítica en v1.0.7. | |||||
| ## Evidencia del kernel | |||||
| ```text | |||||
| Page Summary Pages MB %Tot | |||||
| Kernel 2022564 7900 24% | |||||
| Boot pages 90359 352 1% | |||||
| ZFS File Data 507501 1982 6% | |||||
| Anon 5732179 22391 68% | |||||
| Exec and libs 8299 32 0% | |||||
| Page cache 14675 57 0% | |||||
| Free (cachelist) 32 0 0% | |||||
| Free (freelist) 394 1 0% | |||||
| Total 8376003 32718 | |||||
| Physical 8376002 32718 | |||||
| ``` | |||||
| El hilo de pageout estaba detenido en la ruta: | |||||
| ```text | |||||
| swtch → cv_wait → zio_wait → dbuf_read → dmu_buf_will_dirty_impl | |||||
| → dmu_write → zvol_strategy → bdev_strategy → spec_pageio | |||||
| → swap_putapage → swap_putpage → pageout | |||||
| ``` | |||||
| El dump contenía además numerosas esperas en `page_create_throttle`, | |||||
| `page_create_wait`, `page_create_va`, `segkmem_page_create` y | |||||
| `segkmem_zio_alloc`. Es compatible con agotamiento físico mientras pageout | |||||
| intenta escribir swap sobre un zvol y la propia ruta ZFS/DMU/ZIO no progresa. | |||||
| ## Cronología del guard | |||||
| - 15:18:20: EMERGENCY, unas 48.000 páginas libres; SIGTERM a `clamd` de | |||||
| `servidor.consultaerte.com`. | |||||
| - 15:18:26–15:18:29: nueva EMERGENCY; fallback a `clamd` de `sailandfun.bz2` | |||||
| y SIGKILL. | |||||
| - 15:18:30: 305.511 páginas libres, aproximadamente 1,17 GiB; recuperación real. | |||||
| - 15:18:40: nueva EMERGENCY; candidato `clamd` de `03.base0.es`. | |||||
| - 15:18:49: SIGTERM real. | |||||
| - 15:18:51: decisión de escalar a SIGKILL. | |||||
| - 15:21:16: `signal_sent SIGKILL`, unos 145 segundos después, junto con | |||||
| `sampling_overrun late_seconds≈145.076`. | |||||
| - 15:21:17: recuperación parcial hasta unas 148.619 páginas libres. | |||||
| - 15:21:25: recaída a EMERGENCY en `minfree=49077`; la caché tenía unos | |||||
| 167.582 ms registrados, es decir, unos 167,6 segundos, y se bloqueó la | |||||
| acción con `no_valid_cached_candidate`. | |||||
| - Finalmente ocurrió `pageout_deadman`. | |||||
| El fixture `tests/fixtures/panic32_daemon_stall_2026_09_22.csv` conserva esta | |||||
| secuencia sin ejecutar señales reales. | |||||
| ## Confirmado | |||||
| - El guard dejó de progresar aproximadamente 145 segundos. | |||||
| - El SIGKILL se retrasó ese mismo orden de tiempo. | |||||
| - El retraso superó el umbral de 90 segundos del deadman. | |||||
| - Hubo recuperaciones cuantificables tras señales anteriores y tras el kill tardío. | |||||
| - La caché top-3 quedó demasiado antigua y ya no proporcionó candidato en la recaída. | |||||
| - El nodo terminó en `pageout_deadman` con sólo 424 páginas libres. | |||||
| ## Hipótesis pendiente | |||||
| - `syslog()` pudo bloquearse. | |||||
| - La escritura o rotación de `guard.log` pudo bloquearse. | |||||
| - La validación puntual de `/proc`, kstat u otra syscall pudo bloquearse. | |||||
| - No hay evidencia suficiente para atribuir causalidad específica a syslog. | |||||
| v1.0.7 instrumenta por separado validación, entrada y retorno de la syscall de | |||||
| señal, encolado, escritura de log y syslog. Esa instrumentación permitirá | |||||
| acotar un incidente futuro sin convertir la hipótesis en conclusión. | |||||
| ## Corrección de diseño | |||||
| En acciones reales, v1.0.7 impone: | |||||
| ```text | |||||
| validación O(1) → señal → counters en memoria → enqueue no bloqueante | |||||
| ``` | |||||
| Todo `guard.log` y syslog se ejecuta en un logger independiente. Un scanner | |||||
| independiente publica una caché fija de 16 candidatos; la ruta de rescate no | |||||
| espera al scanner, admite candidatos antiguos tras revalidación completa O(1) | |||||
| y conserva la caché a través de RECOVERY. Si la cola de log está llena u ocupada, | |||||
| se pierde telemetría y se incrementa el contador de descartes: salvar el host | |||||
| tiene prioridad sobre conservar el log. | |||||
| @@ -2,9 +2,9 @@ | |||||
| CC = gcc | CC = gcc | ||||
| CPPFLAGS = -D__EXTENSIONS__ -D_POSIX_C_SOURCE=200809L -Isrc | CPPFLAGS = -D__EXTENSIONS__ -D_POSIX_C_SOURCE=200809L -Isrc | ||||
| CFLAGS = -std=c99 -m64 -O2 -Wall -Wextra -Werror -Wformat=2 | CFLAGS = -std=c99 -m64 -O2 -Wall -Wextra -Werror -Wformat=2 | ||||
| LDLIBS = -lkstat -lrt | |||||
| LDLIBS = -lkstat -lrt -lpthread | |||||
| SOURCES = src/main.c src/core.c src/config.c src/support.c src/illumos.c | SOURCES = src/main.c src/core.c src/config.c src/support.c src/illumos.c | ||||
| TESTFLAGS = -std=c99 -O1 -g -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Isrc -Wall -Wextra -Werror -Wformat=2 -pedantic | |||||
| TESTFLAGS = -std=c99 -O1 -g -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Isrc -Wall -Wextra -Werror -Wformat=2 -pedantic -pthread | |||||
| .PHONY: all native-check test test-distribution test-sanitize portability clean | .PHONY: all native-check test test-distribution test-sanitize portability clean | ||||
| all: o6h-memory-guard | all: o6h-memory-guard | ||||
| @@ -18,7 +18,7 @@ o6h-memory-guard: $(SOURCES) src/guard.h | native-check | |||||
| test-guard: tests/test_guard.c tests/fixtures/pageout_deadman_real_incident_2026_09.csv tests/fixtures/successful_pageout_deadman_avoidance_2026_09_21.csv src/core.c src/config.c src/guard.h | test-guard: tests/test_guard.c tests/fixtures/pageout_deadman_real_incident_2026_09.csv tests/fixtures/successful_pageout_deadman_avoidance_2026_09_21.csv src/core.c src/config.c src/guard.h | ||||
| $(CC) $(TESTFLAGS) -o $@ tests/test_guard.c src/core.c src/config.c | $(CC) $(TESTFLAGS) -o $@ tests/test_guard.c src/core.c src/config.c | ||||
| test-runtime: tests/test_runtime.c tests/fixtures/global_victim_ranking_smartos_2026_09.csv src/main.c src/core.c src/config.c src/guard.h | |||||
| test-runtime: tests/test_runtime.c tests/fixtures/global_victim_ranking_smartos_2026_09.csv tests/fixtures/panic32_daemon_stall_2026_09_22.csv src/main.c src/core.c src/config.c src/guard.h | |||||
| $(CC) $(TESTFLAGS) -o $@ tests/test_runtime.c src/core.c src/config.c | $(CC) $(TESTFLAGS) -o $@ tests/test_runtime.c src/core.c src/config.c | ||||
| test-distribution: tests/test_distribution.sh smf/o6h-memory-guard.xml etc/o6h-memory-guard.conf src/main.c | test-distribution: tests/test_distribution.sh smf/o6h-memory-guard.xml etc/o6h-memory-guard.conf src/main.c | ||||
| @@ -1,4 +1,4 @@ | |||||
| # o6h-memory-guard 1.0.6 | |||||
| # o6h-memory-guard 1.0.7 | |||||
| Primera implementación para SmartOS global zone: daemon C de 64 bits, libkstat, | Primera implementación para SmartOS global zone: daemon C de 64 bits, libkstat, | ||||
| /proc, identidad UUID + generación de zona + PID + instante de creación, | /proc, identidad UUID + generación de zona + PID + instante de creación, | ||||
| @@ -34,6 +34,8 @@ Activación real | |||||
| con v1.0.2, endurecimiento de v1.0.3 y corrección del ranking global en v1.0.4. | con v1.0.2, endurecimiento de v1.0.3 y corrección del ranking global en v1.0.4. | ||||
| - `INCIDENT_PAGEOUT_DEADMAN_AVOIDED_2026_09_21.md`: evidencia de campo de un | - `INCIDENT_PAGEOUT_DEADMAN_AVOIDED_2026_09_21.md`: evidencia de campo de un | ||||
| episodio en el que la acción real liberó memoria antes de un nuevo panic. | episodio en el que la acción real liberó memoria antes de un nuevo panic. | ||||
| - `INCIDENT_PAGEOUT_DEADMAN_PANIC32_2026_09_22.md`: análisis del panic32, el | |||||
| stall de 145 s y el rediseño no bloqueante de v1.0.7. | |||||
| - `CHANGELOG.md`: evolución de versiones y revisión documental del paquete. | - `CHANGELOG.md`: evolución de versiones y revisión documental del paquete. | ||||
| El checkpoint es cronológicamente posterior a `VALIDATION.md`; por ello amplía | El checkpoint es cronológicamente posterior a `VALIDATION.md`; por ello amplía | ||||
| @@ -47,12 +49,12 @@ Si no hay herramientas de desarrollo, compilar en una **zona nativa SmartOS** | |||||
| de la misma plataforma (no LX), ejecutar allí `gmake test`, y copiar el binario | de la misma plataforma (no LX), ejecutar allí `gmake test`, y copiar el binario | ||||
| junto a este árbol al host. No instalar un compilador en un nodo bajo presión. | junto a este árbol al host. No instalar un compilador en un nodo bajo presión. | ||||
| Copiar el archivo distribuido a `/var/tmp/o6h-memory-guard-1.0.6.tar.gz`: | |||||
| Copiar el archivo distribuido a `/var/tmp/o6h-memory-guard-1.0.7.tar.gz`: | |||||
| ```sh | ```sh | ||||
| mkdir -p /opt/custom/src | mkdir -p /opt/custom/src | ||||
| cd /opt/custom/src | cd /opt/custom/src | ||||
| gzip -dc /var/tmp/o6h-memory-guard-1.0.6.tar.gz | tar -xf - | |||||
| gzip -dc /var/tmp/o6h-memory-guard-1.0.7.tar.gz | tar -xf - | |||||
| cd /opt/custom/src/o6h-memory-guard | cd /opt/custom/src/o6h-memory-guard | ||||
| gmake CC=gcc | gmake CC=gcc | ||||
| gmake test CC=gcc | gmake test CC=gcc | ||||
| @@ -229,8 +231,9 @@ antes de usarlo como mecanismo de rescate. No hay garantía de evitar un panic. | |||||
| ## Atribución y protecciones | ## Atribución y protecciones | ||||
| Cada scan completo recorre todos los procesos elegibles de todas las zonas | Cada scan completo recorre todos los procesos elegibles de todas las zonas | ||||
| `allow` y construye un único top‑3 global; no existe una preselección de la mejor | |||||
| zona. El score en KiB equivalentes es: | |||||
| `allow` y construye un ranking global; no existe una preselección de la mejor | |||||
| zona. La caché interna conserva 16 candidatos y el log muestra sólo los tres | |||||
| primeros para limitar ruido. El score en KiB equivalentes es: | |||||
| ```text | ```text | ||||
| total_score = RSS + 4*max(d5,0) + 2*max(d15,0) + max(d60,0) | total_score = RSS + 4*max(d5,0) + 2*max(d15,0) + max(d60,0) | ||||
| @@ -241,9 +244,10 @@ total_score = RSS + 4*max(d5,0) + 2*max(d15,0) + max(d60,0) | |||||
| d5/16, d15/32 y d60/64, pero queda limitado a `process_rss/4`. Así una zona | d5/16, d15/32 y d60/64, pero queda limitado a `process_rss/4`. Así una zona | ||||
| enorme aporta contexto sin poder monopolizar el ranking; un proceso estable de | enorme aporta contexto sin poder monopolizar el ranking; un proceso estable de | ||||
| 1,3 GiB supera normalmente a uno de 250 MiB, mientras un proceso menor con | 1,3 GiB supera normalmente a uno de 250 MiB, mientras un proceso menor con | ||||
| crecimiento explosivo todavía puede ascender. Se usa el mismo top‑3 balanceado | |||||
| en todos los estados para conservar la ruta CRITICAL/EMERGENCY/FLOOR O(1) y no | |||||
| expulsar de la caché procesos grandes necesarios para recuperación inmediata. | |||||
| crecimiento explosivo todavía puede ascender. Se usa el mismo ranking balanceado | |||||
| en todos los estados. La caché fija de 16 entradas permite consumir los budgets | |||||
| 8/12 sin nuevo scan y conserva la ruta CRITICAL/EMERGENCY/FLOOR O(Ncache), con | |||||
| N fijo y pequeño. | |||||
| Empates se resuelven por RSS y después PID. El log expone `rss_component`, | Empates se resuelven por RSS y después PID. El log expone `rss_component`, | ||||
| `growth5_component`, `growth15_component`, `growth60_component`, | `growth5_component`, `growth15_component`, `growth60_component`, | ||||
| `zone_component` y `total_score`. | `zone_component` y `total_score`. | ||||
| @@ -286,36 +290,39 @@ grupos. La integración de esta ruta todavía necesita validación nativa de lab | |||||
| En dry-run nunca se abre `ctl`: se revalida con lecturas, se escribe | En dry-run nunca se abre `ctl`: se revalida con lecturas, se escribe | ||||
| `WOULD_ACTION=SIGTERM/SIGKILL` y se actualiza sólo el estado simulado. Los | `WOULD_ACTION=SIGTERM/SIGKILL` y se actualiza sólo el estado simulado. Los | ||||
| presupuestos evitan repetir infinitamente el mismo sacrificio hipotético. | presupuestos evitan repetir infinitamente el mismo sacrificio hipotético. | ||||
| En armed se escribe `ACTION_INTENT` antes y `action_result=...` después | |||||
| (el campo real es `event=action_result result=sent|failed`). Los errores de | |||||
| escritura de señal consumen presupuesto para evitar reintentos ambiguos. | |||||
| En armed no se escribe nada antes de la operación real: se valida, se intenta la | |||||
| señal, se actualizan counters en memoria y sólo entonces se encolan | |||||
| `action_decision`, `action_result` y, si hubo éxito, `signal_sent`. Los errores | |||||
| de escritura de señal consumen presupuesto para evitar reintentos ambiguos. | |||||
| ## Recursos y fallos | ## Recursos y fallos | ||||
| Almacenamiento de trabajo fijo para 4096 procesos y 256 zonas, prefaulted por | |||||
| inicialización; el tamaño se registra al arrancar. No hay malloc de la aplicación | |||||
| Almacenamiento fijo para 4096 procesos, 256 zonas, 16 candidatos y una cola de | |||||
| 512 eventos, reservado al arrancar; el tamaño se registra. No hay malloc de la aplicación | |||||
| en la ruta de acción, ni `system`, `popen`, `exec`, `vmadm`, `ps` o `vmstat` en | en la ruta de acción, ni `system`, `popen`, `exec`, `vmadm`, `ps` o `vmstat` en | ||||
| el daemon. libkstat/libc sí pueden reservar memoria internamente al actualizar | el daemon. libkstat/libc sí pueden reservar memoria internamente al actualizar | ||||
| su cadena. No se promete cero asignaciones ni tiempo real estricto. | su cadena. No se promete cero asignaciones ni tiempo real estricto. | ||||
| El recorrido /proc tiene presupuesto cooperativo de 200 ms (hasta 500 ms por | El recorrido /proc tiene presupuesto cooperativo de 200 ms (hasta 500 ms por | ||||
| configuración). Un scan incompleto queda registrado pero no sustituye la última | |||||
| caché completa. NORMAL actualiza cada 5 s; PRE-PRESSURE/PRESSURE, cada segundo. | |||||
| CRITICAL/EMERGENCY/FLOOR no ejecutan scans completos: su ruta es lectura kstat, | |||||
| selección O(1) entre tres entradas, validación barata de identidad y decisión. | |||||
| La edad máxima predeterminada es 15 s. Si la caché es vieja o no tiene candidato | |||||
| válido, la acción se bloquea de forma segura; no se intenta descubrir uno bajo | |||||
| presión extrema. Una llamada del kernel durante un scan previo todavía puede | |||||
| exceder el presupuesto; queda registrada como overrun sin borrar la buena caché. | |||||
| configuración) y se ejecuta en un scanner independiente. Un scan incompleto o | |||||
| bloqueado no sustituye la última caché completa ni detiene el muestreo. NORMAL | |||||
| solicita actualización cada 5 s; PRE-PRESSURE/PRESSURE, cada segundo. | |||||
| CRITICAL/EMERGENCY/FLOOR nunca esperan al scanner: recorren como máximo 16 | |||||
| entradas precomputadas y revalidan sólo el proceso elegido. Los 15 s de frescura | |||||
| siguen aplicándose en NORMAL/PRE/PRESSURE. En rescate, una entrada más antigua | |||||
| puede usarse como `stale_cache_revalidated` si PID, inicio, UUID, generación, | |||||
| regla `allow`, nombre y protecciones todavía coinciden. La caché sobrevive a | |||||
| EMERGENCY → RECOVERY → EMERGENCY y sólo un scan completo sano la reemplaza. | |||||
| Muestras obligatorias ausentes/invalidas o demasiado lentas inhiben señales; | Muestras obligatorias ausentes/invalidas o demasiado lentas inhiben señales; | ||||
| ARC/swap/pageout opcionales ausentes se marcan con `*_ok=0` y no bloquean por sí | ARC/swap/pageout opcionales ausentes se marcan con `*_ok=0` y no bloquean por sí | ||||
| solos. Un fallo de | |||||
| log inhibe las acciones hasta reiniciar y también se anuncia en stderr/SMF. | |||||
| Los logs se escriben síncronamente al archivo sin fsync por muestra; el I/O o | |||||
| ZFS bloqueado puede retrasar este daemon. Tampoco garantiza que las últimas | |||||
| líneas sobrevivan a pérdida de alimentación o panic. Es persistencia en disco, | |||||
| no un journal de transacciones de señales. No se cambia swap, ARC ni caps de zona. | |||||
| solos. Un fallo o stall del log ya no inhibe acciones. El hilo principal encola | |||||
| con `pthread_mutex_trylock`; nunca espera al logger. Si la cola está ocupada o | |||||
| llena, descarta telemetría y acumula `dropped_events`. El worker realiza escritura, | |||||
| rotación y syslog, y genera posteriormente `logging_delivery` o `logging_stall` | |||||
| con latencias por canal. No hay `fsync`; las últimas líneas pueden perderse por | |||||
| drop, alimentación o panic. Es observabilidad best-effort, no un journal de | |||||
| transacciones de señales. No se cambia swap, ARC ni caps de zona. | |||||
| La enumeración física admite hasta 64 dispositivos swap con almacenamiento fijo; | La enumeración física admite hasta 64 dispositivos swap con almacenamiento fijo; | ||||
| si el nodo supera ese límite, `swap_physical_used_ok=0` y esa señal diagnóstica se | si el nodo supera ese límite, `swap_physical_used_ok=0` y esa señal diagnóstica se | ||||
| omite sin autorizar acciones. | omite sin autorizar acciones. | ||||
| @@ -353,7 +360,8 @@ resultado del último scan, componentes del score y cualquier override de presup | |||||
| ### Syslog nativo selectivo | ### Syslog nativo selectivo | ||||
| `guard.log` sigue siendo el registro completo. Con `SYSLOG_ENABLED yes` se | |||||
| `guard.log` sigue siendo el registro detallado principal; bajo saturación puede | |||||
| perder eventos antes que bloquear una señal. Con `SYSLOG_ENABLED yes` se | |||||
| envía además un subconjunto operativo mediante `openlog(3C)`, `syslog(3C)` y | envía además un subconjunto operativo mediante `openlog(3C)`, `syslog(3C)` y | ||||
| `closelog(3C)`, con ident `o6h-memory-guard` y facility configurable | `closelog(3C)`, con ident `o6h-memory-guard` y facility configurable | ||||
| `local0`…`local7` (`local5` por defecto). No se invoca `logger`, shell ni un | `local0`…`local7` (`local5` por defecto). No se invoca `logger`, shell ni un | ||||
| @@ -372,9 +380,14 @@ También se envían `state_transition`, `signal_sent`, `action_blocked` e | |||||
| confirmado el envío real e incluye estado, señal, UUID, alias, PID, nombre, RSS | confirmado el envío real e incluye estado, señal, UUID, alias, PID, nombre, RSS | ||||
| y memoria libre; nunca existe en dry-run. `sample`, `delta`, `candidate_cache`, | y memoria libre; nunca existe en dry-run. `sample`, `delta`, `candidate_cache`, | ||||
| `top_process`, `proc_scan` y `zone` no se duplican en syslog. Un fallo al enviar | `top_process`, `proc_scan` y `zone` no se duplican en syslog. Un fallo al enviar | ||||
| a syslog se ignora para la decisión: no revierte ni cambia el resultado de una | |||||
| señal y las notificaciones de estado crítico se difieren hasta después de la | |||||
| ruta O(1) de acción. | |||||
| a syslog se ignora para la decisión: syslog se ejecuta exclusivamente en el | |||||
| worker y no puede preceder ni cambiar el resultado de una señal. | |||||
| Los eventos de acción incluyen `decision_mono`, `kill_attempt_mono`, | |||||
| `kill_return_mono`, `dispatch_latency_ms`, `candidate_validate_latency_ms`, | |||||
| `kill_latency_ms` y `log_enqueue_latency_ms`. El worker mide además | |||||
| `guard_log_write_latency_ms` y `syslog_latency_ms`. Así se distingue un retraso | |||||
| de validación/señal de un stall posterior del canal de observabilidad. | |||||
| Antes de asumir un archivo de destino, inspeccionar la configuración local: | Antes de asumir un archivo de destino, inspeccionar la configuración local: | ||||
| @@ -388,7 +401,7 @@ Alloy/Loki puede configurarse fuera del guard, pero esta entrega no lo hace. | |||||
| ## Modelo de activación real (operación futura; no ejecutada en esta entrega) | ## Modelo de activación real (operación futura; no ejecutada en esta entrega) | ||||
| v1.0.6 debe compilarse y observarse primero en SmartOS con `ACTION_MODE dry-run`. | |||||
| v1.0.7 debe compilarse y observarse primero en SmartOS con `ACTION_MODE dry-run`. | |||||
| El manifiesto ya no contiene una segunda puerta oculta. Cuando exista aprobación | El manifiesto ya no contiene una segunda puerta oculta. Cuando exista aprobación | ||||
| operativa explícita, el único cambio de modo es editar la configuración: | operativa explícita, el único cambio de modo es editar la configuración: | ||||
| @@ -1,25 +1,26 @@ | |||||
| 887569535302f7deba4d7c6ffb2103f1fb90bd96a40280240ff238eddc2e3a6f ./.gitignore | 887569535302f7deba4d7c6ffb2103f1fb90bd96a40280240ff238eddc2e3a6f ./.gitignore | ||||
| 61974bf5d994ca3945d29721ec9d23458dc22263aa90e92589105de0e8dfd3ad ./CHANGELOG.md | |||||
| b9c80b9523c0d371302907d50bc27f3a207f1990c60b4208c310713f8f859aa3 ./CHECKPOINT.md | |||||
| 835934c3bdd868f0b06cc399081952279c68674677513d6edf63b2c323befee8 ./CHANGELOG.md | |||||
| 23f26b11ceec2765e71bae4d193b6a844dfe24081a20b6212037953745f6bb87 ./CHECKPOINT.md | |||||
| a0f96300e751d08ceeef1360325ed54008b703fa6bd07df9d5368674e580f113 ./INCIDENT_PAGEOUT_DEADMAN_2026_09.md | a0f96300e751d08ceeef1360325ed54008b703fa6bd07df9d5368674e580f113 ./INCIDENT_PAGEOUT_DEADMAN_2026_09.md | ||||
| fb6b65a2655ccd808d102fd917b4866cbe939f24fac4e4aeef226a6aeda6d588 ./INCIDENT_PAGEOUT_DEADMAN_AVOIDED_2026_09_21.md | fb6b65a2655ccd808d102fd917b4866cbe939f24fac4e4aeef226a6aeda6d588 ./INCIDENT_PAGEOUT_DEADMAN_AVOIDED_2026_09_21.md | ||||
| a4c564791ca7aeedbebbd68589e72d3a5af24ca494bb46661a55d0aaa289ee43 ./Makefile | |||||
| 26ef74ede1362441f02c27dbcce446c11797ae3457a872df220b0f808d50f93a ./README.md | |||||
| 6ffb6af2a439133d6e9587723fd0c7ff39273dd32747d7fc758ec66609897958 ./VALIDATION.md | |||||
| e6142ab575d390a5744e18ff4a73f023b1cdac4b274e9597f71a111a82354c49 ./etc/o6h-memory-guard.conf | |||||
| 0f18d6dd3e0d027c1e8dbfce05ce6ac3beb70f3f66d1a77a8d0c5e0f4ae2a039 ./INCIDENT_PAGEOUT_DEADMAN_PANIC32_2026_09_22.md | |||||
| 5cdc6a11adf10b5ee9042cf87b8c8e6a5f94a7803be9b5b2f92b57c76c66043c ./Makefile | |||||
| 442e22442b19e1f2576b25bed28f974281ff7b520607d7108fb2e9ca2648338d ./README.md | |||||
| e648242f049aeebf657fc64ce367494abffc9aa54f2833edd234b5dc07d87131 ./VALIDATION.md | |||||
| ca37b8d46de1a5ee432fdda1dd7f8883d4e46e9cc646fb1ff9ad9b6caa7bbeb6 ./etc/o6h-memory-guard.conf | |||||
| f6565d0e1ece0a08721c9893f9ed071d2ab0a2818fa93ed0a14e0050d358ce99 ./install.sh | f6565d0e1ece0a08721c9893f9ed071d2ab0a2818fa93ed0a14e0050d358ce99 ./install.sh | ||||
| 24c1b9bc39fda71d459d6fad6a8d889b0d9b92be89d303677643b3d0b93caf4b ./logs/distribution-tests.log | 24c1b9bc39fda71d459d6fad6a8d889b0d9b92be89d303677643b3d0b93caf4b ./logs/distribution-tests.log | ||||
| 50626dec1924e19b4ca92eec30416fb39c3fe7cff28d98a8e120ccf6247d2834 ./logs/example.log | 50626dec1924e19b4ca92eec30416fb39c3fe7cff28d98a8e120ccf6247d2834 ./logs/example.log | ||||
| 812482297ae6dfb2a0881998781459ed77288ef073f202a18f22055437518fa6 ./logs/policy-tests.log | 812482297ae6dfb2a0881998781459ed77288ef073f202a18f22055437518fa6 ./logs/policy-tests.log | ||||
| d96eedca75907852ca6852310b619cc75a9837d15cbad46f8940ad2a54be6c8a ./logs/portability.log | |||||
| 5388d1968ee02f90654aa9e8bb9903f9c6f93018373f14b98a88414bd6b60182 ./logs/runtime-tests.log | |||||
| 670e23e9b1866de59a488a024d16c4e202ae1824315df111e4f7c9449579ebab ./logs/sanitizers.log | |||||
| 451357017446eea44680b0fa48c69e091a0f1062fafde97d38405fc8b01f4d5a ./logs/portability.log | |||||
| 1963ed6d329842c2a6cedf4780074e0db28161e1c510e36afd532edf286b5930 ./logs/runtime-tests.log | |||||
| 9b3ffe409f024768277d449c3df456cc2c35a21ce8f42e6653886cb758229e88 ./logs/sanitizers.log | |||||
| 55cf49840cf1d22e4e02bedf579a10207856a0a5b40c86ae2cf6ee2f00d3f972 ./smf/o6h-memory-guard.xml | 55cf49840cf1d22e4e02bedf579a10207856a0a5b40c86ae2cf6ee2f00d3f972 ./smf/o6h-memory-guard.xml | ||||
| 99e1fb6a90a7bd2c1002ad09320cd3f21c2deb2499212eff2baddc14f3bdf0e6 ./src/config.c | 99e1fb6a90a7bd2c1002ad09320cd3f21c2deb2499212eff2baddc14f3bdf0e6 ./src/config.c | ||||
| 659da2facd27577c763484694a2ff62eef42aea48ebc496b878853b0164b6539 ./src/core.c | 659da2facd27577c763484694a2ff62eef42aea48ebc496b878853b0164b6539 ./src/core.c | ||||
| 3fc9eff261b150e5595f74dc15cc5aac0250dcb11611049819c3e9177a815747 ./src/guard.h | |||||
| 32ab75bb6a1ff2bbbeb6a1ea658abcb96a03fcc06246f38d019faec7fdc184b8 ./src/illumos.c | |||||
| 9280f8f2b47917752ea923d28d64fbb45c9c761c173e0243582de544ae5287e4 ./src/main.c | |||||
| 727c34a27081c3fedd3180c8e6f932a3cbefeded81906012d04bdfbaf02a9421 ./src/guard.h | |||||
| 71ed7fe84db58b8bb7577fff7eba564b31c38db3d12082df641faec0e7465d09 ./src/illumos.c | |||||
| a9652dddb0aa118b58f462434c3acba425e29bc89ef239589fd91252d2586577 ./src/main.c | |||||
| 1f09afd342ba86580a21f6c04a030d9ea2551a60b669e257295fbf4f43b2fad7 ./src/support.c | 1f09afd342ba86580a21f6c04a030d9ea2551a60b669e257295fbf4f43b2fad7 ./src/support.c | ||||
| f754e71d91d65403ab8e84e9701cfbb89bdd91e13766e668445e689e6890b369 ./tests/api-sketch/kstat.h | f754e71d91d65403ab8e84e9701cfbb89bdd91e13766e668445e689e6890b369 ./tests/api-sketch/kstat.h | ||||
| f0d06353d2e326c7bdce776bee6f0a8606cda50d5de957dfdd586b6848fca8da ./tests/api-sketch/procfs.h | f0d06353d2e326c7bdce776bee6f0a8606cda50d5de957dfdd586b6848fca8da ./tests/api-sketch/procfs.h | ||||
| @@ -28,7 +29,8 @@ f0d06353d2e326c7bdce776bee6f0a8606cda50d5de957dfdd586b6848fca8da ./tests/api-sk | |||||
| 9b99e272008ad00ad0abb366e1e8720dd6c51fc050821c5d1d878eae5719c22b ./tests/api-sketch/zone.h | 9b99e272008ad00ad0abb366e1e8720dd6c51fc050821c5d1d878eae5719c22b ./tests/api-sketch/zone.h | ||||
| 4aa08ded8e98cf25d2a8762543de7a9273f0ebf00b4c7e5b359603ceda46eb56 ./tests/fixtures/global_victim_ranking_smartos_2026_09.csv | 4aa08ded8e98cf25d2a8762543de7a9273f0ebf00b4c7e5b359603ceda46eb56 ./tests/fixtures/global_victim_ranking_smartos_2026_09.csv | ||||
| c5d9d7fc7a39354474d30cd316ea9f122205c2b38e57f927f4a08306b8a62c81 ./tests/fixtures/pageout_deadman_real_incident_2026_09.csv | c5d9d7fc7a39354474d30cd316ea9f122205c2b38e57f927f4a08306b8a62c81 ./tests/fixtures/pageout_deadman_real_incident_2026_09.csv | ||||
| 025983b24045e24a96532c933715b8760edd8c0b0e11b9f0e019481c1c52a50d ./tests/fixtures/panic32_daemon_stall_2026_09_22.csv | |||||
| facc4580ffadef29cdce84bf1226a696cba3880bbbec80161beaceb2752bfd77 ./tests/fixtures/successful_pageout_deadman_avoidance_2026_09_21.csv | facc4580ffadef29cdce84bf1226a696cba3880bbbec80161beaceb2752bfd77 ./tests/fixtures/successful_pageout_deadman_avoidance_2026_09_21.csv | ||||
| 9866c935703149aad042e84aef90d4c3a3083cf00245a809380234da9f0c1e8f ./tests/test_distribution.sh | |||||
| acee8fe2770c612a051ac92022e23f310afc557e23d0e6193f728a2ecd9729ad ./tests/test_distribution.sh | |||||
| 7c0cfe5fc6719d27258bb87b9e68388be7b580d7031a5ed6deac34ddb3050f25 ./tests/test_guard.c | 7c0cfe5fc6719d27258bb87b9e68388be7b580d7031a5ed6deac34ddb3050f25 ./tests/test_guard.c | ||||
| 1da113327176396dca621664ea339f101d990f0412f41a3859376a56a81a68b4 ./tests/test_runtime.c | |||||
| 068f1c641440c285211d70dfb61eff492411c2a549b8c692c3d5f71358d90ce8 ./tests/test_runtime.c | |||||
| @@ -1,103 +1,101 @@ | |||||
| # Validación de la entrega v1.0.6 | |||||
| Fecha: 2026-09-21. Esta entrega se preparó como código fuente. **No se desplegó, | |||||
| no modificó el nodo SmartOS, no activó `armed` y no envió señales reales.** La | |||||
| configuración distribuida mantiene `ACTION_MODE dry-run`, no contiene UUID | |||||
| `allow` y la instancia SMF sigue deshabilitada de fábrica. | |||||
| ## Evidencia de campo preservada | |||||
| La telemetría y ejecución nativas de versiones anteriores ya habían validado | |||||
| SMF, muestreo, kstat, swap físico/virtual, pageout anónimo, scans y caché. El | |||||
| episodio del 21 de septiembre añadió una observación de rescate real: | |||||
| - 11:31:09: 79.960 páginas libres, bajo `lotsfree=130875`. | |||||
| - 11:31:14: EMERGENCY a `minfree=49077`, `nscan=4171921`. | |||||
| - 11:31:17: SIGTERM real a `clamd`, PID 17850, RSS aproximado 1,32 GiB. | |||||
| - 11:31:23: SIGKILL real tras persistir la emergencia. | |||||
| - 11:31:24: RECOVERY con unas 367.019 páginas libres; después NORMAL. | |||||
| - Recuperación aproximada: 1,21 GiB. No apareció `pageout_deadman`. | |||||
| Hubo overruns aproximados de 0,918, 2,331, 1,890 y 4,459 s. La correlación entre | |||||
| señal, terminación y recuperación es evidencia fuerte de prevención aparente, | |||||
| pero no prueba causalidad absoluta. El detalle y sus límites están en | |||||
| `INCIDENT_PAGEOUT_DEADMAN_AVOIDED_2026_09_21.md`. | |||||
| ## Cambios verificados en v1.0.6 | |||||
| 1. PRE-PRESSURE/PRESSURE usan duración monotónica real, no número de muestras. | |||||
| La secuencia t0 normal, t+1,8 bajo `lotsfree`, t+4,2 aún baja alcanza | |||||
| PRESSURE sin reiniciarse por los ticks omitidos. | |||||
| 2. Una muestra tardía que salta desde encima de `desfree` a `free <= desfree` | |||||
| entra en CRITICAL. `free <= minfree` entra inmediatamente en EMERGENCY. | |||||
| 3. Los saltos registran `reason=threshold_jump` y `skipped_states`; el evento | |||||
| `sampling_overrun` incluye `late_seconds`, `state` y `free_pages`. | |||||
| 4. Sólo un error real de muestra/reloj reinicia las confirmaciones. Los overruns | |||||
| no borran la caché ni ejecutan un scan `/proc` en la ruta de rescate. | |||||
| 5. Swap físico y pageout siguen siendo corroboración bajo `lotsfree`, nunca un | |||||
| trigger único. Umbrales, ranking, presupuestos 4/4/8/12 y TERM/KILL no cambian. | |||||
| 6. Syslog usa sólo `openlog/syslog/closelog`, ident `o6h-memory-guard`, facility | |||||
| validada `local0`…`local7` y `local5` por defecto. | |||||
| 7. `guard.log` conserva todos los eventos. Syslog recibe start, transiciones, | |||||
| señales confirmadas, bloqueos, recovery completo y errores; nunca el flujo | |||||
| de `sample`, `delta`, cache, top, scan o zona. | |||||
| 8. `signal_sent` sólo se genera después del éxito de una señal real y contiene | |||||
| estado, señal, UUID, alias, PID, nombre, RSS y memoria libre. Dry-run no lo | |||||
| genera. Un fallo simulado de syslog no cambia ni bloquea la acción. | |||||
| ## Pruebas y análisis ejecutados | |||||
| - `gmake test`: política, runtime simulado y contrato de distribución. | |||||
| - Política: 175 comprobaciones, incluida la fixture | |||||
| `successful_pageout_deadman_avoidance_2026_09_21`. | |||||
| - Runtime: 180 comprobaciones con backend simulado y ninguna señal nativa. | |||||
| - Regresiones syslog: eventos incluidos/excluidos, ident, facility, severidad, | |||||
| pares clave=valor, ausencia de duplicados, desactivación y fallo aislado. | |||||
| - Regresión O(1): la acción EMERGENCY no llama `platform_walk`, incluso con un | |||||
| scan previo incompleto o un candidato rank 1 obsoleto. | |||||
| - `gmake test-sanitize`: AddressSanitizer y UndefinedBehaviorSanitizer. Se usó | |||||
| `ASAN_OPTIONS=detect_leaks=0` porque LeakSanitizer no funciona bajo el ptrace | |||||
| del entorno supervisado; no se atribuye una comprobación de fugas. | |||||
| - `gmake portability`: C99 estricto, warnings como errores, `-fanalyzer`, | |||||
| sintaxis shell y XML bien formado. | |||||
| - Validación de manifiesto/distribución: SMF no fuerza dry-run, configuración | |||||
| mantiene dry-run y los parámetros 1/2/2/15, 5/1/0/15 y 4/4/8/12. | |||||
| - Checksums SHA-256 regenerados sobre la entrega source-only. | |||||
| Los logs exactos se guardan en `logs/distribution-tests.log`, | |||||
| `logs/policy-tests.log`, `logs/runtime-tests.log`, `logs/sanitizers.log` y | |||||
| `logs/portability.log`. Los sketches API de Linux verifican consistencia C, no | |||||
| sustituyen cabeceras, ABI, enlace ni ejecución nativa illumos. | |||||
| ## Validación local de syslog en SmartOS | |||||
| No se presupone un archivo de destino. El operador debe inspeccionar: | |||||
| ```sh | |||||
| grep -v '^[[:space:]]*#' /etc/syslog.conf | |||||
| # Validación de la entrega v1.0.7 | |||||
| Fecha: 2026-09-22. Entrega source-only. **No se desplegó, no se modificó el | |||||
| nodo armado y no se enviaron señales reales.** La configuración conserva | |||||
| `ACTION_MODE dry-run`, syslog `local5`, cero UUID `allow` y SMF deshabilitado. | |||||
| ## Motivo | |||||
| panic32 terminó en `pageout_deadman` con `freemem=424`. El guard había resuelto | |||||
| dos emergencias, pero en una tercera transcurrieron unos 145 s entre la decisión | |||||
| SIGKILL y `signal_sent`. Después hubo recuperación parcial, recaída, caché con | |||||
| unos 167 s de edad, `no_valid_cached_candidate` y panic. Está confirmado el stall; | |||||
| no está confirmado si lo causó syslog, guard.log u otra syscall. Véase | |||||
| `INCIDENT_PAGEOUT_DEADMAN_PANIC32_2026_09_22.md`. | |||||
| ## Propiedades implementadas | |||||
| 1. Una acción real ejecuta validación puntual, señal, counters en memoria y sólo | |||||
| después enqueue de eventos. No hay log, syslog, rotación ni scan antes de la señal. | |||||
| 2. El logger consume un ring fijo de 512 eventos. El productor usa `trylock` y | |||||
| descarta antes que esperar; `dropped_events` queda informado posteriormente. | |||||
| 3. Guard log, rotación y syslog sólo ocurren en el worker. Sus stalls no detienen | |||||
| muestras ni nuevas acciones. | |||||
| 4. El scanner `/proc` es otro worker. El hilo de muestreo solicita trabajo sin | |||||
| esperar y conserva la última caché completa publicada. | |||||
| 5. La caché global contiene 16 candidatos; el log sigue mostrando top-3. | |||||
| 6. CRITICAL/EMERGENCY/FLOOR pueden revalidar caché antigua mediante PID, inicio, | |||||
| UUID, generación, zona `allow`, nombre y protecciones. La fuente se registra | |||||
| como `stale_cache_revalidated`. | |||||
| 7. La caché sobrevive EMERGENCY → RECOVERY → EMERGENCY hasta un scan sano. | |||||
| 8. Se instrumentan validación, entrada/retorno de PCKILL, dispatch, syscall, | |||||
| enqueue, guard log y syslog mediante reloj monotónico. | |||||
| 9. `logging_stall` se produce después de recuperar el worker; nunca se registra | |||||
| recursivamente desde la ruta de señal. | |||||
| ## Pruebas ejecutadas | |||||
| - Política: 175 comprobaciones; incluye los fixtures previos y no enlaza backend. | |||||
| - Runtime simulado: 501 comprobaciones; ninguna señal nativa. | |||||
| - Fixture `panic32_daemon_stall_2026_09_22`: conserva el intervalo de 145 s. | |||||
| - Logger/syslog detenidos 180 s: TERM y SIGKILL se solicitan sin esperar y el | |||||
| tiempo real de cada llamada a `act()` queda por debajo de 100 ms. | |||||
| - Guard log no disponible y cola mutex ocupada: la señal ocurre y la telemetría | |||||
| se descarta. | |||||
| - Scanner detenido/incompleto: no aparece ningún `platform_walk()` en rescate. | |||||
| - Caché de 180 s: se revalida y registra como `stale_cache_revalidated`. | |||||
| - 16 candidatos con scan inutilizable: 8 víctimas EMERGENCY (TERM+KILL) y 12 | |||||
| víctimas floor (KILL directo), respetando budgets, identidad y protecciones. | |||||
| - Regresiones de ranking global, fallback, PID reuse, dry-run, syslog selectivo, | |||||
| swap/pageout y máquina monotónica. | |||||
| - ASan + UBSan con `ASAN_OPTIONS=detect_leaks=0`; LeakSanitizer no funciona bajo | |||||
| el ptrace del entorno y no se atribuye comprobación de fugas. | |||||
| - C99 estricto, warnings como errores y `pthread`; GCC `-fanalyzer` por unidad. | |||||
| - `sh -n` para instalador/prueba de distribución y XML con `xmllint --nonet`. | |||||
| - Checksums SHA-256 del árbol source-only y verificación desde paquete extraído. | |||||
| Los resultados se conservan en `logs/policy-tests.log`, `logs/runtime-tests.log`, | |||||
| `logs/sanitizers.log`, `logs/portability.log` y `logs/distribution-tests.log`. | |||||
| ## Políticas no modificadas | |||||
| ```text | |||||
| PRE_SECONDS 1 | |||||
| PRESSURE_SECONDS 2 | |||||
| CRITICAL_SECONDS 2 | |||||
| CRITICAL_TERM_GRACE_SECONDS 5 | |||||
| EMERGENCY_TERM_GRACE_SECONDS 1 | |||||
| FLOOR_KILL_DELAY_SECONDS 0 | |||||
| MAX_VICTIMS_EPISODE 4 | |||||
| MAX_EMERGENCY_VICTIMS_EPISODE 8 | |||||
| MAX_FLOOR_VICTIMS_EPISODE 12 | |||||
| armed_warmup 60 | |||||
| SYSLOG_FACILITY local5 | |||||
| ``` | ``` | ||||
| y localizar la regla para `local5`. Esta entrega no cambia `syslog.conf`, no | |||||
| configura forwarding y no implanta Alloy/Loki. Un recolector central puede | |||||
| añadirse posteriormente fuera del proceso guard. | |||||
| Tampoco cambian score/ranking global, reglas UUID, procesos protegidos ni | |||||
| semántica monotónica de `sampling_overrun`. | |||||
| ## Validación nativa pendiente | |||||
| ## Validación nativa pendiente — NO GO para despliegue | |||||
| 1. Compilar y ejecutar la suite con cabeceras nativas SmartOS/illumos. | |||||
| 2. Validar/importar SMF manteniendo `ACTION_MODE dry-run` y comprobar | |||||
| `event=start version=1.0.6 mode=dry-run`. | |||||
| 3. Provocar sólo overruns controlados en laboratorio y confirmar continuidad | |||||
| monotónica, saltos registrados y ausencia de scans en rescate. | |||||
| 4. Confirmar en el destino local de `local5` las severidades y que muestras y | |||||
| deltas permanecen exclusivamente en `guard.log`. | |||||
| 5. Simular fallo/indisponibilidad del receptor syslog en laboratorio y medir | |||||
| que no cambia la decisión ni el resultado de señalización. | |||||
| 6. Mantener `armed` como NO GO en producción hasta una revisión humana explícita. | |||||
| 1. Compilar y ejecutar toda la suite con headers/ABI SmartOS nativos. | |||||
| 2. Confirmar soporte y comportamiento de pthreads en el servicio SMF. | |||||
| 3. En laboratorio desechable, bloquear por separado destino de guard.log y | |||||
| receptor local5 mientras se generan decisiones simuladas/controladas. | |||||
| 4. Confirmar que `decision_mono → kill_attempt_mono` permanece varios órdenes | |||||
| por debajo de 90 s y revisar `kill_latency_ms` por separado. | |||||
| 5. Bloquear o ralentizar `/proc` y verificar que muestras/acciones continúan con | |||||
| la caché publicada, sin carreras en la enumeración nativa. | |||||
| 6. Validar 8/12 candidatos reales, protecciones, supervisores y recuperación. | |||||
| 7. Revisar `logging_queue_drop`, `logging_delivery` y `logging_stall` bajo carga. | |||||
| 8. Mantener dry-run hasta revisión humana explícita. v1.0.7 no se considera apta | |||||
| para producción sólo por superar las pruebas portables. | |||||
| ## Límites | ## Límites | ||||
| La caché sólo contiene tres candidatos y falla cerrada si envejecen. RSS puede | |||||
| sobrestimar memoria recuperable. kstat, ZFS, libc, logging y syslog locales | |||||
| pueden experimentar latencia; el diseño difiere la notificación crítica hasta | |||||
| después de la ruta de acción, pero no ofrece tiempo real estricto. El guard | |||||
| reduce una ventana de riesgo y no garantiza evitar `pageout_deadman`. | |||||
| La validación puntual de un candidato y PCKILL siguen siendo syscalls del kernel | |||||
| y no pueden recibir una garantía temporal absoluta desde userland. La cola puede | |||||
| perder logs deliberadamente. Los workers bloqueados sólo se recuperan cuando la | |||||
| syscall subyacente retorna; el proceso no espera por ellos al rescatar ni al salir. | |||||
| RSS no equivale a memoria exclusiva recuperable. El guard reduce riesgo, no | |||||
| garantiza evitar `pageout_deadman`. | |||||
| @@ -1,4 +1,4 @@ | |||||
| # o6h-memory-guard 1.0.6 -- whitespace-separated, no shell evaluation. | |||||
| # o6h-memory-guard 1.0.7 -- whitespace-separated, no shell evaluation. | |||||
| # Restart required for configuration changes. Duplicate scalar keys are errors. | # Restart required for configuration changes. Duplicate scalar keys are errors. | ||||
| ACTION_MODE dry-run | ACTION_MODE dry-run | ||||
| SYSLOG_ENABLED yes | SYSLOG_ENABLED yes | ||||
| @@ -1,7 +1,3 @@ | |||||
| gmake portability | |||||
| PASS C99 syntax with -Wall -Wextra -Werror -Wformat=2 -pedantic | |||||
| PASS GCC -fanalyzer for every production translation unit | |||||
| PASS sh -n install.sh tests/test_distribution.sh | |||||
| PASS xmllint --nonet --noout smf/o6h-memory-guard.xml | |||||
| NOTE tests/api-sketch are local declarations, not native illumos headers or ABI validation. | |||||
| NOTE clang, clang-tidy, cppcheck, scan-build and shellcheck were not installed. | |||||
| gcc -std=c99 -O1 -g -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Isrc -Wall -Wextra -Werror -Wformat=2 -pedantic -pthread -Itests/api-sketch -fsyntax-only src/main.c src/core.c src/config.c src/support.c src/illumos.c | |||||
| sh -n install.sh tests/test_distribution.sh | |||||
| xmllint --nonet --noout smf/o6h-memory-guard.xml | |||||
| @@ -1,3 +1,20 @@ | |||||
| gcc -std=c99 -O1 -g -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Isrc -Wall -Wextra -Werror -Wformat=2 -pedantic -pthread -o test-guard tests/test_guard.c src/core.c src/config.c | |||||
| gcc -std=c99 -O1 -g -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Isrc -Wall -Wextra -Werror -Wformat=2 -pedantic -pthread -o test-runtime tests/test_runtime.c src/core.c src/config.c | |||||
| sh tests/test_distribution.sh | |||||
| PASS distribution defaults to ACTION_MODE dry-run; production SMF does not force it | |||||
| ./test-guard | |||||
| PASS strict configuration, safe defaults, explicit UUID authorization | |||||
| PASS mandatory protections, PID reuse and zone incarnation identity | |||||
| PASS timestamped 5/15/60s windows, missing data, signed growth, overflow | |||||
| PASS all six states, monotonic overruns, threshold jumps, corroboration and recovery | |||||
| PASS pageout_deadman_real_incident_2026_09 exits NORMAL before minfree | |||||
| PASS successful_pageout_deadman_avoidance_2026_09_21 incident fixture | |||||
| SIMULATED t=100 state=CRITICAL UUID=11111111-1111-4111-8111-111111111111 pid=123 WOULD_ACTION=SIGTERM | |||||
| SIMULATED t=105 state=CRITICAL pid=123 WOULD_ACTION=SIGKILL reason=memory_not_recovered | |||||
| SIMULATED state=EMERGENCY floor=1 WOULD_ACTION=SIGKILL reason=floor | |||||
| PASS TERM/KILL sequencing, emergency grace, floor, cooldown, budgets and no-action protections | |||||
| PASS 175 checks; no native backend linked, no signals sent | |||||
| ./test-runtime | |||||
| PASS actual daemon dry-run path records WOULD_ACTION=SIGTERM/SIGKILL; zero real requests | PASS actual daemon dry-run path records WOULD_ACTION=SIGTERM/SIGKILL; zero real requests | ||||
| PASS continuous sample contract and 5/15/60 deltas for commitment, physical swap and pageout | PASS continuous sample contract and 5/15/60 deltas for commitment, physical swap and pageout | ||||
| PASS armed-path mock requests, explicit warmup, stale cache, logging/sampling failures, stop and fresh floor | PASS armed-path mock requests, explicit warmup, stale cache, logging/sampling failures, stop and fresh floor | ||||
| @@ -6,4 +23,9 @@ PASS SmartOS fixture produces a global cross-zone top-3 and bounded zone context | |||||
| PASS incomplete scan preserves cache; emergency uses O(1) fallback without a new walk | PASS incomplete scan preserves cache; emergency uses O(1) fallback without a new walk | ||||
| PASS PID/start-time reuse rejects rank 1 and selects cached fallback | PASS PID/start-time reuse rejects rank 1 and selects cached fallback | ||||
| PASS selective native syslog ident/facility/severity, exclusions, signal ordering, disable and failure isolation | PASS selective native syslog ident/facility/severity, exclusions, signal ordering, disable and failure isolation | ||||
| PASS 180 runtime checks; all platform calls mocked, no signals sent | |||||
| PASS panic32 fault: logger/syslog stopped 180s, stale cache revalidated, signal dispatch <100ms | |||||
| PASS stalled/incomplete scanner cannot delay or trigger a rescue-path proc scan | |||||
| PASS 16-entry stale rescue cache supplies all 8 EMERGENCY victims without scan | |||||
| PASS 16-entry stale rescue cache supplies all 12 FLOOR victims without scan | |||||
| PASS saturated/locked event queue drops telemetry without delaying the signal | |||||
| PASS 501 runtime checks; all platform calls mocked, no signals sent | |||||
| @@ -1,4 +1,4 @@ | |||||
| gcc -std=c99 -O1 -g -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Isrc -Wall -Wextra -Werror -Wformat=2 -pedantic -fsanitize=address,undefined -fno-omit-frame-pointer -o test-sanitize-guard tests/test_guard.c src/core.c src/config.c | |||||
| gcc -std=c99 -O1 -g -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Isrc -Wall -Wextra -Werror -Wformat=2 -pedantic -pthread -fsanitize=address,undefined -fno-omit-frame-pointer -o test-sanitize-guard tests/test_guard.c src/core.c src/config.c | |||||
| ./test-sanitize-guard | ./test-sanitize-guard | ||||
| PASS strict configuration, safe defaults, explicit UUID authorization | PASS strict configuration, safe defaults, explicit UUID authorization | ||||
| PASS mandatory protections, PID reuse and zone incarnation identity | PASS mandatory protections, PID reuse and zone incarnation identity | ||||
| @@ -11,7 +11,7 @@ SIMULATED t=105 state=CRITICAL pid=123 WOULD_ACTION=SIGKILL reason=memory_not_re | |||||
| SIMULATED state=EMERGENCY floor=1 WOULD_ACTION=SIGKILL reason=floor | SIMULATED state=EMERGENCY floor=1 WOULD_ACTION=SIGKILL reason=floor | ||||
| PASS TERM/KILL sequencing, emergency grace, floor, cooldown, budgets and no-action protections | PASS TERM/KILL sequencing, emergency grace, floor, cooldown, budgets and no-action protections | ||||
| PASS 175 checks; no native backend linked, no signals sent | PASS 175 checks; no native backend linked, no signals sent | ||||
| gcc -std=c99 -O1 -g -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Isrc -Wall -Wextra -Werror -Wformat=2 -pedantic -fsanitize=address,undefined -fno-omit-frame-pointer -o test-sanitize-runtime tests/test_runtime.c src/core.c src/config.c | |||||
| gcc -std=c99 -O1 -g -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Isrc -Wall -Wextra -Werror -Wformat=2 -pedantic -pthread -fsanitize=address,undefined -fno-omit-frame-pointer -o test-sanitize-runtime tests/test_runtime.c src/core.c src/config.c | |||||
| ./test-sanitize-runtime | ./test-sanitize-runtime | ||||
| PASS actual daemon dry-run path records WOULD_ACTION=SIGTERM/SIGKILL; zero real requests | PASS actual daemon dry-run path records WOULD_ACTION=SIGTERM/SIGKILL; zero real requests | ||||
| PASS continuous sample contract and 5/15/60 deltas for commitment, physical swap and pageout | PASS continuous sample contract and 5/15/60 deltas for commitment, physical swap and pageout | ||||
| @@ -21,7 +21,9 @@ PASS SmartOS fixture produces a global cross-zone top-3 and bounded zone context | |||||
| PASS incomplete scan preserves cache; emergency uses O(1) fallback without a new walk | PASS incomplete scan preserves cache; emergency uses O(1) fallback without a new walk | ||||
| PASS PID/start-time reuse rejects rank 1 and selects cached fallback | PASS PID/start-time reuse rejects rank 1 and selects cached fallback | ||||
| PASS selective native syslog ident/facility/severity, exclusions, signal ordering, disable and failure isolation | PASS selective native syslog ident/facility/severity, exclusions, signal ordering, disable and failure isolation | ||||
| PASS 180 runtime checks; all platform calls mocked, no signals sent | |||||
| LeakSanitizer was intentionally disabled because the supervised environment rejects ptrace; no leak-check result is claimed. | |||||
| ASAN_OPTIONS=detect_leaks=0 gmake test-sanitize | |||||
| GCC AddressSanitizer + UndefinedBehaviorSanitizer | |||||
| PASS panic32 fault: logger/syslog stopped 180s, stale cache revalidated, signal dispatch <100ms | |||||
| PASS stalled/incomplete scanner cannot delay or trigger a rescue-path proc scan | |||||
| PASS 16-entry stale rescue cache supplies all 8 EMERGENCY victims without scan | |||||
| PASS 16-entry stale rescue cache supplies all 12 FLOOR victims without scan | |||||
| PASS saturated/locked event queue drops telemetry without delaying the signal | |||||
| PASS 501 runtime checks; all platform calls mocked, no signals sent | |||||
| @@ -13,7 +13,7 @@ | |||||
| #define CONF_PATH "/opt/custom/etc/o6h-memory-guard.conf" | #define CONF_PATH "/opt/custom/etc/o6h-memory-guard.conf" | ||||
| #define LOG_DIR "/opt/custom/var/log/o6h-memory-guard" | #define LOG_DIR "/opt/custom/var/log/o6h-memory-guard" | ||||
| #define RUN_DIR "/opt/custom/var/run" | #define RUN_DIR "/opt/custom/var/run" | ||||
| #define VERSION "1.0.6" | |||||
| #define VERSION "1.0.7" | |||||
| enum state { NORMAL, PRE_PRESSURE, PRESSURE, CRITICAL, EMERGENCY, RECOVERY }; | enum state { NORMAL, PRE_PRESSURE, PRESSURE, CRITICAL, EMERGENCY, RECOVERY }; | ||||
| enum policy { OBSERVE, ALLOW, PROTECT }; | enum policy { OBSERVE, ALLOW, PROTECT }; | ||||
| @@ -95,7 +95,8 @@ int platform_init(void); | |||||
| void platform_fini(void); | void platform_fini(void); | ||||
| int platform_sample(sample_t *); | int platform_sample(sample_t *); | ||||
| int platform_walk(int (*)(const process_t *, void *), void *, unsigned); | int platform_walk(int (*)(const process_t *, void *), void *, unsigned); | ||||
| int platform_signal(const config_t *, const process_t *, enum action, int, uint64_t); | |||||
| int platform_signal(const config_t *, const process_t *, enum action, int, | |||||
| uint64_t, double *, double *); | |||||
| double monotime(void); | double monotime(void); | ||||
| int secure_file(const char *, int, unsigned); | int secure_file(const char *, int, unsigned); | ||||
| int secure_directory(const char *); | int secure_directory(const char *); | ||||
| @@ -227,7 +227,7 @@ int platform_walk(int (*visit)(const process_t *, void *), void *arg, unsigned b | |||||
| } | } | ||||
| int platform_signal(const config_t *c, const process_t *expected, enum action action, int real, | int platform_signal(const config_t *c, const process_t *expected, enum action action, int real, | ||||
| uint64_t ceiling) | |||||
| uint64_t ceiling, double *attempt_mono, double *return_mono) | |||||
| { | { | ||||
| char path[64]; | char path[64]; | ||||
| int dir = -1, infofd = -1, ctl = -1, result = -1, saved; | int dir = -1, infofd = -1, ctl = -1, result = -1, saved; | ||||
| @@ -263,7 +263,9 @@ int platform_signal(const config_t *c, const process_t *expected, enum action ac | |||||
| command[0] = PCKILL; command[1] = action == TERM ? SIGTERM : SIGKILL; | command[0] = PCKILL; command[1] = action == TERM ? SIGTERM : SIGKILL; | ||||
| /* Single atomic proc control message; never retry an ambiguous write. */ | /* Single atomic proc control message; never retry an ambiguous write. */ | ||||
| { | { | ||||
| if (attempt_mono) *attempt_mono = monotime(); | |||||
| ssize_t written = write(ctl, command, sizeof (command)); | ssize_t written = write(ctl, command, sizeof (command)); | ||||
| if (return_mono) *return_mono = monotime(); | |||||
| if (written == sizeof (command)) result = 0; | if (written == sizeof (command)) result = 0; | ||||
| else if (written >= 0) errno = EIO; | else if (written >= 0) errno = EIO; | ||||
| } | } | ||||
| @@ -12,6 +12,7 @@ | |||||
| #include <string.h> | #include <string.h> | ||||
| #include <stdlib.h> | #include <stdlib.h> | ||||
| #include <syslog.h> | #include <syslog.h> | ||||
| #include <pthread.h> | |||||
| typedef struct { | typedef struct { | ||||
| process_t p; history_t rss; unsigned seen; double last; | process_t p; history_t rss; unsigned seen; double last; | ||||
| @@ -20,7 +21,10 @@ typedef struct { | |||||
| char uuid[64]; uint64_t generation, rss; | char uuid[64]; uint64_t generation, rss; | ||||
| history_t history; unsigned seen; double last; | history_t history; unsigned seen; double last; | ||||
| } zone_t; | } zone_t; | ||||
| #define CACHED_CANDIDATES 3 | |||||
| #define CACHED_CANDIDATES 16 | |||||
| #define LOGGED_CANDIDATES 3 | |||||
| #define EVENT_QUEUE_SIZE 512 | |||||
| #define EVENT_BODY_SIZE 1536 | |||||
| typedef struct { | typedef struct { | ||||
| process_t p; | process_t p; | ||||
| delta_t d5, d15, d60; | delta_t d5, d15, d60; | ||||
| @@ -35,6 +39,13 @@ static config_t cfg; | |||||
| static tracked_t processes[MAX_PROCS]; | static tracked_t processes[MAX_PROCS]; | ||||
| static zone_t zone_stats[MAX_ZONES]; | static zone_t zone_stats[MAX_ZONES]; | ||||
| static cached_candidate_t candidate_cache[CACHED_CANDIDATES]; | static cached_candidate_t candidate_cache[CACHED_CANDIDATES]; | ||||
| static cached_candidate_t published_cache[CACHED_CANDIDATES]; | |||||
| static pthread_mutex_t cache_mutex = PTHREAD_MUTEX_INITIALIZER; | |||||
| static pthread_mutex_t scan_mutex = PTHREAD_MUTEX_INITIALIZER; | |||||
| static pthread_cond_t scan_cond = PTHREAD_COND_INITIALIZER; | |||||
| static int scanner_async, scan_requested, scan_snapshot_requested; | |||||
| static int published_scan_complete; | |||||
| static double published_latest_complete_scan_at; | |||||
| enum metric_index { | enum metric_index { | ||||
| METRIC_FREE, | METRIC_FREE, | ||||
| METRIC_KERNEL, | METRIC_KERNEL, | ||||
| @@ -56,21 +67,94 @@ static const char *const metric_names[METRIC_COUNT] = { | |||||
| static engine_t engine; | static engine_t engine; | ||||
| static controller_t controller; | static controller_t controller; | ||||
| static unsigned scan_generation; | static unsigned scan_generation; | ||||
| static double scan_at, latest_complete_scan_at, started; | |||||
| static double scan_at, scanner_latest_complete_scan_at, latest_complete_scan_at, started; | |||||
| static int scan_complete; | static int scan_complete; | ||||
| static int logfd = -1, log_ok = 1, lockfd = -1; | static int logfd = -1, log_ok = 1, lockfd = -1; | ||||
| static int syslog_ready, syslog_init_failed; | static int syslog_ready, syslog_init_failed; | ||||
| static uint64_t log_bytes; | static uint64_t log_bytes; | ||||
| static volatile sig_atomic_t stopping, reopen_log; | static volatile sig_atomic_t stopping, reopen_log; | ||||
| static void delta_text(char *, size_t, delta_t); | static void delta_text(char *, size_t, delta_t); | ||||
| static int log_open(void); | |||||
| static int log_rotate(void); | |||||
| typedef struct { | |||||
| char body[EVENT_BODY_SIZE]; | |||||
| time_t wall_time; | |||||
| double mono_time; | |||||
| uint64_t sequence; | |||||
| int guard_log, syslog_channel, priority, diagnostic; | |||||
| } queued_event_t; | |||||
| static queued_event_t event_queue[EVENT_QUEUE_SIZE]; | |||||
| static unsigned event_head, event_tail, event_count; | |||||
| static uint64_t event_sequence; | |||||
| static volatile unsigned dropped_events; | |||||
| static pthread_mutex_t event_mutex = PTHREAD_MUTEX_INITIALIZER; | |||||
| static pthread_cond_t event_cond = PTHREAD_COND_INITIALIZER; | |||||
| static int async_logging; | |||||
| static int queue_event_v(int, int, int, int, const char *, va_list); | |||||
| static void *logger_worker(void *); | |||||
| static int write_event_now(const queued_event_t *event) | |||||
| { | |||||
| char line[2048], stamp[40]; struct tm tm; size_t len; int n; | |||||
| double guard_start = 0, guard_ms = 0, syslog_start = 0, syslog_ms = 0; | |||||
| int guard_failed = 0; | |||||
| if (event->guard_log && log_ok) { | |||||
| guard_start = monotime(); | |||||
| gmtime_r(&event->wall_time, &tm); | |||||
| strftime(stamp, sizeof (stamp), "%Y-%m-%dT%H:%M:%SZ", &tm); | |||||
| n = snprintf(line, sizeof (line), "%s mono=%.3f %s\n", stamp, | |||||
| event->mono_time, event->body); | |||||
| if (n < 0 || (size_t)n >= sizeof (line)) guard_failed = 1; | |||||
| else { | |||||
| len = (size_t)n; | |||||
| if (log_bytes + len > (uint64_t)cfg.log_mib * 1024 * 1024 && | |||||
| log_rotate()) guard_failed = 1; | |||||
| else if (write(logfd, line, len) != (ssize_t)len) guard_failed = 1; | |||||
| else log_bytes += len; | |||||
| } | |||||
| guard_ms = 1000 * (monotime() - guard_start); | |||||
| if (guard_failed) log_ok = 0; | |||||
| } | |||||
| if (event->syslog_channel && syslog_ready) { | |||||
| syslog_start = monotime(); | |||||
| (void)platform_syslog_write(event->priority, event->body); | |||||
| syslog_ms = 1000 * (monotime() - syslog_start); | |||||
| } | |||||
| if (async_logging && !event->diagnostic && | |||||
| (event->syslog_channel || guard_ms >= 100 || syslog_ms >= 100)) { | |||||
| queued_event_t diagnostic; | |||||
| memset(&diagnostic, 0, sizeof (diagnostic)); | |||||
| diagnostic.wall_time = time(NULL); diagnostic.mono_time = monotime(); | |||||
| diagnostic.guard_log = diagnostic.diagnostic = 1; | |||||
| diagnostic.sequence = __sync_add_and_fetch(&event_sequence, 1); | |||||
| snprintf(diagnostic.body, sizeof (diagnostic.body), | |||||
| "event=%s source_seq=%" PRIu64 | |||||
| " guard_log_write_latency_ms=%.3f syslog_latency_ms=%.3f%s", | |||||
| guard_ms >= 100 || syslog_ms >= 100 ? "logging_stall" : "logging_delivery", | |||||
| event->sequence, guard_ms, syslog_ms, | |||||
| guard_ms >= 100 ? " channel=guard_log" : | |||||
| syslog_ms >= 100 ? " channel=syslog" : ""); | |||||
| if (pthread_mutex_trylock(&event_mutex) == 0) { | |||||
| if (event_count < EVENT_QUEUE_SIZE) { | |||||
| event_queue[event_tail] = diagnostic; | |||||
| event_tail = (event_tail + 1) % EVENT_QUEUE_SIZE; event_count++; | |||||
| pthread_cond_signal(&event_cond); | |||||
| } else __sync_fetch_and_add(&dropped_events, 1); | |||||
| pthread_mutex_unlock(&event_mutex); | |||||
| } else __sync_fetch_and_add(&dropped_events, 1); | |||||
| } | |||||
| if (guard_failed && syslog_ready) | |||||
| (void)platform_syslog_write(LOG_CRIT, | |||||
| "event=internal_error reason=persistent_logging_failed continuation=syslog_only"); | |||||
| return guard_failed ? -1 : 0; | |||||
| } | |||||
| static void syslog_event(int priority, const char *fmt, ...) | static void syslog_event(int priority, const char *fmt, ...) | ||||
| { | { | ||||
| char message[1536]; va_list ap; int n; | |||||
| if (!syslog_ready) return; | |||||
| va_start(ap, fmt); n = vsnprintf(message, sizeof (message), fmt, ap); va_end(ap); | |||||
| if (n < 0 || (size_t)n >= sizeof (message)) return; | |||||
| (void)platform_syslog_write(priority, message); | |||||
| va_list ap; | |||||
| va_start(ap, fmt); (void)queue_event_v(0, 1, priority, 0, fmt, ap); va_end(ap); | |||||
| } | } | ||||
| static void handler(int sig) | static void handler(int sig) | ||||
| @@ -101,38 +185,92 @@ static int log_rotate(void) | |||||
| return log_open(); | return log_open(); | ||||
| } | } | ||||
| static int queue_event_v(int guard_log, int syslog_channel, int priority, | |||||
| int diagnostic, const char *fmt, va_list ap) | |||||
| { | |||||
| queued_event_t event; int n; | |||||
| memset(&event, 0, sizeof (event)); | |||||
| n = vsnprintf(event.body, sizeof (event.body), fmt, ap); | |||||
| if (n < 0 || (size_t)n >= sizeof (event.body)) return -1; | |||||
| event.wall_time = time(NULL); event.mono_time = monotime(); | |||||
| event.guard_log = guard_log; event.syslog_channel = syslog_channel; | |||||
| event.priority = priority; event.diagnostic = diagnostic; | |||||
| event.sequence = __sync_add_and_fetch(&event_sequence, 1); | |||||
| if (!async_logging) return write_event_now(&event); | |||||
| if (pthread_mutex_trylock(&event_mutex) != 0) { | |||||
| __sync_fetch_and_add(&dropped_events, 1); return -1; | |||||
| } | |||||
| if (event_count == EVENT_QUEUE_SIZE) { | |||||
| __sync_fetch_and_add(&dropped_events, 1); | |||||
| pthread_mutex_unlock(&event_mutex); return -1; | |||||
| } | |||||
| event_queue[event_tail] = event; | |||||
| event_tail = (event_tail + 1) % EVENT_QUEUE_SIZE; event_count++; | |||||
| pthread_cond_signal(&event_cond); pthread_mutex_unlock(&event_mutex); | |||||
| return 0; | |||||
| } | |||||
| static int emit(const char *fmt, ...) | static int emit(const char *fmt, ...) | ||||
| { | { | ||||
| char line[2048], stamp[40]; struct tm tm; time_t now = time(NULL); | |||||
| size_t len; int n; va_list ap; | |||||
| if (!log_ok) return -1; | |||||
| gmtime_r(&now, &tm); strftime(stamp, sizeof (stamp), "%Y-%m-%dT%H:%M:%SZ", &tm); | |||||
| n = snprintf(line, sizeof (line), "%s mono=%.3f ", stamp, monotime()); | |||||
| if (n < 0 || (size_t)n >= sizeof (line)) goto failed; | |||||
| len = (size_t)n; | |||||
| va_start(ap, fmt); n = vsnprintf(line + len, sizeof (line) - len, fmt, ap); va_end(ap); | |||||
| if (n < 0 || (size_t)n >= sizeof (line) - len - 1) goto failed; | |||||
| len += (size_t)n; line[len++] = '\n'; | |||||
| if (log_bytes + len > (uint64_t)cfg.log_mib * 1024 * 1024 && log_rotate()) goto failed; | |||||
| if (write(logfd, line, len) != (ssize_t)len) goto failed; | |||||
| log_bytes += len; return 0; | |||||
| failed: | |||||
| log_ok = 0; | |||||
| syslog_event(LOG_CRIT, "event=internal_error reason=persistent_logging_failed actions=inhibited"); | |||||
| fprintf(stderr, "o6h-memory-guard: persistent logging failed; actions inhibited until restart\n"); | |||||
| return -1; | |||||
| va_list ap; int result; | |||||
| va_start(ap, fmt); result = queue_event_v(1, 0, 0, 0, fmt, ap); va_end(ap); | |||||
| return result; | |||||
| } | } | ||||
| static int emit_important(int priority, const char *fmt, ...) | static int emit_important(int priority, const char *fmt, ...) | ||||
| { | { | ||||
| char message[1536]; va_list ap; int n, result; | |||||
| va_start(ap, fmt); n = vsnprintf(message, sizeof (message), fmt, ap); va_end(ap); | |||||
| if (n < 0 || (size_t)n >= sizeof (message)) return -1; | |||||
| result = emit("%s", message); | |||||
| syslog_event(priority, "%s", message); | |||||
| va_list ap; int result; | |||||
| va_start(ap, fmt); result = queue_event_v(1, 1, priority, 0, fmt, ap); va_end(ap); | |||||
| return result; | return result; | ||||
| } | } | ||||
| static void *logger_worker(void *unused) | |||||
| { | |||||
| (void)unused; | |||||
| for (;;) { | |||||
| queued_event_t event; struct timespec until; unsigned dropped; | |||||
| if (reopen_log) { | |||||
| reopen_log = 0; | |||||
| if (logfd >= 0) close(logfd); | |||||
| log_ok = log_open() == 0; | |||||
| } | |||||
| pthread_mutex_lock(&event_mutex); | |||||
| while (!event_count && !stopping) { | |||||
| clock_gettime(CLOCK_REALTIME, &until); until.tv_sec++; | |||||
| (void)pthread_cond_timedwait(&event_cond, &event_mutex, &until); | |||||
| if (reopen_log) break; | |||||
| } | |||||
| if (!event_count) { | |||||
| pthread_mutex_unlock(&event_mutex); | |||||
| if (stopping) { | |||||
| platform_syslog_close(); | |||||
| if (logfd >= 0) close(logfd); | |||||
| return NULL; | |||||
| } | |||||
| continue; | |||||
| } | |||||
| event = event_queue[event_head]; | |||||
| event_head = (event_head + 1) % EVENT_QUEUE_SIZE; event_count--; | |||||
| pthread_mutex_unlock(&event_mutex); | |||||
| (void)write_event_now(&event); | |||||
| dropped = __sync_lock_test_and_set(&dropped_events, 0); | |||||
| if (dropped) | |||||
| emit("event=logging_queue_drop dropped_events=%u policy=drop_nonblocking", | |||||
| dropped); | |||||
| } | |||||
| } | |||||
| static int logger_start(void) | |||||
| { | |||||
| pthread_t thread; | |||||
| async_logging = 1; | |||||
| if (pthread_create(&thread, NULL, logger_worker, NULL)) { | |||||
| async_logging = 0; return -1; | |||||
| } | |||||
| (void)pthread_detach(thread); | |||||
| return 0; | |||||
| } | |||||
| static int state_priority(enum state state, int floor) | static int state_priority(enum state state, int floor) | ||||
| { | { | ||||
| if (floor || state == EMERGENCY) return LOG_CRIT; | if (floor || state == EMERGENCY) return LOG_CRIT; | ||||
| @@ -278,6 +416,33 @@ static int cache_better(const cached_candidate_t *a, const cached_candidate_t *b | |||||
| return a->p.pid < b->p.pid; | return a->p.pid < b->p.pid; | ||||
| } | } | ||||
| static void publish_candidate_cache(const cached_candidate_t *next, int complete) | |||||
| { | |||||
| pthread_mutex_lock(&cache_mutex); | |||||
| if (complete) { | |||||
| memcpy(published_cache, next, sizeof (published_cache)); | |||||
| published_latest_complete_scan_at = scan_at; | |||||
| } | |||||
| published_scan_complete = complete; | |||||
| pthread_mutex_unlock(&cache_mutex); | |||||
| if (!scanner_async) { | |||||
| if (complete) { | |||||
| memcpy(candidate_cache, next, sizeof (candidate_cache)); | |||||
| latest_complete_scan_at = scan_at; | |||||
| } | |||||
| scan_complete = complete; | |||||
| } | |||||
| } | |||||
| static void refresh_candidate_cache(void) | |||||
| { | |||||
| if (!scanner_async || pthread_mutex_trylock(&cache_mutex) != 0) return; | |||||
| memcpy(candidate_cache, published_cache, sizeof (candidate_cache)); | |||||
| scan_complete = published_scan_complete; | |||||
| latest_complete_scan_at = published_latest_complete_scan_at; | |||||
| pthread_mutex_unlock(&cache_mutex); | |||||
| } | |||||
| static void rebuild_candidate_cache(void) | static void rebuild_candidate_cache(void) | ||||
| { | { | ||||
| cached_candidate_t next[CACHED_CANDIDATES]; | cached_candidate_t next[CACHED_CANDIDATES]; | ||||
| @@ -288,7 +453,7 @@ static void rebuild_candidate_cache(void) | |||||
| tracked_t *t = &processes[i]; | tracked_t *t = &processes[i]; | ||||
| zone_t *z; | zone_t *z; | ||||
| if (t->seen != scan_generation || !t->p.pid || | if (t->seen != scan_generation || !t->p.pid || | ||||
| protected_process(&cfg, &t->p) || was_victim(&controller, &t->p)) continue; | |||||
| protected_process(&cfg, &t->p)) continue; | |||||
| z = zone_for_process(&t->p); | z = zone_for_process(&t->p); | ||||
| if (!z) continue; | if (!z) continue; | ||||
| memset(&entry, 0, sizeof (entry)); | memset(&entry, 0, sizeof (entry)); | ||||
| @@ -305,30 +470,32 @@ static void rebuild_candidate_cache(void) | |||||
| next[j] = entry; break; | next[j] = entry; break; | ||||
| } | } | ||||
| } | } | ||||
| memcpy(candidate_cache, next, sizeof (candidate_cache)); | |||||
| latest_complete_scan_at = scan_at; | |||||
| for (i = 0; i < CACHED_CANDIDATES && candidate_cache[i].valid; i++) { | |||||
| scanner_latest_complete_scan_at = scan_at; | |||||
| for (i = 0; i < CACHED_CANDIDATES && next[i].valid; i++) | |||||
| next[i].rank = i + 1; | |||||
| publish_candidate_cache(next, 1); | |||||
| for (i = 0; i < CACHED_CANDIDATES && next[i].valid; i++) { | |||||
| char d5[64], d15[64], d60[64]; | char d5[64], d15[64], d60[64]; | ||||
| candidate_cache[i].rank = i + 1; | |||||
| delta_text(d5, sizeof (d5), candidate_cache[i].d5); | |||||
| delta_text(d15, sizeof (d15), candidate_cache[i].d15); | |||||
| delta_text(d60, sizeof (d60), candidate_cache[i].d60); | |||||
| delta_text(d5, sizeof (d5), next[i].d5); | |||||
| delta_text(d15, sizeof (d15), next[i].d15); | |||||
| delta_text(d60, sizeof (d60), next[i].d60); | |||||
| if (i >= LOGGED_CANDIDATES) continue; | |||||
| emit("event=candidate_cache rank=%u UUID=%s alias=%s zone_generation=%" PRIu64 | emit("event=candidate_cache rank=%u UUID=%s alias=%s zone_generation=%" PRIu64 | ||||
| " pid=%d start=%" PRId64 ".%09" PRId64 " comm=%s rss_kib=%" PRIu64 | " pid=%d start=%" PRId64 ".%09" PRId64 " comm=%s rss_kib=%" PRIu64 | ||||
| " d5_kib=%s d15_kib=%s d60_kib=%s age_ms=0 eligible=%d" | " d5_kib=%s d15_kib=%s d60_kib=%s age_ms=0 eligible=%d" | ||||
| " rss_component=%" PRIu64 " growth5_component=%" PRIu64 | " rss_component=%" PRIu64 " growth5_component=%" PRIu64 | ||||
| " growth15_component=%" PRIu64 " growth60_component=%" PRIu64 | " growth15_component=%" PRIu64 " growth60_component=%" PRIu64 | ||||
| " zone_component=%" PRIu64 " total_score=%" PRIu64, | " zone_component=%" PRIu64 " total_score=%" PRIu64, | ||||
| candidate_cache[i].rank, candidate_cache[i].p.uuid, | |||||
| candidate_cache[i].alias, candidate_cache[i].p.zone_generation, | |||||
| candidate_cache[i].p.pid, candidate_cache[i].p.start_sec, | |||||
| candidate_cache[i].p.start_nsec, candidate_cache[i].p.name, | |||||
| candidate_cache[i].p.rss_kib, d5, d15, d60, | |||||
| candidate_cache[i].eligible, candidate_cache[i].rss_component, | |||||
| candidate_cache[i].growth5_component, | |||||
| candidate_cache[i].growth15_component, | |||||
| candidate_cache[i].growth60_component, | |||||
| candidate_cache[i].zone_component, candidate_cache[i].total_score); | |||||
| next[i].rank, next[i].p.uuid, | |||||
| next[i].alias, next[i].p.zone_generation, | |||||
| next[i].p.pid, next[i].p.start_sec, | |||||
| next[i].p.start_nsec, next[i].p.name, | |||||
| next[i].p.rss_kib, d5, d15, d60, | |||||
| next[i].eligible, next[i].rss_component, | |||||
| next[i].growth5_component, | |||||
| next[i].growth15_component, | |||||
| next[i].growth60_component, | |||||
| next[i].zone_component, next[i].total_score); | |||||
| } | } | ||||
| } | } | ||||
| @@ -355,29 +522,31 @@ static void log_process(const char *event, const tracked_t *t) | |||||
| static void scan_processes(int snapshot) | static void scan_processes(int snapshot) | ||||
| { | { | ||||
| unsigned i, j, ntop = 0; | unsigned i, j, ntop = 0; | ||||
| int top[20]; | |||||
| int top[20], complete; | |||||
| scan_at = monotime(); scan_generation++; | scan_at = monotime(); scan_generation++; | ||||
| if (!scan_generation) scan_generation++; /* practically unreachable */ | if (!scan_generation) scan_generation++; /* practically unreachable */ | ||||
| scan_complete = platform_walk(visit, NULL, cfg.scan_budget_ms) == 0; | |||||
| complete = platform_walk(visit, NULL, cfg.scan_budget_ms) == 0; | |||||
| emit("event=proc_scan complete=%d duration_ms=%.1f capacity=%d" | emit("event=proc_scan complete=%d duration_ms=%.1f capacity=%d" | ||||
| " latest_complete_age_ms=%.1f", scan_complete, | |||||
| " latest_complete_age_ms=%.1f", complete, | |||||
| 1000 * (monotime() - scan_at), MAX_PROCS, | 1000 * (monotime() - scan_at), MAX_PROCS, | ||||
| latest_complete_scan_at > 0 ? 1000 * (scan_at - latest_complete_scan_at) : -1.0); | |||||
| scanner_latest_complete_scan_at > 0 ? | |||||
| 1000 * (scan_at - scanner_latest_complete_scan_at) : -1.0); | |||||
| for (i = 0; i < MAX_ZONES; i++) { | for (i = 0; i < MAX_ZONES; i++) { | ||||
| zone_t *z = &zone_stats[i]; | zone_t *z = &zone_stats[i]; | ||||
| char d5[64], d15[64], d60[64]; | char d5[64], d15[64], d60[64]; | ||||
| if (z->seen != scan_generation) continue; | if (z->seen != scan_generation) continue; | ||||
| /* Incomplete totals must never become valid zone-growth baselines. */ | /* Incomplete totals must never become valid zone-growth baselines. */ | ||||
| if (scan_complete) history_add(&z->history, scan_at, z->rss); | |||||
| if (complete) history_add(&z->history, scan_at, z->rss); | |||||
| if (!snapshot) continue; | if (!snapshot) continue; | ||||
| delta_text(d5, sizeof (d5), history_delta(&z->history, scan_at, 5)); | delta_text(d5, sizeof (d5), history_delta(&z->history, scan_at, 5)); | ||||
| delta_text(d15, sizeof (d15), history_delta(&z->history, scan_at, 15)); | delta_text(d15, sizeof (d15), history_delta(&z->history, scan_at, 15)); | ||||
| delta_text(d60, sizeof (d60), history_delta(&z->history, scan_at, 60)); | delta_text(d60, sizeof (d60), history_delta(&z->history, scan_at, 60)); | ||||
| emit("event=zone UUID=%s alias=%s generation=%" PRIu64 " rss_sum_kib=%" PRIu64 | emit("event=zone UUID=%s alias=%s generation=%" PRIu64 " rss_sum_kib=%" PRIu64 | ||||
| " complete=%d d5_kib=%s d15_kib=%s d60_kib=%s", z->uuid, alias_for(z->uuid), | " complete=%d d5_kib=%s d15_kib=%s d60_kib=%s", z->uuid, alias_for(z->uuid), | ||||
| z->generation, z->rss, scan_complete, d5, d15, d60); | |||||
| z->generation, z->rss, complete, d5, d15, d60); | |||||
| } | } | ||||
| if (scan_complete) rebuild_candidate_cache(); | |||||
| if (complete) rebuild_candidate_cache(); | |||||
| else publish_candidate_cache(NULL, 0); | |||||
| if (!snapshot) return; | if (!snapshot) return; | ||||
| for (i = 0; i < MAX_PROCS; i++) { | for (i = 0; i < MAX_PROCS; i++) { | ||||
| if (processes[i].seen != scan_generation || !processes[i].p.pid) continue; | if (processes[i].seen != scan_generation || !processes[i].p.pid) continue; | ||||
| @@ -391,6 +560,43 @@ static void scan_processes(int snapshot) | |||||
| for (i = 0; i < ntop; i++) log_process("top_process", &processes[top[i]]); | for (i = 0; i < ntop; i++) log_process("top_process", &processes[top[i]]); | ||||
| } | } | ||||
| static void *scanner_worker(void *unused) | |||||
| { | |||||
| (void)unused; | |||||
| for (;;) { | |||||
| int snapshot; | |||||
| pthread_mutex_lock(&scan_mutex); | |||||
| while (!scan_requested && !stopping) | |||||
| pthread_cond_wait(&scan_cond, &scan_mutex); | |||||
| if (stopping) { pthread_mutex_unlock(&scan_mutex); return NULL; } | |||||
| snapshot = scan_snapshot_requested; | |||||
| scan_requested = scan_snapshot_requested = 0; | |||||
| pthread_mutex_unlock(&scan_mutex); | |||||
| scan_processes(snapshot); | |||||
| } | |||||
| } | |||||
| static int scanner_start(void) | |||||
| { | |||||
| pthread_t thread; | |||||
| scanner_async = 1; | |||||
| if (pthread_create(&thread, NULL, scanner_worker, NULL)) { | |||||
| scanner_async = 0; return -1; | |||||
| } | |||||
| (void)pthread_detach(thread); | |||||
| return 0; | |||||
| } | |||||
| static void request_scan(int snapshot) | |||||
| { | |||||
| if (!scanner_async) { scan_processes(snapshot); return; } | |||||
| if (pthread_mutex_trylock(&scan_mutex) != 0) return; | |||||
| scan_requested = 1; | |||||
| if (snapshot) scan_snapshot_requested = 1; | |||||
| pthread_cond_signal(&scan_cond); | |||||
| pthread_mutex_unlock(&scan_mutex); | |||||
| } | |||||
| static const process_t *candidate(void) | static const process_t *candidate(void) | ||||
| { | { | ||||
| unsigned i; double now = monotime(); | unsigned i; double now = monotime(); | ||||
| @@ -413,17 +619,33 @@ static int metric_valid(enum metric_index metric, const sample_t *s) | |||||
| return 1; | return 1; | ||||
| } | } | ||||
| static void record_metrics(const sample_t *s) | |||||
| static void record_pressure_metrics(const sample_t *s) | |||||
| { | |||||
| if (metric_valid(METRIC_FREE, s)) | |||||
| history_add(&metrics[METRIC_FREE], s->at, s->free); | |||||
| if (metric_valid(METRIC_SWAP_PHYSICAL_USED, s)) | |||||
| history_add(&metrics[METRIC_SWAP_PHYSICAL_USED], s->at, | |||||
| s->swap_physical_used_bytes); | |||||
| else memset(&metrics[METRIC_SWAP_PHYSICAL_USED], 0, | |||||
| sizeof (metrics[METRIC_SWAP_PHYSICAL_USED])); | |||||
| if (metric_valid(METRIC_ANON_PAGEOUT, s)) | |||||
| history_add(&metrics[METRIC_ANON_PAGEOUT], s->at, s->anon_pageout_pages); | |||||
| else memset(&metrics[METRIC_ANON_PAGEOUT], 0, | |||||
| sizeof (metrics[METRIC_ANON_PAGEOUT])); | |||||
| } | |||||
| static void record_auxiliary_metrics(const sample_t *s) | |||||
| { | { | ||||
| uint64_t values[] = {s->free, s->kernel, s->locked, s->arc, | uint64_t values[] = {s->free, s->kernel, s->locked, s->arc, | ||||
| s->swap_alloc + s->swap_reserved, s->swap_physical_used_bytes, | s->swap_alloc + s->swap_reserved, s->swap_physical_used_bytes, | ||||
| s->anon_pageout_pages, s->scan, s->swap_available}; | s->anon_pageout_pages, s->scan, s->swap_available}; | ||||
| unsigned i; | unsigned i; | ||||
| for (i = 0; i < METRIC_COUNT; i++) { | for (i = 0; i < METRIC_COUNT; i++) { | ||||
| if (i == METRIC_FREE || i == METRIC_SWAP_PHYSICAL_USED || | |||||
| i == METRIC_ANON_PAGEOUT) continue; | |||||
| if (metric_valid((enum metric_index)i, s)) | if (metric_valid((enum metric_index)i, s)) | ||||
| history_add(&metrics[i], s->at, values[i]); | history_add(&metrics[i], s->at, values[i]); | ||||
| else | |||||
| memset(&metrics[i], 0, sizeof (metrics[i])); | |||||
| else memset(&metrics[i], 0, sizeof (metrics[i])); | |||||
| } | } | ||||
| } | } | ||||
| @@ -460,19 +682,16 @@ static void act(sample_t *sample) | |||||
| sample_t fresh; | sample_t fresh; | ||||
| engine_t current_engine; | engine_t current_engine; | ||||
| const char *source = "pending", *blocked_reason = "no_valid_cached_candidate"; | const char *source = "pending", *blocked_reason = "no_valid_cached_candidate"; | ||||
| double candidate_age = -1; | |||||
| unsigned i, rank = 0; | |||||
| int err, rc, validated = 0; | |||||
| double candidate_age = -1, decision_mono = 0, validation_start = 0; | |||||
| double validation_ms = 0, kill_attempt_mono = 0, kill_return_mono = 0; | |||||
| double dispatch_ms = 0, kill_ms = 0, enqueue_start, enqueue_ms; | |||||
| unsigned i, rank = 0, rejected = 0; | |||||
| int err = 0, rc = 0, validated = 0, rejected_pid = 0, rejected_errno = 0; | |||||
| if (stopping) return; | if (stopping) return; | ||||
| if (!engine.danger) { | if (!engine.danger) { | ||||
| (void)action_plan(&controller, &cfg, &engine, sample, NULL); | (void)action_plan(&controller, &cfg, &engine, sample, NULL); | ||||
| return; | return; | ||||
| } | } | ||||
| if (!log_ok) { | |||||
| emit_important(LOG_WARNING, | |||||
| "event=action_blocked reason=log_failure state=%s", | |||||
| state_name(engine.state)); return; | |||||
| } | |||||
| if (cfg.armed && monotime() - started < 60) { | if (cfg.armed && monotime() - started < 60) { | ||||
| emit_important(LOG_WARNING, | emit_important(LOG_WARNING, | ||||
| "event=action_blocked reason=armed_warmup state=%s elapsed_seconds=%.1f" | "event=action_blocked reason=armed_warmup state=%s elapsed_seconds=%.1f" | ||||
| @@ -497,33 +716,34 @@ retry_candidate: | |||||
| "event=action_blocked reason=%s state=%s", plan.reason, | "event=action_blocked reason=%s state=%s", plan.reason, | ||||
| state_name(current_engine.state)); return; | state_name(current_engine.state)); return; | ||||
| } | } | ||||
| if (platform_signal(&cfg, &plan.target, plan.action, 0, 0)) { | |||||
| decision_mono = validation_start = monotime(); | |||||
| if (platform_signal(&cfg, &plan.target, plan.action, 0, 0, NULL, NULL)) { | |||||
| err = errno; controller.has_pending = 0; | err = errno; controller.has_pending = 0; | ||||
| emit("event=action_rejected reason=identity_or_protection source=pending" | |||||
| " pid=%d errno=%d", plan.target.pid, err); | |||||
| rejected++; rejected_pid = plan.target.pid; rejected_errno = err; | |||||
| goto retry_candidate; | goto retry_candidate; | ||||
| } | } | ||||
| validation_ms = 1000 * (monotime() - validation_start); | |||||
| validated = 1; | validated = 1; | ||||
| } else { | } else { | ||||
| for (i = 0; i < CACHED_CANDIDATES; i++) { | for (i = 0; i < CACHED_CANDIDATES; i++) { | ||||
| cached_candidate_t *entry = &candidate_cache[i]; | cached_candidate_t *entry = &candidate_cache[i]; | ||||
| candidate_age = fresh.at - entry->scan_time; | candidate_age = fresh.at - entry->scan_time; | ||||
| if (!entry->valid || candidate_age < 0 || | if (!entry->valid || candidate_age < 0 || | ||||
| candidate_age > cfg.max_cached_candidate_age_s || | |||||
| protected_process(&cfg, &entry->p) || | protected_process(&cfg, &entry->p) || | ||||
| was_victim(&controller, &entry->p)) continue; | was_victim(&controller, &entry->p)) continue; | ||||
| plan = action_plan(&controller, &cfg, ¤t_engine, &fresh, &entry->p); | plan = action_plan(&controller, &cfg, ¤t_engine, &fresh, &entry->p); | ||||
| if (plan.action == NONE) { blocked_reason = plan.reason; break; } | if (plan.action == NONE) { blocked_reason = plan.reason; break; } | ||||
| if (platform_signal(&cfg, &plan.target, plan.action, 0, 0)) { | |||||
| decision_mono = validation_start = monotime(); | |||||
| if (platform_signal(&cfg, &plan.target, plan.action, 0, 0, NULL, NULL)) { | |||||
| err = errno; entry->valid = 0; | err = errno; entry->valid = 0; | ||||
| emit("event=action_rejected reason=identity_or_protection" | |||||
| " source=%s rank=%u pid=%d errno=%d", | |||||
| i == 0 ? "cached_candidate" : "fallback_candidate", | |||||
| i + 1, plan.target.pid, err); | |||||
| rejected++; rejected_pid = plan.target.pid; rejected_errno = err; | |||||
| continue; | continue; | ||||
| } | } | ||||
| validation_ms = 1000 * (monotime() - validation_start); | |||||
| rank = i + 1; | rank = i + 1; | ||||
| source = i == 0 ? "cached_candidate" : "fallback_candidate"; | |||||
| if (candidate_age > cfg.max_cached_candidate_age_s) | |||||
| source = "stale_cache_revalidated"; | |||||
| else source = i == 0 ? "cached_candidate" : "fallback_candidate"; | |||||
| validated = 1; break; | validated = 1; break; | ||||
| } | } | ||||
| if (!validated) { | if (!validated) { | ||||
| @@ -536,41 +756,73 @@ retry_candidate: | |||||
| return; | return; | ||||
| } | } | ||||
| } | } | ||||
| emit("event=action_decision state=%s candidate_source=%s rank=%u" | |||||
| " scan_complete=%d candidate_age_ms=%.1f victim_budget_override=%s" | |||||
| " pid=%d UUID=%s", state_name(current_engine.state), source, rank, | |||||
| scan_complete, candidate_age >= 0 ? 1000 * candidate_age : -1.0, | |||||
| plan.budget_override, plan.target.pid, plan.target.uuid); | |||||
| if (emit("event=action %s=%s reason=%s UUID=%s alias=%s zone_generation=%" PRIu64 | |||||
| " pid=%d start=%" PRId64 ".%09" PRId64 " comm=%s rss_kib=%" PRIu64 | |||||
| " free_pages=%" PRIu64 " victim_budget_override=%s", | |||||
| cfg.armed ? "ACTION_INTENT" : "WOULD_ACTION", plan.action == TERM ? "SIGTERM" : "SIGKILL", | |||||
| plan.reason, plan.target.uuid, alias_for(plan.target.uuid), plan.target.zone_generation, | |||||
| plan.target.pid, plan.target.start_sec, plan.target.start_nsec, plan.target.name, | |||||
| plan.target.rss_kib, fresh.free, plan.budget_override)) return; | |||||
| /* Consume a budget slot even if the subsequent signal write is ambiguous. */ | |||||
| action_commit(&controller, &cfg, &fresh, &plan); | |||||
| if (cfg.armed && !stopping) { | |||||
| if (cfg.armed && stopping) return; | |||||
| if (cfg.armed) { | |||||
| rc = platform_signal(&cfg, &plan.target, plan.action, 1, | rc = platform_signal(&cfg, &plan.target, plan.action, 1, | ||||
| !strcmp(plan.reason, "floor") ? fresh.min / 2 + 1 : fresh.des); err = errno; | |||||
| emit("event=action_result pid=%d signal=%s result=%s errno=%d", plan.target.pid, | |||||
| plan.action == TERM ? "SIGTERM" : "SIGKILL", rc ? "failed" : "sent", rc ? err : 0); | |||||
| syslog_event(plan.action == TERM ? LOG_ERR : LOG_CRIT, | |||||
| !strcmp(plan.reason, "floor") ? fresh.min / 2 + 1 : fresh.des, | |||||
| &kill_attempt_mono, &kill_return_mono); err = errno; | |||||
| if (!kill_attempt_mono) kill_attempt_mono = monotime(); | |||||
| if (!kill_return_mono) kill_return_mono = monotime(); | |||||
| dispatch_ms = 1000 * (kill_attempt_mono - decision_mono); | |||||
| kill_ms = 1000 * (kill_return_mono - kill_attempt_mono); | |||||
| } | |||||
| /* State and budgets are memory-only and are updated after the signal attempt. */ | |||||
| action_commit(&controller, &cfg, &fresh, &plan); | |||||
| if (cfg.armed && rc) controller.has_pending = 0; | |||||
| enqueue_start = monotime(); | |||||
| if (rejected) | |||||
| emit("event=action_rejected reason=identity_or_protection rejected_count=%u" | |||||
| " last_pid=%d last_errno=%d", rejected, rejected_pid, rejected_errno); | |||||
| if (cfg.armed) { | |||||
| emit_important(plan.action == TERM ? LOG_ERR : LOG_CRIT, | |||||
| "event=action_decision state=%s signal=%s result=%s UUID=%s alias=%s" | "event=action_decision state=%s signal=%s result=%s UUID=%s alias=%s" | ||||
| " pid=%d free_pages=%" PRIu64, | |||||
| " pid=%d free_pages=%" PRIu64 " candidate_source=%s rank=%u" | |||||
| " candidate_age_ms=%.1f candidate_validate_latency_ms=%.3f" | |||||
| " decision_mono=%.6f kill_attempt_mono=%.6f kill_return_mono=%.6f" | |||||
| " dispatch_latency_ms=%.3f action_dispatch_latency_ms=%.3f" | |||||
| " kill_latency_ms=%.3f", | |||||
| state_name(current_engine.state), | state_name(current_engine.state), | ||||
| plan.action == TERM ? "SIGTERM" : "SIGKILL", rc ? "failed" : "sent", | plan.action == TERM ? "SIGTERM" : "SIGKILL", rc ? "failed" : "sent", | ||||
| plan.target.uuid, alias_for(plan.target.uuid), plan.target.pid, fresh.free); | |||||
| plan.target.uuid, alias_for(plan.target.uuid), plan.target.pid, fresh.free, | |||||
| source, rank, candidate_age >= 0 ? 1000 * candidate_age : -1.0, | |||||
| validation_ms, decision_mono, kill_attempt_mono, kill_return_mono, | |||||
| dispatch_ms, dispatch_ms, kill_ms); | |||||
| if (!rc) | if (!rc) | ||||
| emit_important(plan.action == TERM ? LOG_ERR : LOG_CRIT, | emit_important(plan.action == TERM ? LOG_ERR : LOG_CRIT, | ||||
| "event=signal_sent state=%s signal=%s UUID=%s alias=%s pid=%d" | "event=signal_sent state=%s signal=%s UUID=%s alias=%s pid=%d" | ||||
| " comm=%s rss_kib=%" PRIu64 " free_pages=%" PRIu64, | |||||
| " comm=%s rss_kib=%" PRIu64 " free_pages=%" PRIu64 | |||||
| " decision_mono=%.6f kill_attempt_mono=%.6f kill_return_mono=%.6f" | |||||
| " dispatch_latency_ms=%.3f", | |||||
| state_name(current_engine.state), | state_name(current_engine.state), | ||||
| plan.action == TERM ? "SIGTERM" : "SIGKILL", plan.target.uuid, | plan.action == TERM ? "SIGTERM" : "SIGKILL", plan.target.uuid, | ||||
| alias_for(plan.target.uuid), plan.target.pid, plan.target.name, | alias_for(plan.target.uuid), plan.target.pid, plan.target.name, | ||||
| plan.target.rss_kib, fresh.free); | |||||
| if (rc) controller.has_pending = 0; | |||||
| plan.target.rss_kib, fresh.free, decision_mono, kill_attempt_mono, | |||||
| kill_return_mono, dispatch_ms); | |||||
| emit("event=action_result pid=%d signal=%s result=%s errno=%d", | |||||
| plan.target.pid, plan.action == TERM ? "SIGTERM" : "SIGKILL", | |||||
| rc ? "failed" : "sent", rc ? err : 0); | |||||
| } else { | |||||
| emit("event=action_decision state=%s candidate_source=%s rank=%u" | |||||
| " scan_complete=%d candidate_age_ms=%.1f victim_budget_override=%s" | |||||
| " pid=%d UUID=%s candidate_validate_latency_ms=%.3f", | |||||
| state_name(current_engine.state), source, rank, scan_complete, | |||||
| candidate_age >= 0 ? 1000 * candidate_age : -1.0, | |||||
| plan.budget_override, plan.target.pid, plan.target.uuid, validation_ms); | |||||
| emit("event=action WOULD_ACTION=%s reason=%s UUID=%s alias=%s" | |||||
| " zone_generation=%" PRIu64 " pid=%d start=%" PRId64 ".%09" PRId64 | |||||
| " comm=%s rss_kib=%" PRIu64 " free_pages=%" PRIu64 | |||||
| " victim_budget_override=%s", plan.action == TERM ? "SIGTERM" : "SIGKILL", | |||||
| plan.reason, plan.target.uuid, alias_for(plan.target.uuid), | |||||
| plan.target.zone_generation, plan.target.pid, plan.target.start_sec, | |||||
| plan.target.start_nsec, plan.target.name, plan.target.rss_kib, | |||||
| fresh.free, plan.budget_override); | |||||
| } | } | ||||
| enqueue_ms = 1000 * (monotime() - enqueue_start); | |||||
| emit("event=action_timing pid=%d signal=%s candidate_validate_latency_ms=%.3f" | |||||
| " kill_latency_ms=%.3f action_dispatch_latency_ms=%.3f" | |||||
| " log_enqueue_latency_ms=%.3f", plan.target.pid, | |||||
| plan.action == TERM ? "SIGTERM" : "SIGKILL", validation_ms, kill_ms, | |||||
| dispatch_ms, enqueue_ms); | |||||
| } | } | ||||
| static int acquire_lock(void) | static int acquire_lock(void) | ||||
| @@ -668,10 +920,14 @@ int main(int argc, char **argv) | |||||
| /* Touch all fixed working storage before allowing armed operation. */ | /* Touch all fixed working storage before allowing armed operation. */ | ||||
| memset(processes, 0, sizeof (processes)); memset(zone_stats, 0, sizeof (zone_stats)); | memset(processes, 0, sizeof (processes)); memset(zone_stats, 0, sizeof (zone_stats)); | ||||
| memset(candidate_cache, 0, sizeof (candidate_cache)); | memset(candidate_cache, 0, sizeof (candidate_cache)); | ||||
| memset(published_cache, 0, sizeof (published_cache)); | |||||
| if (logger_start()) { perror("logger thread initialization"); return 1; } | |||||
| if (scanner_start()) { perror("scanner thread initialization"); return 1; } | |||||
| if (emit_important(LOG_INFO, | if (emit_important(LOG_INFO, | ||||
| "event=start version=%s pid=%ld mode=%s page_bytes=%ld fixed_storage_bytes=%zu armed_warmup=60", | "event=start version=%s pid=%ld mode=%s page_bytes=%ld fixed_storage_bytes=%zu armed_warmup=60", | ||||
| VERSION, (long)getpid(), cfg.armed ? "armed" : "dry-run", sysconf(_SC_PAGESIZE), | VERSION, (long)getpid(), cfg.armed ? "armed" : "dry-run", sysconf(_SC_PAGESIZE), | ||||
| sizeof (processes) + sizeof (zone_stats) + sizeof (candidate_cache))) return 1; | |||||
| sizeof (processes) + sizeof (zone_stats) + sizeof (candidate_cache) + | |||||
| sizeof (published_cache) + sizeof (event_queue))) return 1; | |||||
| if (syslog_init_failed) | if (syslog_init_failed) | ||||
| emit("event=internal_error reason=syslog_initialization_failed continuation=guard_log_only"); | emit("event=internal_error reason=syslog_initialization_failed continuation=guard_log_only"); | ||||
| if (readyfd >= 0) { | if (readyfd >= 0) { | ||||
| @@ -681,16 +937,6 @@ int main(int argc, char **argv) | |||||
| if (fd >= 0) { (void)dup2(fd, STDIN_FILENO); (void)dup2(fd, STDOUT_FILENO); if (fd > 2) close(fd); } | if (fd >= 0) { (void)dup2(fd, STDIN_FILENO); (void)dup2(fd, STDOUT_FILENO); if (fd > 2) close(fd); } | ||||
| } | } | ||||
| while (!stopping) { | while (!stopping) { | ||||
| if (reopen_log) { | |||||
| reopen_log = 0; | |||||
| if (logfd >= 0) close(logfd); | |||||
| if (log_open()) { | |||||
| log_ok = 0; | |||||
| syslog_event(LOG_CRIT, | |||||
| "event=internal_error reason=log_reopen_failed actions=inhibited"); | |||||
| } | |||||
| emit("event=log_reopen configuration=unchanged"); | |||||
| } | |||||
| if (platform_sample(&sample) || monotime() - sample.at > 2) { | if (platform_sample(&sample) || monotime() - sample.at > 2) { | ||||
| int err = errno; | int err = errno; | ||||
| engine_gap(&engine); controller.has_pending = 0; scan_complete = 0; | engine_gap(&engine); controller.has_pending = 0; scan_complete = 0; | ||||
| @@ -700,46 +946,44 @@ int main(int argc, char **argv) | |||||
| } else { | } else { | ||||
| enum state previous = engine.state; | enum state previous = engine.state; | ||||
| char transition[512] = ""; | char transition[512] = ""; | ||||
| record_metrics(&sample); | |||||
| int acted = 0; | |||||
| refresh_candidate_cache(); | |||||
| /* A retained rescue state gets the first opportunity to act. */ | |||||
| if (engine.danger) { act(&sample); acted = 1; } | |||||
| record_pressure_metrics(&sample); | |||||
| engine_step(&engine, &cfg, &sample, | engine_step(&engine, &cfg, &sample, | ||||
| history_delta(&metrics[METRIC_FREE], sample.at, 5), | history_delta(&metrics[METRIC_FREE], sample.at, 5), | ||||
| history_delta(&metrics[METRIC_SWAP_PHYSICAL_USED], sample.at, 5), | history_delta(&metrics[METRIC_SWAP_PHYSICAL_USED], sample.at, 5), | ||||
| history_delta(&metrics[METRIC_ANON_PAGEOUT], sample.at, 5)); | history_delta(&metrics[METRIC_ANON_PAGEOUT], sample.at, 5)); | ||||
| if (engine.state != previous && | |||||
| !transition_message(transition, sizeof (transition), previous, | |||||
| engine.state, &sample, engine.low_samples)) | |||||
| emit("%s", transition); | |||||
| log_sample(&sample); | |||||
| if (previous != RECOVERY && engine.state == RECOVERY) | |||||
| emit("event=recovery_check free_pages=%" PRIu64 | |||||
| " target=desfree target_pages=%" PRIu64 " reached=%d", | |||||
| sample.free, sample.des, sample.free >= sample.des); | |||||
| else if (engine.state == RECOVERY) | |||||
| emit("event=recovery_check free_pages=%" PRIu64 | |||||
| " target=lotsfree target_pages=%" PRIu64 " stable_seconds=%.1f", | |||||
| sample.free, sample.lots, engine.recovery_since >= 0 ? | |||||
| sample.at - engine.recovery_since : 0.0); | |||||
| /* CRITICAL/EMERGENCY is O(1): never wait for a full /proc walk. */ | |||||
| if (engine.danger) { | |||||
| act(&sample); | |||||
| } else { | |||||
| if (engine.danger && !acted) act(&sample); | |||||
| if (!engine.danger) { | |||||
| /* Cancel pending escalation/reset episode budgets before rebuilding cache. */ | /* Cancel pending escalation/reset episode budgets before rebuilding cache. */ | ||||
| act(&sample); | act(&sample); | ||||
| if (sample.free < sample.lots || sample.at - last_scan >= cfg.normal_scan_s) { | if (sample.free < sample.lots || sample.at - last_scan >= cfg.normal_scan_s) { | ||||
| int snapshot = sample.at - last_snapshot >= (sample.free < sample.lots ? 5 : 60); | int snapshot = sample.at - last_snapshot >= (sample.free < sample.lots ? 5 : 60); | ||||
| scan_processes(snapshot); last_scan = sample.at; | |||||
| request_scan(snapshot); last_scan = sample.at; | |||||
| if (snapshot) last_snapshot = sample.at; | if (snapshot) last_snapshot = sample.at; | ||||
| if (sample.free < sample.lots && !engine.danger) (void)candidate(); | if (sample.free < sample.lots && !engine.danger) (void)candidate(); | ||||
| } | } | ||||
| } | } | ||||
| /* Critical notifications are deferred until after the O(1) action path. */ | |||||
| record_auxiliary_metrics(&sample); | |||||
| /* All telemetry is queued only after the rescue path has returned. */ | |||||
| if (engine.state != previous) | |||||
| (void)transition_message(transition, sizeof (transition), previous, | |||||
| engine.state, &sample, engine.low_samples); | |||||
| if (transition[0]) | if (transition[0]) | ||||
| syslog_event(state_priority(engine.state, engine.floor), "%s", transition); | |||||
| emit_important(state_priority(engine.state, engine.floor), "%s", transition); | |||||
| log_sample(&sample); | |||||
| if (previous != RECOVERY && engine.state == RECOVERY) | if (previous != RECOVERY && engine.state == RECOVERY) | ||||
| syslog_event(LOG_INFO, | |||||
| emit_important(LOG_INFO, | |||||
| "event=recovery_check phase=entered free_pages=%" PRIu64 | "event=recovery_check phase=entered free_pages=%" PRIu64 | ||||
| " target=desfree target_pages=%" PRIu64 " reached=%d", | " target=desfree target_pages=%" PRIu64 " reached=%d", | ||||
| sample.free, sample.des, sample.free >= sample.des); | sample.free, sample.des, sample.free >= sample.des); | ||||
| else if (engine.state == RECOVERY) | |||||
| emit("event=recovery_check phase=stabilizing free_pages=%" PRIu64 | |||||
| " target=lotsfree target_pages=%" PRIu64 " stable_seconds=%.1f", | |||||
| sample.free, sample.lots, engine.recovery_since >= 0 ? | |||||
| sample.at - engine.recovery_since : 0.0); | |||||
| if (previous == RECOVERY && engine.state == NORMAL) | if (previous == RECOVERY && engine.state == NORMAL) | ||||
| emit_important(LOG_INFO, | emit_important(LOG_INFO, | ||||
| "event=recovery_complete state=NORMAL free_pages=%" PRIu64 | "event=recovery_complete state=NORMAL free_pages=%" PRIu64 | ||||
| @@ -751,7 +995,7 @@ int main(int argc, char **argv) | |||||
| log_sampling_overrun(monotime() - next, &sample); | log_sampling_overrun(monotime() - next, &sample); | ||||
| next = monotime() + 1; | next = monotime() + 1; | ||||
| } | } | ||||
| while (!stopping && !reopen_log && monotime() < next) { | |||||
| while (!stopping && monotime() < next) { | |||||
| double remain = next - monotime(); struct timespec delay; | double remain = next - monotime(); struct timespec delay; | ||||
| if (remain <= 0) break; | if (remain <= 0) break; | ||||
| delay.tv_sec = (time_t)remain; delay.tv_nsec = (long)((remain - delay.tv_sec) * 1e9); | delay.tv_sec = (time_t)remain; delay.tv_nsec = (long)((remain - delay.tv_sec) * 1e9); | ||||
| @@ -759,9 +1003,6 @@ int main(int argc, char **argv) | |||||
| } | } | ||||
| } | } | ||||
| emit("event=stop"); | emit("event=stop"); | ||||
| platform_syslog_close(); syslog_ready = 0; | |||||
| platform_fini(); | |||||
| if (logfd >= 0) close(logfd); | |||||
| if (lockfd >= 0) close(lockfd); | |||||
| /* Detached workers may be blocked in kernel I/O; process exit owns cleanup. */ | |||||
| return 0; | return 0; | ||||
| } | } | ||||
| @@ -0,0 +1,8 @@ | |||||
| seconds,free_pages,nscan_pages,event | |||||
| 0,48000,1,emergency_term_consultaerte | |||||
| 6,305511,0,recovery_1170_mib | |||||
| 20,48000,1,emergency_term_base0 | |||||
| 31,48000,1,kill_decision | |||||
| 176,148619,1,kill_sent_after_145s | |||||
| 177,148619,0,partial_recovery | |||||
| 185,49077,1,relapse_no_valid_candidate | |||||
| @@ -40,4 +40,13 @@ if grep -Eq '(system|popen|execl|execv)[[:space:]]*\(' src/support.c; then | |||||
| echo 'FAIL syslog backend must not invoke commands or a shell' >&2 | echo 'FAIL syslog backend must not invoke commands or a shell' >&2 | ||||
| exit 1 | exit 1 | ||||
| fi | fi | ||||
| grep -Fq '#define CACHED_CANDIDATES 16' src/main.c | |||||
| grep -Fq '#define EVENT_QUEUE_SIZE 512' src/main.c | |||||
| grep -Fq 'pthread_mutex_trylock(&event_mutex)' src/main.c | |||||
| grep -Fq 'pthread_mutex_trylock(&scan_mutex)' src/main.c | |||||
| signal_line=$(grep -n 'rc = platform_signal' src/main.c | cut -d: -f1) | |||||
| commit_line=$(grep -n 'action_commit(&controller' src/main.c | cut -d: -f1) | |||||
| enqueue_line=$(grep -n 'enqueue_start = monotime' src/main.c | cut -d: -f1) | |||||
| test "$signal_line" -lt "$commit_line" | |||||
| test "$commit_line" -lt "$enqueue_line" | |||||
| echo 'PASS distribution defaults to ACTION_MODE dry-run; production SMF does not force it' | echo 'PASS distribution defaults to ACTION_MODE dry-run; production SMF does not force it' | ||||
| @@ -12,6 +12,7 @@ typedef struct { int priority; char message[1536]; } syslog_record_t; | |||||
| static syslog_record_t syslog_records[64]; | static syslog_record_t syslog_records[64]; | ||||
| static unsigned syslog_count, syslog_open_count, syslog_close_count; | static unsigned syslog_count, syslog_open_count, syslog_close_count; | ||||
| static int syslog_facility, fail_syslog; | static int syslog_facility, fail_syslog; | ||||
| static double syslog_delay_seconds; | |||||
| static char syslog_ident[64]; | static char syslog_ident[64]; | ||||
| #define CHECK(x) do { if (!(x)) { fprintf(stderr, "FAIL runtime line %d: %s\n", __LINE__, #x); exit(1); } checks++; } while (0) | #define CHECK(x) do { if (!(x)) { fprintf(stderr, "FAIL runtime line %d: %s\n", __LINE__, #x); exit(1); } checks++; } while (0) | ||||
| @@ -30,6 +31,7 @@ int platform_syslog_open(const char *ident, int facility) | |||||
| int platform_syslog_write(int priority, const char *message) | int platform_syslog_write(int priority, const char *message) | ||||
| { | { | ||||
| if (fail_syslog) { errno = EIO; return -1; } | if (fail_syslog) { errno = EIO; return -1; } | ||||
| fake_now += syslog_delay_seconds; | |||||
| CHECK(syslog_count < sizeof (syslog_records) / sizeof (syslog_records[0])); | CHECK(syslog_count < sizeof (syslog_records) / sizeof (syslog_records[0])); | ||||
| syslog_records[syslog_count].priority = priority; | syslog_records[syslog_count].priority = priority; | ||||
| snprintf(syslog_records[syslog_count].message, | snprintf(syslog_records[syslog_count].message, | ||||
| @@ -48,14 +50,18 @@ int platform_walk(int (*fn)(const process_t *, void *), void *arg, unsigned budg | |||||
| return partial ? -1 : 0; | return partial ? -1 : 0; | ||||
| } | } | ||||
| int platform_signal(const config_t *c, const process_t *p, enum action action, | int platform_signal(const config_t *c, const process_t *p, enum action action, | ||||
| int real, uint64_t ceiling) | |||||
| int real, uint64_t ceiling, double *attempt_mono, double *return_mono) | |||||
| { | { | ||||
| unsigned j; int found = 0; | unsigned j; int found = 0; | ||||
| CHECK(action == TERM || action == KILL); | CHECK(action == TERM || action == KILL); | ||||
| CHECK(!protected_process(c, p)); | CHECK(!protected_process(c, p)); | ||||
| for (j = 0; j < nfixtures; j++) | for (j = 0; j < nfixtures; j++) | ||||
| if (fixtures[j].pid == p->pid && same_process(&fixtures[j], p)) { found = 1; break; } | if (fixtures[j].pid == p->pid && same_process(&fixtures[j], p)) { found = 1; break; } | ||||
| if (real) { CHECK(c->armed && fake_sample.free < ceiling); real_requests++; } | |||||
| if (real) { | |||||
| CHECK(c->armed && fake_sample.free < ceiling); real_requests++; | |||||
| if (attempt_mono) *attempt_mono = fake_now; | |||||
| if (return_mono) *return_mono = fake_now; | |||||
| } | |||||
| else validation_requests++; | else validation_requests++; | ||||
| if (fail_validation || p->pid == invalid_pid || !found) { errno = ESTALE; return -1; } | if (fail_validation || p->pid == invalid_pid || !found) { errno = ESTALE; return -1; } | ||||
| return 0; /* This test backend cannot send any signal. */ | return 0; /* This test backend cannot send any signal. */ | ||||
| @@ -70,14 +76,21 @@ static void reset(void) | |||||
| strcpy(cfg.zones[1].alias, "web-b"); cfg.zones[1].policy = ALLOW; | strcpy(cfg.zones[1].alias, "web-b"); cfg.zones[1].policy = ALLOW; | ||||
| memset(processes, 0, sizeof (processes)); memset(zone_stats, 0, sizeof (zone_stats)); | memset(processes, 0, sizeof (processes)); memset(zone_stats, 0, sizeof (zone_stats)); | ||||
| memset(candidate_cache, 0, sizeof (candidate_cache)); | memset(candidate_cache, 0, sizeof (candidate_cache)); | ||||
| memset(published_cache, 0, sizeof (published_cache)); | |||||
| memset(metrics, 0, sizeof (metrics)); | memset(metrics, 0, sizeof (metrics)); | ||||
| memset(&controller, 0, sizeof (controller)); memset(fixtures, 0, sizeof (fixtures)); | memset(&controller, 0, sizeof (controller)); memset(fixtures, 0, sizeof (fixtures)); | ||||
| engine_init(&engine); engine.state = CRITICAL; engine.danger = 1; | engine_init(&engine); engine.state = CRITICAL; engine.danger = 1; | ||||
| stopping = 0; started = 0; log_ok = 1; log_bytes = 0; | stopping = 0; started = 0; log_ok = 1; log_bytes = 0; | ||||
| async_logging = scanner_async = 0; | |||||
| event_head = event_tail = event_count = dropped_events = 0; | |||||
| event_sequence = 0; scan_requested = scan_snapshot_requested = 0; | |||||
| syslog_ready = syslog_init_failed = 0; | syslog_ready = syslog_init_failed = 0; | ||||
| syslog_count = syslog_open_count = syslog_close_count = 0; | syslog_count = syslog_open_count = syslog_close_count = 0; | ||||
| syslog_facility = -1; fail_syslog = 0; syslog_ident[0] = 0; | syslog_facility = -1; fail_syslog = 0; syslog_ident[0] = 0; | ||||
| scan_generation = 0; scan_complete = 0; latest_complete_scan_at = 0; | |||||
| syslog_delay_seconds = 0; | |||||
| scan_generation = 0; scan_complete = published_scan_complete = 0; | |||||
| scanner_latest_complete_scan_at = latest_complete_scan_at = 0; | |||||
| published_latest_complete_scan_at = 0; | |||||
| fake_now = 100; partial = fail_sample = fail_validation = invalid_pid = 0; | fake_now = 100; partial = fail_sample = fail_validation = invalid_pid = 0; | ||||
| real_requests = validation_requests = walk_requests = 0; | real_requests = validation_requests = walk_requests = 0; | ||||
| memset(&fake_sample, 0, sizeof (fake_sample)); fake_sample.at = fake_now; | memset(&fake_sample, 0, sizeof (fake_sample)); fake_sample.at = fake_now; | ||||
| @@ -91,6 +104,128 @@ static void reset(void) | |||||
| scan_processes(0); | scan_processes(0); | ||||
| } | } | ||||
| static void populate_many_candidates(unsigned count) | |||||
| { | |||||
| unsigned i; | |||||
| reset(); | |||||
| CHECK(count <= sizeof (fixtures) / sizeof (fixtures[0])); | |||||
| for (i = 0; i < count; i++) { | |||||
| fixtures[i] = fixtures[0]; fixtures[i].pid = 1000 + (int)i; | |||||
| fixtures[i].start_sec += i; fixtures[i].start_nsec += i; | |||||
| fixtures[i].rss_kib = 2000000 - 1000 * i; | |||||
| } | |||||
| nfixtures = count; fake_now = 101; scan_processes(0); | |||||
| for (i = 0; i < count; i++) CHECK(candidate_cache[i].valid); | |||||
| } | |||||
| static int queued_contains(const char *text) | |||||
| { | |||||
| unsigned i; | |||||
| for (i = 0; i < event_count; i++) { | |||||
| unsigned slot = (event_head + i) % EVENT_QUEUE_SIZE; | |||||
| if (strstr(event_queue[slot].body, text)) return 1; | |||||
| } | |||||
| return 0; | |||||
| } | |||||
| static void critical_path_faults(void) | |||||
| { | |||||
| struct timespec before, after; double elapsed_ms; unsigned walks; | |||||
| struct stat st_before, st_after; | |||||
| { | |||||
| FILE *fixture; char line[160], event[64]; | |||||
| unsigned seconds, rows = 0, decision_at = 0, sent_at = 0; | |||||
| uint64_t free_pages, nscan; | |||||
| fixture = fopen("tests/fixtures/panic32_daemon_stall_2026_09_22.csv", "r"); | |||||
| CHECK(fixture != NULL); CHECK(fgets(line, sizeof (line), fixture) != NULL); | |||||
| while (fscanf(fixture, "%u,%" SCNu64 ",%" SCNu64 ",%63s", | |||||
| &seconds, &free_pages, &nscan, event) == 4) { | |||||
| rows++; | |||||
| if (!strcmp(event, "kill_decision")) decision_at = seconds; | |||||
| if (!strcmp(event, "kill_sent_after_145s")) sent_at = seconds; | |||||
| } | |||||
| fclose(fixture); | |||||
| CHECK(rows == 7 && sent_at - decision_at == 145); | |||||
| } | |||||
| { | |||||
| queued_event_t event; | |||||
| reset(); async_logging = 1; syslog_ready = 1; | |||||
| syslog_delay_seconds = 0.200; | |||||
| memset(&event, 0, sizeof (event)); | |||||
| strcpy(event.body, "event=test_syslog_stall"); | |||||
| event.wall_time = time(NULL); event.mono_time = fake_now; | |||||
| event.syslog_channel = 1; event.priority = LOG_WARNING; event.sequence = 1; | |||||
| CHECK(write_event_now(&event) == 0); | |||||
| CHECK(queued_contains("event=logging_stall")); | |||||
| CHECK(queued_contains("channel=syslog")); | |||||
| CHECK(queued_contains("syslog_latency_ms=200.000")); | |||||
| } | |||||
| populate_many_candidates(16); cfg.armed = 1; engine.state = EMERGENCY; | |||||
| engine.danger = 1; fake_sample.free = 40000; fake_now += 180; | |||||
| async_logging = 1; fail_syslog = 1; | |||||
| CHECK(fstat(logfd, &st_before) == 0); walks = walk_requests; | |||||
| clock_gettime(CLOCK_MONOTONIC, &before); act(&fake_sample); | |||||
| clock_gettime(CLOCK_MONOTONIC, &after); | |||||
| elapsed_ms = 1000.0 * (after.tv_sec - before.tv_sec) + | |||||
| (after.tv_nsec - before.tv_nsec) / 1000000.0; | |||||
| CHECK(real_requests == 1 && validation_requests == 1); | |||||
| CHECK(elapsed_ms < 100.0 && walk_requests == walks && syslog_count == 0); | |||||
| CHECK(fstat(logfd, &st_after) == 0 && st_after.st_size == st_before.st_size); | |||||
| CHECK(queued_contains("candidate_source=stale_cache_revalidated")); | |||||
| CHECK(queued_contains("dispatch_latency_ms=")); | |||||
| CHECK(queued_contains("candidate_validate_latency_ms=")); | |||||
| CHECK(queued_contains("kill_latency_ms=")); | |||||
| CHECK(queued_contains("log_enqueue_latency_ms=")); | |||||
| fake_now += cfg.emergency_term_s; | |||||
| clock_gettime(CLOCK_MONOTONIC, &before); act(&fake_sample); | |||||
| clock_gettime(CLOCK_MONOTONIC, &after); | |||||
| elapsed_ms = 1000.0 * (after.tv_sec - before.tv_sec) + | |||||
| (after.tv_nsec - before.tv_nsec) / 1000000.0; | |||||
| CHECK(real_requests == 2 && validation_requests == 2 && elapsed_ms < 100.0); | |||||
| CHECK(queued_contains("signal=SIGKILL")); | |||||
| puts("PASS panic32 fault: logger/syslog stopped 180s, stale cache revalidated, signal dispatch <100ms"); | |||||
| populate_many_candidates(16); cfg.armed = 1; engine.state = EMERGENCY; | |||||
| engine.danger = 1; fake_sample.free = 40000; async_logging = scanner_async = 1; | |||||
| scan_requested = 1; walks = walk_requests; | |||||
| act(&fake_sample); | |||||
| CHECK(real_requests == 1 && walk_requests == walks); | |||||
| puts("PASS stalled/incomplete scanner cannot delay or trigger a rescue-path proc scan"); | |||||
| populate_many_candidates(16); cfg.armed = 1; engine.state = EMERGENCY; | |||||
| engine.danger = 1; fake_sample.free = 40000; async_logging = 1; | |||||
| walks = walk_requests; | |||||
| while (controller.nvictims < cfg.max_emergency_victims) { | |||||
| act(&fake_sample); | |||||
| if (controller.has_pending) { fake_now += cfg.emergency_term_s; act(&fake_sample); } | |||||
| fake_now += 0.01; | |||||
| } | |||||
| CHECK(controller.nvictims == 8 && real_requests == 16 && walk_requests == walks); | |||||
| act(&fake_sample); CHECK(real_requests == 16); | |||||
| puts("PASS 16-entry stale rescue cache supplies all 8 EMERGENCY victims without scan"); | |||||
| populate_many_candidates(16); cfg.armed = 1; engine.state = EMERGENCY; | |||||
| engine.danger = engine.floor = 1; fake_sample.free = 20000; async_logging = 1; | |||||
| walks = walk_requests; | |||||
| while (controller.nvictims < cfg.max_floor_victims) { | |||||
| act(&fake_sample); fake_now += 0.01; | |||||
| } | |||||
| CHECK(controller.nvictims == 12 && real_requests == 12 && walk_requests == walks); | |||||
| act(&fake_sample); CHECK(real_requests == 12); | |||||
| puts("PASS 16-entry stale rescue cache supplies all 12 FLOOR victims without scan"); | |||||
| populate_many_candidates(16); cfg.armed = 1; engine.state = EMERGENCY; | |||||
| engine.danger = 1; fake_sample.free = 40000; async_logging = 1; | |||||
| CHECK(pthread_mutex_lock(&event_mutex) == 0); | |||||
| clock_gettime(CLOCK_MONOTONIC, &before); act(&fake_sample); | |||||
| clock_gettime(CLOCK_MONOTONIC, &after); | |||||
| CHECK(pthread_mutex_unlock(&event_mutex) == 0); | |||||
| elapsed_ms = 1000.0 * (after.tv_sec - before.tv_sec) + | |||||
| (after.tv_nsec - before.tv_nsec) / 1000000.0; | |||||
| CHECK(real_requests == 1 && elapsed_ms < 100.0 && dropped_events > 0); | |||||
| puts("PASS saturated/locked event queue drops telemetry without delaying the signal"); | |||||
| } | |||||
| static void load_global_ranking_fixture(void) | static void load_global_ranking_fixture(void) | ||||
| { | { | ||||
| FILE *f; | FILE *f; | ||||
| @@ -157,7 +292,8 @@ int main(void) | |||||
| fake_sample.swap_available = 6200000 - second; | fake_sample.swap_available = 6200000 - second; | ||||
| fake_sample.swap_physical_used_bytes = UINT64_C(4143648768) + 4096 * second; | fake_sample.swap_physical_used_bytes = UINT64_C(4143648768) + 4096 * second; | ||||
| fake_sample.anon_pageout_pages = 1000 + 2 * second; | fake_sample.anon_pageout_pages = 1000 + 2 * second; | ||||
| record_metrics(&fake_sample); | |||||
| record_pressure_metrics(&fake_sample); | |||||
| record_auxiliary_metrics(&fake_sample); | |||||
| } | } | ||||
| log_sample(&fake_sample); | log_sample(&fake_sample); | ||||
| CHECK(fflush(log) == 0 && fseek(log, 0, SEEK_SET) == 0); | CHECK(fflush(log) == 0 && fseek(log, 0, SEEK_SET) == 0); | ||||
| @@ -182,13 +318,13 @@ int main(void) | |||||
| n = fread(contents, 1, sizeof (contents) - 1, log); contents[n] = 0; | n = fread(contents, 1, sizeof (contents) - 1, log); contents[n] = 0; | ||||
| CHECK(strstr(contents, "event=action_blocked reason=armed_warmup") != NULL); | CHECK(strstr(contents, "event=action_blocked reason=armed_warmup") != NULL); | ||||
| CHECK(fseek(log, 0, SEEK_END) == 0); | CHECK(fseek(log, 0, SEEK_END) == 0); | ||||
| reset(); cfg.armed = 1; log_ok = 0; act(&fake_sample); CHECK(real_requests == 0); | |||||
| reset(); cfg.armed = 1; log_ok = 0; act(&fake_sample); CHECK(real_requests == 1); | |||||
| reset(); cfg.armed = 1; partial = 1; fake_now++; scan_processes(0); | reset(); cfg.armed = 1; partial = 1; fake_now++; scan_processes(0); | ||||
| act(&fake_sample); CHECK(real_requests == 1 && !scan_complete); | act(&fake_sample); CHECK(real_requests == 1 && !scan_complete); | ||||
| reset(); cfg.armed = 1; fail_sample = 1; act(&fake_sample); CHECK(real_requests == 0); | reset(); cfg.armed = 1; fail_sample = 1; act(&fake_sample); CHECK(real_requests == 0); | ||||
| reset(); cfg.armed = 1; fail_validation = 1; act(&fake_sample); CHECK(real_requests == 0); | reset(); cfg.armed = 1; fail_validation = 1; act(&fake_sample); CHECK(real_requests == 0); | ||||
| reset(); cfg.armed = 1; fake_sample.free = 150000; act(&fake_sample); CHECK(real_requests == 0); | reset(); cfg.armed = 1; fake_sample.free = 150000; act(&fake_sample); CHECK(real_requests == 0); | ||||
| reset(); cfg.armed = 1; fake_now += 16; act(&fake_sample); CHECK(real_requests == 0); | |||||
| reset(); cfg.armed = 1; fake_now += 16; act(&fake_sample); CHECK(real_requests == 1); | |||||
| reset(); cfg.armed = 1; stopping = 1; act(&fake_sample); CHECK(real_requests == 0); | reset(); cfg.armed = 1; stopping = 1; act(&fake_sample); CHECK(real_requests == 0); | ||||
| reset(); fake_sample.free = 20000; act(&fake_sample); | reset(); fake_sample.free = 20000; act(&fake_sample); | ||||
| CHECK(!controller.has_pending && controller.nvictims == 1); | CHECK(!controller.has_pending && controller.nvictims == 1); | ||||
| @@ -311,7 +447,7 @@ int main(void) | |||||
| syslog_open_count == 1); | syslog_open_count == 1); | ||||
| emit_important(LOG_INFO, "event=start version=%s mode=dry-run", VERSION); | emit_important(LOG_INFO, "event=start version=%s mode=dry-run", VERSION); | ||||
| CHECK(syslog_count == 1 && syslog_records[0].priority == LOG_INFO && | CHECK(syslog_count == 1 && syslog_records[0].priority == LOG_INFO && | ||||
| strstr(syslog_records[0].message, "event=start version=1.0.6") != NULL); | |||||
| strstr(syslog_records[0].message, "event=start version=1.0.7") != NULL); | |||||
| log_sample(&fake_sample); scan_processes(0); | log_sample(&fake_sample); scan_processes(0); | ||||
| CHECK(syslog_count == 1); /* High-volume sample/delta/proc/cache events stay local. */ | CHECK(syslog_count == 1); /* High-volume sample/delta/proc/cache events stay local. */ | ||||
| { | { | ||||
| @@ -362,6 +498,7 @@ int main(void) | |||||
| n = fread(contents, 1, sizeof (contents) - 1, log); contents[n] = 0; | n = fread(contents, 1, sizeof (contents) - 1, log); contents[n] = 0; | ||||
| CHECK(strstr(contents, "event=sampling_overrun late_seconds=4.459 state=CRITICAL free_pages=60000") != NULL); | CHECK(strstr(contents, "event=sampling_overrun late_seconds=4.459 state=CRITICAL free_pages=60000") != NULL); | ||||
| puts("PASS selective native syslog ident/facility/severity, exclusions, signal ordering, disable and failure isolation"); | puts("PASS selective native syslog ident/facility/severity, exclusions, signal ordering, disable and failure isolation"); | ||||
| critical_path_faults(); | |||||
| fclose(log); | fclose(log); | ||||
| printf("PASS %u runtime checks; all platform calls mocked, no signals sent\n", checks); | printf("PASS %u runtime checks; all platform calls mocked, no signals sent\n", checks); | ||||
| return 0; | return 0; | ||||