From 7812e2ec696a3ce83601d13c17b919a42cdf4998 Mon Sep 17 00:00:00 2001 From: Guzman Date: Thu, 12 Mar 2026 15:50:57 +0100 Subject: [PATCH 1/4] Document sensor-to-central message flow architecture Diagram made by @guzalv, text written by Claude after several "deep dive" sessions looking into the architecture with the author. Co-authored-by: Claude --- AGENTS.md | 5 + central/sensor/service/connection/README.md | 149 ++++++++++++++++++ .../sensor-central-data-flow.excalidraw.svg | 4 + 3 files changed, 158 insertions(+) create mode 100644 central/sensor/service/connection/README.md create mode 100644 central/sensor/service/connection/sensor-central-data-flow.excalidraw.svg diff --git a/AGENTS.md b/AGENTS.md index 016880fbcf31c..da62b9e767c91 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -149,6 +149,11 @@ StackRox is a Kubernetes-native security platform with a distributed microservic ### Detailed Documentation +**In-code documentation:** Packages with non-obvious architecture or operational +gotchas often contain a markdown file (e.g. `README.md`) co-located with the +source code. Before researching a subsystem from scratch, check for markdown +files in its package directory. + When working on specific areas, refer to these detailed guides: **Operator Development:** diff --git a/central/sensor/service/connection/README.md b/central/sensor/service/connection/README.md new file mode 100644 index 0000000000000..926618167ea8b --- /dev/null +++ b/central/sensor/service/connection/README.md @@ -0,0 +1,149 @@ +# Sensor-to-Central Message Flow + +![Data flow diagram](sensor-central-data-flow.excalidraw.svg) + +## Overview + +Central receives events from N Sensor clusters concurrently. Each cluster gets a fully +isolated set of queues and workers, so a slow or misbehaving cluster cannot stall others. +Within a cluster, messages are routed through two levels of queuing before reaching a +pipeline fragment that performs validation, enrichment, and datastore writes. + +## Architecture + +### Layer 1 — gRPC stream (one per cluster) + +[`central/sensor/service/service_impl.go`](../service_impl.go) `Communicate()` accepts the +bidirectional gRPC stream opened by Sensor. +[`connection/manager_impl.go`](manager_impl.go) `HandleConnection()` wraps it in a +`sensorConnection` struct that owns all state for that cluster. + +`runRecv()` in [`connection_impl.go`](connection_impl.go) is the read loop: it pulls +`MsgFromSensor` off the gRPC stream and forwards each message into the first queue. + +### Layer 2 — Per-message-type DedupingQueue + +`multiplexedPush()` in [`connection_impl.go`](connection_impl.go) routes every incoming +message into a `DedupingQueue` keyed by `MsgFromSensor` type. There are ~16 distinct +message types (Event, NetworkFlowUpdate, ClusterHealthInfo, ComplianceResponse, …). Queues +are allocated lazily on first use; a dedicated goroutine drains each one via +`handleMessages()`. + +The per-type fan-out is primarily about **category isolation**: a surge of one message type +(e.g. `Event` during initial cluster sync) cannot stall processing of another (e.g. +`NetworkFlowUpdate`). These queues use `DedupingQueue`, so messages carrying a `DedupeKey` +are collapsed (see [Deduplication](#deduplication) for which messages set this key and when). + +**Gotcha:** only `Event` messages continue into further queuing (Layers 3–4). All other +types (NetworkFlow, Compliance, Telemetry, …) are handled inline in `handleMessage()` +without further queuing. + +### Layer 3 — Per-resource-type workerQueue (Events only) + +[`sensorevents.go`](sensorevents.go) `addMultiplexed()` receives a decoded `SensorEvent` +and routes it to a `workerQueue` keyed by resource type (Deployment, Pod, Node, …). Up to +~30 workerQueues per `sensorConnection`, allocated lazily. + +Before pushing, the content-hash deduper (`deduper.ShouldProcess()`) decides whether this +event carries new information or can be dropped as a duplicate (see +[Content-hash deduplication](#content-hash-deduplication-layer-3)). + +### Layer 4 — Sharded DedupingQueues inside each workerQueue + +Each `workerQueue` contains **17** internal `DedupingQueue` slots and 17 goroutines — one +per slot. Slot 0 handles messages with no `HashKey`; slots 1–16 are sharded by +`FNV32(HashKey) % 16 + 1`. Because all messages for the same entity (same ID) always land +in the same slot, processing is serialized per entity without a global lock. + +This design solves two problems at once: different entities are processed **in parallel** +across slots, while updates to the **same** entity are **serialized** within one slot +(preventing lost-update races in the pipeline). As with Layer 2, these are `DedupingQueue`s, +so messages with a matching `DedupeKey` are collapsed before they reach the pipeline. + +**Gotcha (queue count):** the total number of goroutines and queues grows multiplicatively: + +``` +N clusters × M resource types seen × 17 workers +``` + +For 10 clusters each sending 15 resource types that is 2 550 goroutines just for workers, +plus Layer-2 queues and their goroutines on top. + +### Layer 5 — Pipeline fragments + +[`central/sensor/service/pipeline/all/pipeline.go`](../pipeline/all/pipeline.go) `Run()` +finds the fragment matching the resource type and runs it. Each fragment handles +validation, enrichment, and datastore writes. For deployments this includes SAC checks, +risk scoring, network baseline updates, and policy re-evaluation. + +## Deduplication + +Two independent deduplication mechanisms operate on the message flow. Both are optional and +apply to different subsets of messages. + +### DedupingQueue replace-in-place semantics + +[`pkg/dedupingqueue/deduping_queue.go`](../../../../pkg/dedupingqueue/deduping_queue.go) +is used at both Layer 2 and Layer 4. When a message is pushed with a non-empty `DedupeKey` +that matches an already-queued item, the queue **replaces the old item in-place** (the new +item takes the old item's queue position). This means only the latest state of an entity is +ever processed — intermediate updates are silently dropped. + +When `DedupeKey` is empty (the zero value), the queue behaves as a plain FIFO — every push +appends without deduplication. + +#### Which messages carry a DedupeKey + +For the vast majority of traffic — regular Sensor Events arriving from the gRPC stream — no +`DedupeKey` is set when messages enter Layer 2, so those queues behave as plain FIFOs. +The `DedupeKey` is assigned later, in `handleMessage()`, via `shallDedupe()`, which means it +only takes effect at Layer 4. The rules are: + +- **No dedup** (key = empty): `CREATE` actions; and for non-`REMOVE` actions: + `NodeInventory`, `IndexReport`, `VirtualMachine`, `VirtualMachineIndexReport`. +- **Dedup on entity ID**: `UPDATE` and `REMOVE` actions for all other resource types + (Pod, Deployment, Namespace, etc.). + +`shallDedupe()` in [`connection_impl.go`](connection_impl.go) controls this logic. + +Two special cases carry a `DedupeKey` **before** reaching Layer 2, so deduplication occurs +at both layers: + +- **`AuditLogStatusInfo`** — Sensor sets `DedupeKey = clusterID` before sending + (`sensor/common/compliance/auditlog_manager_impl.go`). Multiple audit-log state + snapshots queued before the goroutine drains them collapse to the latest, avoiding + redundant syncs. +- **`ReprocessDeployment`** — Central's reprocessor (`central/reprocessor/reprocessor.go`) + injects synthetic Event messages with `DedupeKey = UUIDv5(riskDedupeNamespace, + deploymentID)`. Because the reprocessor also sets `HashKey` and the message type is + `ReprocessDeployment`, `handleMessage()` short-circuits before `shallDedupe()`, + preserving the key through both layers. This collapses redundant risk-reprocessing + requests for the same deployment. + +All other non-Event message types (`NetworkFlowUpdate`, `ComplianceResponse`, +`TelemetryDataResponse`, …) carry no `DedupeKey` and are therefore never deduplicated. + +### Content-hash deduplication (Layer 3) + +Before an Event enters a `workerQueue`, `deduper.ShouldProcess()` +([`central/hash/manager/deduper.go`](../../../hash/manager/deduper.go)) checks the +content hash of the event. The hash is normally **computed by Sensor** and sent as +`SensorHash`; Central only computes it as a backward-compatibility fallback when +`SensorHashOneof` is nil. If the same entity was already processed with an identical hash, +the message is dropped. + +This deduplication state is periodically flushed to the `hashes` postgres table and +reloaded on Central restart, so it can survive restarts. However, persistence is +**conditional**: when the `ROX_HASH_FLUSH_INTERVAL` environment variable is set to `0`, +Central truncates the `hashes` table on startup and does not flush — effectively disabling +cross-restart deduplication. + +## Key Code Locations + +- [central/sensor/service/service_impl.go](../service_impl.go) — gRPC entry point `Communicate():88` +- [central/sensor/service/connection/connection_impl.go](connection_impl.go) — `sensorConnection` struct; `multiplexedPush():171`; `handleMessage():342`; `shallDedupe()` +- [central/sensor/service/connection/sensorevents.go](sensorevents.go) — Layer-3 queue dispatch `addMultiplexed():80` +- [central/sensor/service/connection/worker_queue.go](worker_queue.go) — 17-slot sharded queue; FNV hashing `indexFromKey():51` +- [central/hash/manager/deduper.go](../../../hash/manager/deduper.go) — content-hash deduplication `ShouldProcess()` +- [pkg/dedupingqueue/deduping_queue.go](../../../../pkg/dedupingqueue/deduping_queue.go) — replace-in-place queue semantics +- [central/sensor/service/pipeline/all/pipeline.go](../pipeline/all/pipeline.go) — pipeline fragment dispatch `Run():64` diff --git a/central/sensor/service/connection/sensor-central-data-flow.excalidraw.svg b/central/sensor/service/connection/sensor-central-data-flow.excalidraw.svg new file mode 100644 index 0000000000000..0bb309cddfd13 --- /dev/null +++ b/central/sensor/service/connection/sensor-central-data-flow.excalidraw.svg @@ -0,0 +1,4 @@ + + +eyJ2ZXJzaW9uIjoiMSIsImVuY29kaW5nIjoiYnN0cmluZyIsImNvbXByZXNzZWQiOnRydWUsImVuY29kZWQiOiJ4nO19aVfq2rbt9/MrbOt+ea+1be6si/1NUbGu63dus1GLXCIghYi3nf/+xkCFQFxuklx1MDAxMFx1MDAxNdnk7LbOWlx1MDAwNmJmMmafvc9R/e+/Vlb+dPrN0p+/V/6UXlx1MDAwYrlatdjK9f78hT9/KbXa1UZcdTAwMWROscG/241uqzD45H2n02z//d//PfqGU2g8vX+rVCs9leqdNnzu/8G/V1b+d/AnnKlcdTAwMTbxu5erJ2v7nG+fZ/ZNM3+9rzLna9XBV1x1MDAwN1x1MDAxZvq8mVap0MnVK7XS6NQr/JxcdTAwMGLuaFxyf2o5/HlcdTAwMWZ+vsq0dohWzHIrJKdsdLpXLXbu4SOUXG7qfJ4lcLDhR+5L1cp9XHUwMDA3PmONcqjSVlx1MDAxOT34XGJcdTAwMWR+5P1e/l4hw5+0O63GYynTqDVaeMP/RUv4v9Ht5nOFx0qr0a1cdTAwMTdHnynpsiyXR58pV2u1s05/cGV4vvAs/0xcXP/q4+7ZxM+Dvlx1MDAwNb+wcl8vtfH5j+6+0cxcdTAwMTWqXHUwMDFkfFCUjEaAd9fcKVx1MDAwZV7V/4zuqZV7Ku3gu6p3a7Xhj6v1Ylx038CfXHUwMDFjXHUwMDE5+2314sdv+3zPo5fIP37yn9G9l0p4YS41t0wwMTwxsjVKzeRPXHUwMDBmXHUwMDFi9YHdMVwiXHUwMDA1N1x1MDAwNl7Q6K7aXHUwMDFiYHCdwVXLuVq7NHr8eGubk8boNsgxe+uUXjvDcbnMtZBcdTAwMTGbd+S6snp+s56/bmSL/e554c/wc//5+Nvo8XWbxdz7/VCtmVx1MDAxNFZbMFc+PF+r1lx1MDAxZiefba1RePRcdTAwMTlCu1x1MDAwZuMurqFh/mGEqVXC4b9zKv5cdTAwMTbsb8pcdTAwMWPJ1a3LXobTs9xq1DulevFcdTAwMGUv8GfscufVp1K7k3tqXHUwMDA2XpPq2/fhfby1ienr/zw803fscX7MXFxcdTAwMTM0c9mUmauMw1x1MDAxNU5HRVx0l1Qq78R1fSuVmdpp5ertZq5cdTAwMDXG89tn66X/dFx1MDAxZPv4x7yUknKlXHUwMDE4Zz7zUtCgaak4oUTBm00yK79k4lxmrFx1MDAwZlx1MDAwNp+B39LK1Vxcb1x1MDAxMCbGWfXtXHUwMDFkTMZ+upV7qtbwkcuxq6zVqlx1MDAxNVx1MDAxY/2fWqnsslx1MDAwNHhcdTAwMDKdKqx9w9OdRnN0tlx1MDAwMNfLVeul1k6UXHUwMDE1r9GqVqr1XFztPOimc91O47TUfr/tTqtbcj+S0vbnXHUwMDE0oI5rXHUwMDEyzCN0mJ3S3YPgR5nV4mbmuf7Syp7urEZf+SWH30A1NZNcdTAwMDBcdTAwMDJG61CqXHUwMDE4JdZcci6f6MGAMkjDyFx1MDAxOLRcZoGDwlmi0oaPxVjo+cxcdTAwMGK9oZxcYmO0XHUwMDFmoFx1MDAwNOJcdLwqZph0TdAvX+WfyNNcdTAwMTYv9lx1MDAxZatH5nCtfZpcdTAwMTO9t0Letcr/5X/Z9y+vXz62zzbrq/lM6a2ePym2brqF6/Hf8vn7c61Wo1x1MDAxN4c9cGGB71D3jP/N7MH/OUdgXHUwMDBm8CBcdTAwMWPFmeaMW1xu9FGMQ4CQwiFGWOVcdTAwMGZcdTAwMDFcdTAwMDaIv1x1MDAwMP7ArdLMSPpcclx1MDAwNGIxXHUwMDEwQETnXHUwMDBl1Fx1MDAxYVivXHLw+jhzXHUwMDFkcJtwoahkSSZ78LxhXHUwMDE21lx0I5PMm0/ycFaqt1x1MDAxYq1cdTAwMTWaXHUwMDA2eyjAXHUwMDFkl1oh/OGpWiy6V7xcdFxuMWXpnKRcdTAwMTDeW19cdTAwMTRcdTAwMGVxffKaedvQh9dPJ6tcdTAwMTenLV2ud2o2ModQgjlMXHUwMDEw5qNCkEQg901IXCKko7TUhi5hxFx1MDAwZkbkzESCamWEtlIr91x1MDAxY5qKLsLCXHUwMDFik0aNlosvp1x1MDAxMk22cXJw2t3sXHUwMDFm6053/yp72+NX11GpxFpho69y+budNcVcdTAwMGLiYjNX3uifpEUlJNFAolx1MDAxN4VK+D/nXGJUQjNcdTAwMDJUgjJcdTAwMDFcdTAwMDYlKKN0XHUwMDFjXGJcdTAwMDToXGbCXGafmOtDKqFcdTAwMWODW1x1MDAxMZzAXHLijF9Sifd7moZcdTAwMDEqXHUwMDA2lZCcM2nd0zbKZJdEXG5cIlx1MDAxOEmkXHUwMDFiwqiEMMQkouBcdTAwMTNUgs1cdTAwMDOVmLKCXHUwMDA2UFx0tnhUonpfXju1tVx1MDAxYmY2mkd9vnV0/2oqkalcdTAwMDQlwFxijFx1MDAxY2xHMlx1MDAwZpdgjpbEajV2blx1MDAxMblEMde+L30vkOjZyVx1MDAwNOhEaqhcdTAwMTFcdTAwMWVPQ7hUXHUwMDAxhWMliMzvI1x1MDAxMyeX9dZ9w1x1MDAxNFdzrcPdvbvOa+u4RKOSiVxyazR96N7kePeK0uOXRmv19CklMiEoXHUwMDA341x1MDAxNu6p/5vJhP9zjkAmKGXMsVxmSFx1MDAwMyHMMqvsJJtgXHUwMDBlPCtipP/GhHRcdTAwMDRuTDB4oGJsXHUwMDFicskmQkHARGdcdTAwMTNKWYbbPvEmO+FGK5CJiSZ7XHUwMDE4mVBIw1MgXHUwMDEzh/NAJqasoVx1MDAwMWTicPHIRH3tcb17Ue7tllx1MDAxZSp3O+x2/TV73PNcdTAwMDJcYlx1MDAwZWpcdTAwMWNALIPlnnI2tty/Q1x1MDAwN9dcdTAwMGVRlnN/6NDEYcrKXHUwMDAwXHUwMDEy4Wg2Tj7mXHUwMDE2O4qNTuebXHUwMDE5hI3KIFggg2BgZJaxmFxmgktl4UhcdTAwMWZUNOVCu6dOVFBpNqqTXHUwMDA0ZfS3lZG1XGb+Mfz7//zl+2lcdTAwMWaTxGN1aI2jXHUwMDBieFxiRi3X7mRcdTAwMWFPT1W0hmO8K1x1MDAwZpx3cq3OOrzFar0yeVx1MDAwZWZ1wJnBt9aQ3tyXclx1MDAxZZuA71x1MDAwNZ77MZhxvZxavzIwqcFLXHUwMDBigZ/9m8Jz8dZS2uvfXFyeVLdcdTAwMWL6JJ+NrGUkXHUwMDEzXHUwMDBlWjRlZrCrMTLq91x1MDAwMFxyZlx1MDAxY2qktpRQX0FcdTAwMDO/XHUwMDE3XHUwMDA0jTWAWFx1MDAwMu17xIGGgGSYdoA7XHUwMDAzhVaEMcpGMyBcdTAwMWRgKpfLtphPXHUwMDE3mL5f2mSjXHUwMDAyU7C0UVJcdTAwMThBYnpcXEHWwKvVavTiv1xc2VxcUr1Xv1q9aGyK1cP1l1bBiJeHWFxuRGnOzcJsZ/o/j1xiXG5EMlx1MDAxYj55efjkZVxuPlx1MDAwMFx1MDAxYURIqlx1MDAxNPmW4KrFkCDb0SWI1Vxm1mdcdTAwMTVPgnBJXHS+05TJXHUwMDAy51x1MDAxNoxEJZk341xuJNOo12E9gTteSVOSzFx1MDAxMGg1ZVx1MDAxNfRXI6HDWFx1MDAxNHlcdTAwMTK+XHUwMDE1NLFccjRcdTAwMDYvlFx1MDAxYeaAXHUwMDFh5oxcdTAwMWLDOFFcdTAwMTP4wlx1MDAxNepcdTAwMTcwLMBcdTAwMTimyWhHaFx1MDAxOHbNlaOFXHUwMDAxXmgolUz5kFx1MDAwM0aBf8CliVx1MDAwNZRcdTAwMTJcZlxmdMlcdTAwMGX8XHUwMDAwJyCSM45uMUbCM4bFXCJcdTAwMGVcdTAwMTJJQSRcdTAwMTecJfKiXHUwMDA2I5GRSlx1MDAxMpZcYolSlS2rgSaKh9c401Qxrnf3kThcdTAwMTFlY1x1MDAwNT791ChcdTAwMGXssNHKV91cdTAwMDRcdTAwMGJs+3X0q8dcdTAwMWWKo2CEhFx1MDAxMW6k1ED41F9hZ71ibWRNY3IraFxiT+zuwrz1K3eH5Pm4/XS8/rR2Xpp1XGKSWIBcdTAwMWFccm+CXG5uxt9cdTAwMTRxhCXo0jXwXCKlosyGXGYhli6cXHUwMDA0xlIt3+j9XHUwMDEyYllcXKudXHUwMDFmnF1cdTAwMGLTypzeNOu7NzXNi17oXHUwMDBmkIbwkFx1MDAxZG2kUJZIXHUwMDFmdklB8ltAfYOB+ZJz7cV/K7ybUSNZSKUjXGJcdTAwMTdUc/lLZOH3XHUwMDEzzZ1cdTAwMTSiZ2BpXHUwMDA2sVx1MDAxNC80T8Jsolayb1SFrFxcqOxedM71iyXXmUwmK15PVVx1MDAxY1WoXHUwMDAwwIVdmHhZ/+dcdTAwMTFBXHUwMDE1XG7Mk1x1MDAwYpm3wNqmzFumhCOXgS7xp+tunEBcdTAwMTfMuOHaxFxudFHoaUw5yEVozrSwSebNNFWYStTLXGaqcMpcdTAwMDJcdTAwMThVXHUwMDE1LmBcdTAwMDRMeLRhmCrUljqGK82JIEyriWh8rrijmDBAJTnFjScvusAnXHUwMDE4M5Rrqoyh7kjtXHUwMDExujBM+CHEcGIoo4qnjTZcdTAwMGJBXHUwMDBldlNcdTAwMTCFwsJCQayMR1x1MDAwZoRGMWnT3p+aXHUwMDE3VVx1MDAxOGiheHht0yM1vkBcdTAwMTSGh+4lVVQ4j1x1MDAxNVx1MDAwMTnIMX3mr7CT1DPK+dCEozx9PODVgIhcdTAwMDc5XHUwMDAx9sSpXHUwMDExJZApITf+j1GC4Vx1MDAwZn9q/qXljpB6PGrxnUxy6lx1MDAxMMa1NspcdTAwMTBgM6Pdu2hcdTAwMWGQXHUwMDEw+DpRwEaXXHUwMDFhMFxi5vdm14Cg51x1MDAxOKbQxcvFZJh+aHWi9KzhzcXSgGtPeveEN5p9slNVje1K7qJ4vvdcclx0XHUwMDE0odf9yljKXHUwMDE5Vrx5hFx1MDAxOf/3XHUwMDE3QbNya4JcdTAwMTCGqSlcYsOkdYCLXHUwMDEybpdRlNFRZT+GVEXssMSdXVx1MDAxMcmHXHTcklqTaFx1MDAwYulLZsw0tZpKuudcZmp1ylwiXHUwMDFkVa0uYOpn5rJQ7lrF38yhvO49ZW+UvFx1MDAxN5Hpi1x1MDAxNsSBtVxmKIhvmIRcIlx1MDAwZafAcYVcdTAwMTG+6Z9cdTAwMDBAXHUwMDEzWaNDaNEghdV4WYolcVx1MDAxOULM0ezEhTNcdTAwMDX6g8TcvbaGS4XVJ76PucizfvX4rVfU/Wzz5ny3y25f8uJcdTAwMWKqSIRet7T3cHKr8sV2NkduatvXV4eV3kFcbtc9vt542JL1yutRs93eejvNtVv72Vx1MDAxNK57ojKl8tWOqVx1MDAxNJvHXHUwMDBm2fZhpbp2Xk2JaVx0So1cdTAwMTQjO/rlTMvf3lwiMC1DtEOM0cJYXHUwMDEwyO48qI/9O8dcdTAwMDDP0lx1MDAwMbnwljmKYdKKXHUwMDE2apDH9lxydGshsPA4Ot3SXHUwMDAy9DZI4lhsXHUwMDBiU+Q1sSTlOFxyjmaSLHH8k221uvXTUuHl//zfNNjVrEkrU4jEJMXyufdFoVTha06YXHUwMDAzQFx1MDAwMutcdTAwMDFLXHUwMDAzPiX9/ItcXFx1MDAxOcdYYpSdSGtzR51Okq0henBcIlx1MDAxZE/27Fx1MDAxMkeGOHJcdTAwMTKVU1x1MDAwNW/5U1wiJSdWxYpItVx1MDAwMtORiEnZ9/he4yrRqpzqjr+PTeLhtsbRJTxcXCO1Xf7wWj8ryfbItVGKW4kxborT0Vx1MDAwNorfWVx1MDAxNbZbXHUwMDFlaZs/XHUwMDFjYVx1MDAxM1x1MDAwZVx1MDAwMW5ccuiIpVx1MDAxNMDFsFFcdTAwMDTj+1lcdTAwMDGrJbBcdTAwMTJjYFx1MDAwMFRcdTAwMWKrQsbwj9nxz1/cZYvZq+PVbOHwrr55Ur/f7USvliT4YNdcdTAwMWWELfeRzFpidlwiQFx1MDAwMjPWXHUwMDFm4jl3XGZcdTAwMWZkXHUwMDA1XHJqKfvxRFxyy1x1MDAwNFXiK1xuMObL+VLpl1x1MDAwM/3pzOJcdTAwMTlrXHUwMDFmsbHyalx1MDAxMXDeXHUwMDEwhbXcklx1MDAxNVJcdTAwMTneWyzpXFzPr5+z68xLfr1ZIK36y71aK99ElYyP3dfe9eHry/bD3u7jJVdcdTAwMTeqYyd+S7JN/6uLvYunyrY4O+vuZLd2XHUwMDFle5nNm1x1MDAxNK57fUsyZ1dcdTAwMWQq1/b2b1x1MDAxZW5uz/onpctcdTAwMTSuO4PU9yxnXulMLF2cdCt/e4sgnTGwTiil0S02qFx1MDAxNjdcdTAwMWX7Qlx0dSycXHUwMDE0drKgw1x1MDAxMFx1MDAxNilcdTAwMTlUs4ZlimnLXFy5XHUwMDBlS/VcdTAwMWNcboZn0dUzI4pqpWi8uDp4XHUwMDFm8DWqSMreXG5u4FUnq+H6qZ83X+DXrGyUit0mMK+TbqnrXtp+TElP4Vx1MDAxN5NKOnRcdTAwMTSLoqk7m28nlyftl/t27vT2/ohXL8+LMjLn0kQ5SjHBXHUwMDE4NcSPczFguHbJub5cdTAwMGVmzmd3WMBCKbS18ZKsKNGMYVx04290WJxcdTAwMTZcdTAwMGYuN2+Kr4Wt+/37m96VbD6a529wXHUwMDAwTGVcdTAwMWJKXGKhXHUwMDE3JyTC/zlHYFx1MDAxYlpYh1n0PH44J8fgwFhcdTAwMDewwNhArkGFcSSWq1x1MDAwNOZGiVx1MDAxND5cXEOSlEFgIbjGRZzAXGKhjJLu1Ncos1x1MDAxZNgjyCx3q41UuMZcZlx1MDAxM+eTa1x1MDAxY5Y6W7VG72Jw5X/X5450TFlgJ0nH2HBcdTAwMTadfFx1MDAxY/TLLbtuVuWV1TfN4nW/drBbjkw+KNBmR9BA9iEwZ8gwSa1/n56FY1x1MDAxZj+Q5Fx1MDAxZDmcP5B+WFx1MDAxMDFcdTAwMDJcdTAwMDR7rK19ZbnmhiXb2k9GPlxur2190d9vNmzuaWet93B7YFx1MDAwZraXUVx1MDAwMul37PJ9zlx1MDAxMchcdTAwMDcwUmy8XHUwMDA1s1oyXCKJO1x1MDAwNfCDfXBtQ0plU2tcdTAwMWQrgHxoXHJ/WOmDXHUwMDA2S/bhh1x1MDAwMVfR2YcxXHUwMDA0INnGJFx1MDAxZsxcbitx9yld8pHCRlx1MDAwN1x1MDAxZF+hV5ql1r/rXHUwMDA37cpWq/H0XHUwMDEx2jgw2DngIVPW2klcdTAwMWXiM7KVsIEtXG4lyb10ds/3bePkqvp2fnq08XpZ2Tzwwo9PZUzgXHUwMDFiVGKxXHUwMDA3YUD78PHKupRqR0hcdTAwMDMjYsZQYCwjdPnEXHUwMDFmKVx1MDAwMX5cdTAwMTixXHUwMDFhg1x1MDAxNDg1PurnXHUwMDE3wM9cdTAwMGbUx7yOykGC41x1MDAwYrBoKcPUsXiiSMH3gFx1MDAxZc7PXHUwMDA2bKpcdTAwMDFcdTAwMDaBXHUwMDA2OfFtXHUwMDBmQ1lcdTAwMTbHnKU45t314fHe3dPpyfP64V22/XyiV1+6MbzgzCGGcqVcdTAwMDXWkZ7QRCh5pIC3yVx1MDAxOFx1MDAwYqj2/6OiqGzKpFxc+OV86CYqXHUwMDFlhbVcdTAwMGbSTJuAjsOBeFx1MDAwNMyWYaeAb6z4n1dcdTAwMDePN4XGeuU6e9h/3u6frp4+bvy0I/yrXHUwMDFj1runzef2hiDV+sVzt70rmtv2fj+F63b2Lo6v869cdTAwMTf3b+3CjiquZ0q3XHUwMDBmaTjui8WnxuPl9Wnr7KJ2mbk6OCWbV/lo1/Vgukd1XHUwMDFhXHRr38J0VPC341xiqlMo6Vx1MDAxMEZhtsKyjVUzR5NvXHUwMDAwuIpcdTAwMDDv49jJSvpcdTAwMTc0pUo7Vlx1MDAxMbiGXHLMOV562P1w9tZcdTAwMWZnfXe9tVx1MDAwNF7OaDzhyZjQRFKasu5cdTAwMTToXHUwMDA1STZ1PnXncaO40mu0XHUwMDFlS6252eeeQlsm9WXgXGJcdTAwMTZFSFx1MDAxZb+tPXdBQ1x1MDAxNjeet2rqKdNbv+757GNcdTAwMDU61rVDKMajY0VcdTAwMGI/XHUwMDFhp5VgZknjvlx1MDAwZV5ys9M4ySg3cT3rXGbrtHCVUFYmo3FZ0tnsrauDLjneb1x1MDAxZr6VXHUwMDBm61x1MDAwN1dcdTAwMDc/XHUwMDFkd3jBry/XqubopSHzjd5Fdm+78HqYwnVfVq2s3qx2ykdXN3uXrcZL1rS2U7huuXR6ftg8O66fXHUwMDFmXHUwMDE0j/Y3+rXzaqWWXHUwMDE23bLoVF+YXGJcdTAwMDN/e4tAt7QwXHUwMDBlU1xuyFx1MDAxNtHAqsh45Vx1MDAwNVx1MDAwNmephj+D2ZakXHUwMDBl59ZahXXsMCrJi4vLXX4/OMzHYFvUMKs0j1x1MDAxOc9oKS5YabfWNlx1MDAwMkvwzlQq8PJgXHUwMDA3XHUwMDFmw2mp2Wh1/l2fM+I1hWhMXHUwMDEyr7HBLDZcdTAwMDUr3bcvc9Xsereynz3avFeWNNhVZFxuNlxiL1x1MDAwMD1cdTAwMTfCwYDKXHUwMDBiSifbYS0sXHUwMDA3+4H4gsLMJFxmpJaSNm4/Pa6sXHUwMDEw3H5nMeHNveKLPOk81+8r7bvM6ct6/7bX/OmtNHncfljPPD/svFx1MDAxZF+fiMPs5uvx4V1cbtd9O+ucnW3n9OPdvr14vTy/XHUwMDEytdJaXG7X3Xu6fCtcdTAwMTdeuqerjaOmXrvafNq6SavwlSRcXHKzMIFcdTAwMTb+9lx1MDAxNoGDUWKJg601XHUwMDE4MjDh7lx1MDAxM/NOwrSDpZypYFx1MDAxM22BhyRMK1x1MDAwN8s0MvnZSXRJwt7vaVx1MDAxYVx1MDAxY1x1MDAxNuOQMEmBKceOtSDaaMKS1aFcdNvzklryXHUwMDE5Yy1cXGTlPdLiPVx1MDAxMuE9T2Ne4iymkFx1MDAwZW+cxcSoVoJcdTAwMDe1MLysly13Kt2X41K1X9i9u34s7Fx1MDAxNu+jxlhISrhcdTAwMDJCpulkjFx1MDAwNdPSXHUwMDAxmUxcdTAwMTlcdTAwMWNSXHQ2mrc/XHUwMDE5YvFcdTAwMDU07PtDLEpRaVhIK1x1MDAxZqtcZjWExOJhjFx1MDAxMmPhvSbjYWEhXHUwMDE2glx1MDAwYtdcdTAwMTKzXGaxcH9r4UMswtlyWCFcdTAwMTlcdTAwMDWakCnQdMKvUVx1MDAwNZWIT5YqOVnZ81x1MDAxM4G4lU5AXHUwMDE5XHUwMDE5IEVUiPFQ9fnFoO+XguWZMVxiXHUwMDAze4Vk8aIqQNxzTsB6U0Ygy1x1MDAxOaGJtqVSRSCPQeLhMsU0MSio/kp4Xu5KsvorIN5cdKg1gj2subRcdTAwMTOV4lx1MDAxNdGYeGBcdTAwMDUnRFx0zyBjXHUwMDE2kFx053xcdFx1MDAwN6Bg6dNcdTAwMTJcdTAwMTRcdTAwMTblUlx1MDAwMJn/a2J4XHUwMDE0WDpCkVx1MDAxMtKosPZn/5j6MeGOqND6YFx1MDAwNmA9sP9cdTAwMTCVzNGYI+sp2OzOXFxcdTAwMTRgTsr4XCI7TLHJ6oRzi+zfL2orM1x1MDAwMztwKKYx0TyW1DWMXHUwMDBig41cIlNHdkvZz4fv+tkkXHUwMDFlLmucK3Cv1tvVYikqOMK6XHUwMDA1yzlcdTAwMTdY8Vx1MDAxZp75ODhcdTAwMGXatnOYzFxcXHUwMDEwYVRYca1I+Fx1MDAxZe5iSYjvzDBtNdO4XHUwMDA1hlx1MDAwMWR/hZ01S3yfXHUwMDE211x1MDAxOYrv2Fx1MDAwMIdTa6hcdTAwMDWEN1x1MDAxM6lhSNtcdTAwMTlWI4V5XHUwMDBiM0a6XHUwMDEykkfJYciSXHUwMDE0Y1ZTTsC8/GDeOExjdFx1MDAxN3zKMumK4F+C/Vx1MDAxMOzvZ1x1MDAwNnt4fUbYmFx0o1RcZkjg14Ty/Xz3p2DzxMNjmHOF+7FIvZJaXHUwMDAxoFx1MDAwYqxw6SpcdTAwMTP3flbBcsAph1x1MDAxN1xy6G8k91xmMybsh4c0Jlx1MDAxY8Jk+yfvLs8/XHUwMDEw2Vc3tjf05Vopo17l8e164/agtXboRXbvjjDlRjvUcKNAJIGNi/Hwa8VcdTAwMTRMXGZM0aKcauLqtDp00lvn3YnlR9xcdTAwMTlzPEHZc+uN+oFt4erMYK5cciZrm5hgrlxypVx1MDAwML9pd1x1MDAxNVVcdTAwMTRj8Eev+scq+3ptXHUwMDEyj1WXOaaJ4Mut4XdcdTAwMTN638pcdC1KXHUwMDE52mRUiLA9XHUwMDA0jYtcdTAwMTeXanKL91x1MDAxM4i0ciyjvjAklVx1MDAwM4u6XFzuXHUwMDFm+KPQw+zOKSUlaC0qYlx1MDAxNkXSXHUwMDEycSPtXHUwMDA2XHUwMDA2M9QuTVx1MDAxNYZWPSaJh8tcdTAwMTi/g0Z+SW1uarGtXHUwMDEy0fhcdTAwMDalNlx1MDAxM243x2AjeWGxVFxivIzZ24h+XHRcdTAwMTVWRGNJLs1cclx1MDAwMS00Slx1MDAxNvA7u+wn+ve0XHUwMDAygGHIbrAundKWcYLqaVx1MDAxNDUwXHUwMDAwMOGgx88y+E9cdTAwMThGiLfbXHUwMDFmdShRXHUwMDE4OmasVSBcXFxcXHUwMDBlqFx1MDAxMcZjcTyLPSpcZlx1MDAwMXZD5egzS6hcdTAwMWZC/WNcblCPjYCs5jH3ii0xhlOVNuNcdTAwMWPUv1x1MDAxYlnMj0F9sIni4TXOX1x1MDAwYv2T8ntmcFx1MDAwZq/vl/QuLVx1MDAxN4JTqYjgkomxXHUwMDA1mDhcdTAwMDJelVRGcFx1MDAwYnRcdTAwMTKIZpj/8lx1MDAxZlx1MDAwM+/hJdbC4N1q7OiMXHUwMDFkU/yaXHUwMDAzwa/WWGU7kLhb4XjSjYawroTDl/H9wYBem1x1MDAxOdBhXHUwMDFlXGKYXHUwMDA3Mlx1MDAxNp5LmDRcdTAwMWGbTaTN3JlcdTAwMTXk511/PiaJh8tcdTAwMTh/LX4rYlB+XGIgYkpTPbZMXHKiXCIwX0dqwTkzzGedionu4VXTXHUwMDEyjkHASlxuz0WhuFx1MDAxMnoy6lx1MDAwZsBcdTAwMWbOc8WxhKGEUSzRXHUwMDFkwHZ1x1xcbJfsY77AyVU729g9rm550T2oKJKxXHUwMDBlo1x1MDAwMN7ErzWQXHUwMDAwXG5cdTAwMGa3XHUwMDA0yyyl441xh1x1MDAwNN5cYkdcdTAwMTksXHUwMDEyXHUwMDA21Fx1MDAxMVx1MDAwZp/AYVxmXHUwMDEyXHUwMDAx7vRFUVx1MDAxZbac+/VQL6JifXBCPTFMMFx1MDAxYrdPhtEghFWyrt7Dm4uVy1XcqGa3WHmtUs4+N87pSea5enM7XHUwMDBmZXuwXHUwMDExOU3msJxHWPB/zlx1MDAxZVjwKdtjpUM+0GA8SWmAXGJKOlx1MDAxMi2N2ok92VFcblx1MDAxM3dcYlwiXHUwMDAxMYNcbubyW1KYXHUwMDE2g/JcdTAwMTl/XHUwMDFj8O+MXHUwMDAzT5gybmNV0NCDKr9umEhHrFx1MDAwZuZOXCKx7qrbM1GiXHUwMDFlru16oT+WtTRlffWr3lx1MDAxMzyMRclTXG5cdTAwMDfi8HgjrMKoXHTIdFx1MDAwYkJDjnulOeWOxiBcdTAwMDUgf9ZSb7SRkFx1MDAwZceCx9ixllhtfcDFcodjwFx1MDAxMrY7ZMSF6EvK4YKazOzyXHUwMDEy1Fx1MDAwNjC6eFx1MDAwMESNpFxu41RS3i6EK1x1MDAxYZYsizJVfVx1MDAxOWifeExapke+fIHU/JJIXHUwMDFkLrixRFOKvSVA2Y+rNFjyKcG6XHUwMDAyXHUwMDE2OKnRM+8jhiNwwiEsg43w/yZgfffhQt5cXOcvL9r982r5+fS+t7+7XHUwMDE3XUtcbutQmIagzX12XHUwMDBihVx1MDAwMsuwQChJsJaET7xvwpCgqiC/S0t+v1x1MDAxYqiWjYrrwVKSUcI48a9cYumJRlx1MDAxYXU9I1x1MDAwMlZt4cLgL9eS91lSfF7vP5xcdTAwMTfyzbt6/5BcdTAwMTNyVv+G0rJcdTAwMWW0nlxcjTCzwLCFKVx1MDAwMev/nKNoSSlcdTAwMWSDXHUwMDFkUqWR2HF+nOzByuFQq4BcdTAwMGZcdTAwMDRqSaVcdTAwMWQsSWa5XHUwMDE5NE3jxotcYkst6Vx1MDAwZlx1MDAwNFx1MDAwMf2H/LQkXHUwMDE3lGDNTN9cdTAwMWH/IdVcdTAwMTi1XHUwMDAwXHUwMDFjdmVqpsLkZpg7IVqSzYOUnLK8TpeSrlEsipJcZofhUCVJMfZEXHUwMDAw0+ZcdTAwMDBccsylQ96VJLY8ooBcdTAwMWNCXHUwMDBm4kK9WtJiXHUwMDFkWcVcdTAwMDF4JJNcdTAwMWGUjFx1MDAwZt3gysGKo1x1MDAxZpXIhP2C2JNcdTAwMDUgXHUwMDFks/sqJXa1pP6+ykDOIbFlJtM0XHUwMDEx5ZiGQD+furJcdTAwMWFooHh4TdOjYH6NnMTOpkZxzFx1MDAwZSXaTFx1MDAwNnXALLZcdTAwMTaGq4jRQFx1MDAxNGZOXlx0h+GkY6BcdTAwMDJew+B9Kan5RNDhxFnxz1x0O1x1MDAxY37lqfFSOqi+P732VbVz/zFcdTAwMTn/TI88XHUwMDE33fLz00U2+3D33HmrZM9a+bddXHUwMDFhXZYy6XBBqH9cdTAwMTCL0MaxRlx1MDAwMlxytUG9d7V2/Fx1MDAwMliWUjR8VXiKuihcdTAwMDRcdTAwMTeoXHUwMDE0hFhD4lx1MDAxNajE0nmAJyyRi2N4b7F06DW7Kzzlj3fu+1xc1Hezra3OcyFy9+1l61x1MDAxNFeU0GK1TvG3iyi6mSO0SKqYheVf6XFqKzlxgPHCkvlep9tPN1PHgnA2mlx1MDAwMpGh+ltKeS+EbK77g5afalx1MDAxNoz4e189P1x1MDAxYnlfXHUwMDAxl1x1MDAxNEmfrlLiKlx1MDAwM5eS83VcdTAwMWVcdTAwMDTzlIU/gu918Vx1MDAwNLPmLHf4unFY2Lyurbezr+3X61x1MDAxZJ94L29CMCCJXHUwMDAzxkex8La3XHKntMTBXHUwMDEy81JcdTAwMTI+aMPpXHUwMDAxlWU+8EpyXFxpRiVDwelcdTAwMTlcdTAwMDTeXHUwMDFhQL//Ll0gXHUwMDFiotxySlnaXHUwMDExXHUwMDFmoNaEYT9fpG2ZXHUwMDEw/HNcdMHHO4XMyT4rbN+96lx1MDAwYvpSOtvsMeFFoqA+TlxuR1x1MDAxM5RcdTAwMTQsMLmAXGJKXHUwMDAympVcdTAwMTjjoJ9qXHUwMDE4duqTN/a79NlcdTAwMGZwnVZUTFxudlx1MDAxNWKQtnS31YxcdTAwMDBJuGGrXHUwMDAxPVx1MDAxMkWA+CuTaVx1MDAwMu1yM39AXHUwMDBl+3uZq87reTWTub59PVmdh+ZFXHUwMDAw6Li7tSiCx/85R1x1MDAxMDxcdTAwMWFcdTAwMTONhFwiXGZzj32CToVcdTAwMDO3Q8V4fdhlxOnMXHUwMDEw0I5cdTAwMTFwarnkmijfXGLzYMkjXHUwMDE5odjPIGXNoylROs3WRfNcdTAwMTh6OmWBXHLvX7T4Qah78vRMbFdebnq1fX6ao4+HXHUwMDA3bs0zjX9cdTAwMDB15Lgrj1x1MDAwNVx1MDAwZrz8Q8JZLa32b1wiiaFKWr3HMVxyQpV8IOd38Y/v31x1MDAxZu7MTD8w1sxyLv02YXiwJOJcdTAwMWMwxIhvXGZU2uq1VO74+altz8ubT5nL7Crrm29oyjiVf1jNqUlW7X9cdTAwMWVcdTAwMTHB/zlH4Vx1MDAxZlxcOYRhPzzGqU+gXHUwMDEycYxcdTAwMTaEcX/+sYxSSopcdTAwMDHd6PxcdTAwMDMkn+FYscVvtktcdTAwMWU424nQklx1MDAwYldpkpRcYlxiTpyZuvaEXHUwMDEykLmIV5qywMbhXHUwMDFmXHUwMDBiXHUwMDE4uXRQLF7Uri5fZLHXPKvZ/T47PKpEp1x1MDAxZoQ5lFxmUlx1MDAwMfzoh9ZcdTAwMGVGftD31NqlUzotwHmZmXRcdTAwMDBvxFx1MDAwNDpfXHUwMDE4XG7p4CotXHUwMDA2fiRqlZGMc9yfvGw8POxztp/J9bcq16dHR6zzi1x1MDAxYkxP5TJcdTAwMTLXbrUwXFzG//1F4TLEOloqY0Ce+DiPXHUwMDE5JmlQo2Qgm1n6jpNhSy9cdTAwMGWZgdfPXHUwMDE5891NXHTcTKFcZqtcdTAwMWZcdTAwMDKhXHUwMDE5sZ2UPMhcdTAwMTab2nxcdTAwMTmZmVx1MDAwYl/ylOU61mbK4pGZdk5cXGZo5pHzyt5++eDhfPPuYCOKV9lS6XDFLGGa+3iVjXVcdTAwMTCWKcV+XCJSeCtcdTAwMDAuvcoryVx1MDAxMac/s1dZ4Fx1MDAwNi3o3ZgtXHUwMDAzpNSwMqSfxMtcdTAwMTmXiVwiWZZO5Vx1MDAwNXEqXHUwMDFmXGJ++7y6etnUZ3Rn5+7kTdremlx1MDAxN4iCOtNTorBtvKZcdTAwMDFeZen4+JhGu7rSke5cZlS/jJBfJbC+n1x1MDAwNeVcdJlZYjEqXHUwMDAxXHUwMDBl4tacXHUwMDA2wsso1eRcdTAwMWL9yneb590yPzlcdTAwMTJ35KbV3njtt3bWI1x1MDAwN/5+ZUN2bFx1MDAwNKNIXCIknUd24v+cI2ghIFx1MDAxZdwhwlqmfKtcdTAwMTlpTLExUo7zjziO5WX1Sn9cdTAwMTSg/ijgXHUwMDFiSGuwa4WIXHUwMDE3RVwiJcd6u8nC/L9k4nxKoXn0JE9ZVSfFz+L7jp87jdZ+O3+YPcpuXHUwMDFlqGLn+f62eFx1MDAxY51mXHUwMDEwJVx1MDAxZMNVUNlEaVx1MDAxZGOpXG5cZl6zXHUwMDBlXHUwMDAzLHqPXFxbXHUwMDA2ryVcdTAwMDNcdTAwMTg2M83g2nCiaLx655JzwzlcdTAwMTUpsox3i3w765ydbef0492+vXi9PL9cdTAwMTK10lo0Olx1MDAxMEAzpkbFXHUwMDE12vnW3t5rVcvbu7fS6jm963fjsVxmpXmyIOB5xFx1MDAwNP/nXHUwMDExhWVcdTAwMTCtXHUwMDFjrTFtmPvVuWDcMVx1MDAxZq7hhO7jtFlGp5Wrt5u5XHUwMDE25l+mXHRcdTAwMDTf7tDJXHUwMDEzXHUwMDFlY9dcdTAwMTU0XHUwMDAyyD9cdTAwMTGvZJnE71x1MDAxMJ52M1xcY7RSiiWZPP5MYy5cXMZT1tVworGATuKb24OnTKN7q0Sv3yrtXHUwMDFkvFx1MDAxNW5zOlx1MDAwNs/g1Fx1MDAwMSpcdTAwMTBQgFx1MDAxZnOXsVO74EFeYqtcdTAwMWOLtfk/qcbIyEdMQ32BN2dcdTAwMTFcYkZcbkWZObFGXHUwMDAyzYi1kVx1MDAwMauqZYIla8iajFx03J6UT/N3T/r+Vl1cdTAwMTbEgz1pb99eRN3HkMfth/XM88PO2/H1iTjMbr5cdTAwMWVcdTAwMWbepeArnoFcdTAwMTBNZy5KaM5cdTAwMTfGV+z//lwiMVx1MDAxN2GRmzBGP+pcdTAwMWaMM1x1MDAxNzJccmBA6VxiXHLAgk3iJLe+cbBk6S72XHUwMDA1XHUwMDE4XHUwMDE5g7lg815cdTAwMDE0M1byXHUwMDFmkElcdTAwMDVcdTAwMTCU+lx1MDAxZUnyyfPJXFzoSrvUWWmU4YpcdTAwMTPO4va/681Sa+WsVG83WpsvcDf/rlx1MDAwZux3XHUwMDBlyM2UxXyS3LhcdTAwMDY57kBur0xcZnFlfISLQn5e5Pb1S8nWT1x1MDAwZliZXHUwMDFk37+sXHUwMDBive5cdTAwMDNLPr2LmULJpDjG3IIoXHUwMDFhXHUwMDE58GeqMuFGc7V0Kn9YXqq4pKNcdTAwMTKfQK8y+mG0XHUwMDEy1jcyP9iDQ8yg96ROXHUwMDE0JVx1MDAxN1JVRFx1MDAxOViZXHUwMDEy4dXSrbwgbuXwKjvTXG7WW6zkLbGBXHUwMDE1MSPzfS8ziFx1MDAxYsFEcWtcdTAwMTihLn32XHRF3GLqolx1MDAwMNO2RGhK/VxcyphcdM2oJMpIYGFSj1x1MDAwMG1uXHUwMDA12Vx1MDAwZmz0rM1cZktcdTAwMTZEXHUwMDE1U0bEXCJRTFwii5Kpg5KxQpGf76i+XHUwMDFhaKB4eE0zTYT63lwig1ZwhFikXGaWuVx1MDAxYVW9n4Z3oVx1MDAwNNdYg1x1MDAxMmSQsZ5xxiwyXHUwMDE4Xrom4Vx1MDAxOJhEMWYwZVG7XHUwMDFi4fqe9S4mi1pjMFx1MDAwNPfDk/dDm9ZL4UjBXHUwMDE5zlJ4rq5cZqPPRiUgmjlX2lgrXFxOkE/gV8rBTWPGXHUwMDE5sbh97LOrb4VDmVTESiCxxrh+xdzi/k/o5PWZgV9cdTAwMWJcdC/R8Fh0XHUwMDE0qFx1MDAxYShcdTAwMTDJXHUwMDEyNUf7kpJcdTAwMGapXHUwMDAyf6B94uGxzO+A/eO3tedu9e28uPG8VVNPmd76dW97VsikymC7d1xccLHvxHjbT1x1MDAwMtyNYsKnkJJcdTAwMWHDrJlcdTAwMTX2w1P2ky5dhGKoXHUwMDAxZZpcbsLtxFx1MDAxMCbOqrD6uP9cdTAwMTjcXHUwMDBmTzRcbsd9hj0vtVJWS2wxPYH7xpFgT+azrLhcdTAwMDf3seu95Fx1MDAxNub6wNtIfTticuFoUMfwXHUwMDAxM9hY/Vx1MDAwNU2qfoDxz96jXG57S1x1MDAxOEG1by+T4Fx1MDAxMlx1MDAwMVx1MDAwNjuQXHUwMDAxl0q7tcEgaTiRyzdV6Fx1MDAwZjbRwVmPcf5e8Mdmh1x1MDAxOL8plFx1MDAwNYU+1oyLXGbixFx1MDAwN1HekmNt7rBuwpGwPzxdOinlZ1x1MDAxYaRcIlx1MDAwM2WCXHUwMDFhdMRg/M4y+c/piDz8ylxmdcXDs1xcp4lcdTAwMDOQkVx1MDAxYZttXHUwMDBieP6TvSe4XHUwMDAztFx0bM4whpmWI/RcdTAwMTmFXHUwMDE2O1pbXHUwMDA211FEW+5cdTAwMWJaTFx1MDAxNXe4UnSQhym1dIWlLleJ0SqxmcIqoTXlXHUwMDAxpa2C1lxi0H5cYp2cpOxde09j/vlOhqvBJoqH1zh/7yqBZV+wgyRcdTAwMTBoY8xE61x1MDAwNkO00Fx1MDAxMt42M5LCMesqXHUwMDExnoeadFxiglhsumix7VwiU/yvkLPShKmcxVolQsC/Ut2t1rM5Ub47uur1u52H/Vx1MDAxZOFTK8gn41VcdFx1MDAwN0RcdTAwMTjR2FpI6ImUXHUwMDEySZVDqTVcdTAwMDaVXHUwMDE4iH2vR0BKx1xuXHUwMDEw3Vx1MDAxYYO6uLtt8lx1MDAxMPl/Q8DEXHUwMDBmOCa3UkB6XHUwMDAyoIWbXHUwMDAxsaBcdTAwMWW0XHUwMDAwbqjalH1cdTAwMDBGccvmoIhyoEVOfHt2XFxf+iTfrSdCXHUwMDAyYFx1MDAxOP2kYJGOpVx1MDAxY9Ccg1x1MDAxOTE96ZRcdTAwMDSBp+F9XG5cdTAwMDNcdTAwMWNFevcomHRcZqxcdTAwMDbKaFx1MDAxNHrKXG6fXGZcdTAwMTSrXHUwMDFjuDBcdTAwMDNDxWYq8lx1MDAxN/gkf2JvendmSEL/glx1MDAwMe5cdTAwMTmvyVxykULDq09cdTAwMTYkOi397edcdTAwMTFpNdhC8Zi0ze+gnqX79mWuml3vVvazR5v3ypJcdTAwMDa7mnlnVzAglkAujeW4hTuxQ6FA1CtcdTAwMTCSXG77c1pcdTAwMWLm0YvGPUPTXHUwMDAwXHUwMDEzjoFg4zbJJOghXHUwMDFjXHUwMDA1nSCfk6eXbsmpsc3h0C+xcJyyXHUwMDA2XHUwMDBiOClcIrx+SSCdzGI7PVx1MDAwNYu6t+2lpJirZFx00Fx1MDAxOWYs81x1MDAwYkhhXHUwMDE41atgXHUwMDEwzFx1MDAwMCRQs9x48MP+vZmxXHUwMDFm3lx1MDAxMWVaxNx4wK6XXHUwMDE4ypCyY9IoyYWcg7aXQVx1MDAwNoqH1zR/L/ZcdTAwMWLAe6OEwNRcdTAwMWXubXppNExmXHRaXHUwMDA1tya8e/AxoT88MSvhXHUwMDEwMD+JKmWM1Vx1MDAxYdewv8LOXHUwMDEyb4PSfyDyh2fLTCX9jDIhqKbEjqeCIeWHnzKcMtKS0Vx1MDAwZdBox1x1MDAxOWvfXGJlLbxcdTAwMTKwKWp9XHUwMDAyUqhcdTAwMDIlXGLnleGMXHUwMDEz6+qIvMT9XHUwMDEx7u/PjPtYqIZJIK8xSb/ihGDYYNrAj/lcdTAwMWM/75ZcXFxyNlE8PMb5i5FfMobSnGilXHUwMDE5o1x1MDAxZehcdTAwMTdaU2U1OpbM7H7J8LSVpKxcdTAwMWZ4qNVcdTAwMDQraVx1MDAxMbzXiU2jyfN2yfvxO5eMnveP997WzlQlw2/OXHS9Wrvyor83UVx1MDAwZuR+SLcszqWjQGIp9d5cdTAwMTPLJyqFXHUwMDE5x1pAXHUwMDFjwTGWkfrVjmYpbzsvUJGBI3/I9+2Tg4xcdTAwMWWIacySilpwhc3P0yb1ckBcYpJg+2eu3tl9rlUsXHUwMDE1V/L9lftcXPve9SpHiXiu2k1RXHUwMDEy8WqlssskPGl4nYarXeJ4XHUwMDBl3tiNTybcXHUwMDA13umiJNRdXHUwMDFlXHUwMDFk35nnOj8rX29cdTAwMTa7XHUwMDBm1e7FTe80XG58XHUwMDAwPDuCa0swTsRcdTAwMDJPnIBcdTAwMGbhSE1cdTAwMTR8alx0XHUwMDFmXHUwMDFmppUqfFx1MDAxY0eHXHUwMDBmXHUwMDEwTcRcIr2PXHUwMDA1XHUwMDFmVmjKsWpi2uihuUrWj3uJXHUwMDFlc4dcdTAwMWV8+2I7f17JXZ6eVlj+ZstcdTAwMWWY7Z0o6DHYc5RSgmFcdTAwMWE/+Fx1MDAxMMpRhlx1MDAxMMmX8PFpW6nCx0l0+Fx1MDAxMNJIg07uOOghqKBMsbRlpdRaykRcdTAwMTluS/D4SfD414eS/pNrNs868E6H2vLPS7XUW/fPfIdcdTAwMDO/P4BcdTAwMWWcZ6XBnsh//vWf/1x1MDAwZqX7rXUifQ==CentralSensor 1Sensor 2Sensor NSensorConnection Sensor NSensorConnection Sensor 2SensorConnection Sensor 1runRecv()Event DedupingQueueNetFlowUpdateDedupingQueue1 DedupingQueue perMsgFromSensor typePod workerQueueVMIndexReportworkerQueue1 workerQueue perSensorEvent typePodDedupingQueue 17PodDedupingQueue 2PodDedupingQueue 1VMIndexReportDedupingQueue 17VMIndexReportDedupingQueue 2VMIndexReportDedupingQueue 1DedupingQueue 17DedupingQueue 21 set of 17DedupingQueuesper SensorEventtypeSharded by hashSharded by hashSharded by hash \ No newline at end of file From b08fb0c449684be690eb616999a0f9c324cfd691 Mon Sep 17 00:00:00 2001 From: Guzman Date: Thu, 12 Mar 2026 22:50:23 +0100 Subject: [PATCH 2/4] Add SVG file to allowed-large-files --- tools/allowed-large-files | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/allowed-large-files b/tools/allowed-large-files index ca1a4f031c4d0..a53972b63d524 100644 --- a/tools/allowed-large-files +++ b/tools/allowed-large-files @@ -16,6 +16,7 @@ central/networkpolicies/graph/evaluator_test.go central/processlisteningonport/datastore/datastore_impl_test.go central/pruning/pruning_test.go central/reports/service/v2/service_impl_test.go +central/sensor/service/connection/sensor-central-data-flow.excalidraw.svg central/splunk/violations_test.go central/testutils/testdata/imgdata.json.gz central/vulnmgmt/vulnerabilityrequest/datastore/datastore_impl_postgres_test.go From 6929b7d08a547f8a1b403fc6bf2d061fa42f43df Mon Sep 17 00:00:00 2001 From: Guzman Date: Thu, 12 Mar 2026 22:55:52 +0100 Subject: [PATCH 3/4] docs: note that all DedupingQueues are unbounded (OOM risk) All DedupingQueue instances at every layer have no capacity cap. If a producer outpaces its consumer goroutine, memory grows without limit. Added a Gotcha in Layer 2 (where DedupingQueues are first introduced) covering all layers at once. Partially generated by AI (Claude Opus 4.6). --- central/sensor/service/connection/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/central/sensor/service/connection/README.md b/central/sensor/service/connection/README.md index 926618167ea8b..8ffb4b7010195 100644 --- a/central/sensor/service/connection/README.md +++ b/central/sensor/service/connection/README.md @@ -34,6 +34,11 @@ The per-type fan-out is primarily about **category isolation**: a surge of one m `NetworkFlowUpdate`). These queues use `DedupingQueue`, so messages carrying a `DedupeKey` are collapsed (see [Deduplication](#deduplication) for which messages set this key and when). +**Gotcha (unbounded queues):** all `DedupingQueue` instances at every layer are unbounded. +If a producer outpaces the consumer goroutine, the queue grows without limit and will +eventually cause an OOM. The replace-in-place dedup provides partial relief for messages +that carry a `DedupeKey`, but messages with an empty key are always appended. + **Gotcha:** only `Event` messages continue into further queuing (Layers 3–4). All other types (NetworkFlow, Compliance, Telemetry, …) are handled inline in `handleMessage()` without further queuing. From 93222312105b51d06ce5007906d34d7539d73203 Mon Sep 17 00:00:00 2001 From: Guzman Date: Thu, 12 Mar 2026 22:57:47 +0100 Subject: [PATCH 4/4] docs: move unbounded-queue OOM gotcha to Layer 4 Items move through Layers 2 and 3 by dispatching only (no heavy work). They actually accumulate in Layer 4 while waiting for pipeline fragments to process them (DB writes etc.). That is the realistic backlog point. Partially generated by AI (Claude Opus 4.6). --- central/sensor/service/connection/README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/central/sensor/service/connection/README.md b/central/sensor/service/connection/README.md index 8ffb4b7010195..248c74f642ceb 100644 --- a/central/sensor/service/connection/README.md +++ b/central/sensor/service/connection/README.md @@ -34,11 +34,6 @@ The per-type fan-out is primarily about **category isolation**: a surge of one m `NetworkFlowUpdate`). These queues use `DedupingQueue`, so messages carrying a `DedupeKey` are collapsed (see [Deduplication](#deduplication) for which messages set this key and when). -**Gotcha (unbounded queues):** all `DedupingQueue` instances at every layer are unbounded. -If a producer outpaces the consumer goroutine, the queue grows without limit and will -eventually cause an OOM. The replace-in-place dedup provides partial relief for messages -that carry a `DedupeKey`, but messages with an empty key are always appended. - **Gotcha:** only `Event` messages continue into further queuing (Layers 3–4). All other types (NetworkFlow, Compliance, Telemetry, …) are handled inline in `handleMessage()` without further queuing. @@ -74,6 +69,13 @@ N clusters × M resource types seen × 17 workers For 10 clusters each sending 15 resource types that is 2 550 goroutines just for workers, plus Layer-2 queues and their goroutines on top. +**Gotcha (unbounded queues):** the Layer-4 `DedupingQueue` slots have no capacity cap. +Items flow through Layers 2 and 3 quickly (dispatch only), but sit in Layer 4 until a +worker goroutine runs the pipeline fragment — which involves DB writes and can be slow. If +events arrive faster than the pipeline can process them, the queues grow without limit and +will eventually cause an OOM. Replace-in-place dedup provides partial relief for messages +that carry a `DedupeKey`, but messages with an empty key are always appended. + ### Layer 5 — Pipeline fragments [`central/sensor/service/pipeline/all/pipeline.go`](../pipeline/all/pipeline.go) `Run()`