Grounded Intelligences

Tag: osint

Open-source intelligence techniques, threat monitoring, and certificate transparency log analysis.

  • OSINT at the Edge: Building a Threat Monitoring System with Cloudflare Workers

    Running meaningful threat intelligence monitoring at any scale — certificate transparency logs, domain registration alerts, threat feed normalization — costs either a commercial subscription in the thousands of dollars per month or custom infrastructure that requires active maintenance, unless you build it on Cloudflare Workers, where the entire stack runs at $0.

    I’ve covered the Digital Sentinel system overview in a previous post. This one goes deeper — into the specific data sources, the edge architecture that connects them, and the numbers that came out of running it in production against real client scope.

    Certificate Transparency: Every Certificate, Indexed in Public

    Every SSL/TLS certificate issued by a trusted Certificate Authority gets logged to public Certificate Transparency logs — crt.sh, Google Aviator, Cloudflare Nimbus, and others. This is a regulatory requirement, and it means that if someone registers a typosquatting domain and immediately obtains an HTTPS certificate for it, that certificate appears in the CT logs within minutes of issuance. Most organizations don’t have a system watching those logs for their own name patterns. They find out about impersonation domains when a customer calls.

    A Cron Trigger Worker polls the crt.sh JSON API on an hourly schedule. The worker queries for certificates issued to domains matching configurable pattern sets — brand name variants, common typosquats, executive name combinations — and filters the response against a D1 database of previously seen certificate IDs. New matches are written to D1 with full certificate metadata: issuing CA, subject alternative names, issuance timestamp, and the registrant domain. The total runtime per execution is under 800 milliseconds for a scope of approximately 40 pattern variants. Cloudflare’s free-tier Cron Triggers handle this without any cost.

    Threat Feeds: Normalization Is the Actual Work

    AlienVault OTX and AbuseIPDB both offer free API tiers with meaningful data volumes. The operational problem isn’t access — it’s normalization. OTX returns indicators in a pulse-based format with nested JSON. AbuseIPDB returns scored IP addresses with abuse category codes. Neither format matches the other, and neither maps cleanly to a schema you’d want to query for alerting logic.

    A second Worker fetches and normalizes both feeds every four hours. It extracts indicators — IPs, domains, hashes, URLs — maps them to a unified schema with source, confidence score, indicator type, and first-seen timestamp, then deduplicates against D1 before writing. Deduplication runs against a composite key of indicator value plus source, because the same IP might appear in both feeds with different confidence scores — and that delta is itself a signal worth preserving.

    Rate-limit state is tracked in KV, not D1. KV’s per-key TTL makes it the right store for ephemeral rate-window tracking — writing a counter to D1 for something that expires in 15 minutes is the wrong tool. This distinction between KV for transient state and D1 for durable records runs through every part of the architecture.

    RDAP Domain Monitoring: The WHOIS Replacement That Actually Has an API

    WHOIS is largely broken for programmatic use — rate limiting, inconsistent formatting, and increasingly redacted registrant data make it unreliable. RDAP, the Registration Data Access Protocol, is the IANA-standardized replacement. It returns structured JSON, it’s queried via HTTPS, and it works inside a Workers environment without requiring a separate proxy layer.

    The domain monitoring Worker queries RDAP endpoints for newly registered domains matching typosquat patterns — character substitutions, homoglyph variants, hyphenated versions, TLD variations — and stores registration records in D1. In the first month of running this against a client’s brand scope, the system identified three domains that had been registered with clear impersonation intent: two used homoglyph substitutions in the brand name, one combined the brand name with a plausible-sounding financial suffix. None had been caught by any other monitoring the client had in place.

    The Alerting Pipeline: From Detection to Discord in Under 90 Seconds

    All three monitors — CT logs, threat feeds, and RDAP — write findings to a shared D1 table with a processed flag. A fourth Worker, AlertDispatch, runs on a two-minute Cron Trigger, queries for unprocessed records, enriches each finding with contextual data from KV, formats a Discord embed with severity color coding and direct links to supporting sources, and POSTs to a configured webhook. End-to-end alert latency from detection event to Discord notification runs under 90 seconds in normal operation.

    The system currently processes approximately 47,000 CT log entries daily across active client scope. The false positive rate — certificates that match patterns but aren’t actual threats — runs at about 4%, which is low enough that analysts review every alert rather than implementing suppression rules. Keeping the false positive rate manageable required iterative refinement of the pattern matching logic over the first six weeks; the initial version was closer to 18%.

    What Transfers

    The architectural pattern here — scheduled edge Workers feeding a central D1 store, KV for rate control and transient state, a dispatch Worker for enrichment and notification — applies to any event-driven monitoring use case that doesn’t require persistent server processes. Price monitoring, compliance change detection, API health surveillance, social mention tracking: the structure is the same. The zero-cost operation isn’t incidental to the design. It’s the reason this architecture is worth understanding — it puts monitoring capability that used to require dedicated infrastructure into a pattern any developer can deploy and maintain solo. That’s the broader lesson from running 60+ Cloudflare Workers in production: the constraint that keeps most teams from building this kind of tooling isn’t technical complexity. It’s the assumption that it requires budget they don’t have.

    → Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

  • Building an OSINT Monitoring System on the Edge with Cloudflare Workers

    Building an OSINT Monitoring System on the Edge with Cloudflare Workers

    One of the less visible but most practical systems in my Cloudflare Workers fleet is Digital Sentinel — a set of OSINT (Open Source Intelligence) monitoring pipelines that scan certificate transparency logs, threat intelligence feeds, and domain registration activity, then fire automated chat alerts when something interesting shows up.

    What It Monitors

    The system pulls from multiple sources: certificate-transparency logs (catching new SSL certificates issued for domains I care about), threat-intelligence feeds for SEO and competitive-intelligence signals, URL-analysis services for malicious-URL detection, and additional threat-intelligence feeds for enrichment. Each source has its own dedicated Worker that runs on a cron schedule, queries the API, diffs against the last known state in D1, and posts alerts for anything new.

    The Architecture Pattern

    Each monitoring pipeline follows the same pattern: a cron-triggered Worker fetches data from an external API, compares it against a D1 table of previously seen items (deduplication by hash), persists new items, and sends a formatted automated chat alert for anything that crosses a threshold. The pattern is simple enough to replicate for any data source in about 30 minutes.

    What makes this interesting as an engineering exercise is the constraint set. Cloudflare Workers have a 30-second execution limit on cron triggers, so the pipelines have to be efficient — no long-running batch jobs, no streaming connections. D1’s SQLite engine handles the state management cleanly, and the Workers runtime’s built-in fetch makes external API calls trivial.

    Why This Matters

    For personal security monitoring, this system provides continuous visibility into certificate issuance, domain impersonation attempts, and threat-intel signals that would otherwise require a paid SIEM or manual checking. For my portfolio, it demonstrates the ability to build event-driven monitoring systems on modern edge infrastructure — the same pattern that scales to enterprise OSINT, brand protection, and security operations.

    The full system runs alongside 60+ other Workers in my Cloudflare fleet, all orchestrated by a meta-orchestrator that monitors the health of the entire platform and self-heals when individual Workers encounter issues.

    See more on my Projects page, or get in touch to discuss edge-native monitoring architectures.