From Noise to Signal – Part 2: Log Ingestion, UDM Normalization, and the Art of Structuring Data

Part 2 of 4 in our series on Google SecOps in Enterprise Environments

In Part 1 of this series, we explored the harsh realities of modern security operations. We discussed why traditional SIEM setups inevitably struggle under the sheer weight of modern data volumes, eventually leading to alert fatigue, missed signals, and burned-out analysts. We ended that discussion by identifying four foundational pillars that actually make a difference when designing a modern security architecture.

The very first of those pillars was a unified, structured data model.

In this second part, we want to show you exactly what that looks like in practice. Because before we can talk about advanced threat hunting, machine learning detections, or automated playbooks, we have to start at the absolute beginning: getting the logs in, and making them make sense.

The Onboarding Reality: A Symphony of Chaos

Before any detection rule can trigger, before any dashboard can populate, and before any SOAR playbook can isolate a compromised host, the underlying data has to arrive somewhere in a usable form.

On paper, this sounds incredibly simple: point your log sources at an endpoint, hit send, and start analyzing. In enterprise reality, it is anything but.

Modern enterprise environments are heavily fragmented. During a typical onboarding process, it is entirely normal to connect dozens of disparate log sources. You have ephemeral Kubernetes clusters spinning containers up and down in Google Cloud or AWS. You have traditional perimeter firewalls, zero-trust network access (ZTNA) gateways, identity providers like Azure AD or Okta, cloud control plane APIs, and legacy on-premises servers running business-critical, bespoke ERP systems.

Each of these systems speaks a completely different language. They have their own proprietary log formats (JSON, CEF, Syslog, CSV, XML, or just plain unstructured text). They have their own timestamp logic—some use UTC, others use local time zones without offsets, and some use Unix epoch time down to the microsecond. Most importantly, they all have completely different field names for the exact same concepts.

This is the chaos that security engineers face daily. If you simply dump all this raw data into a data lake without normalizing it, you haven’t built a security platform; you have built a digital landfill.

The „Easy“ Part: Prebuilt Parsers and the Power of UDM

Getting the systems connected via forwarders, APIs, or webhooks is generally the easy part. The real magic happens the millisecond the log hits the platform.

Google SecOps (formerly Chronicle) ships with a massive, continuously updated catalog of prebuilt parsers for common enterprise vendors. Whether you are ingesting logs from Cisco routers, Check Point firewalls, Palo Alto Networks Cortex, AWS CloudTrail, or Microsoft 365, the platform already knows how to read them.

For these standard sources, the process is seamless: you connect the source, the parser catches the raw log, and the data immediately lands in the Unified Data Model (UDM).

UDM is Google SecOps’ standardized schema. It acts as a universal translator for your entire IT estate. Consider a simple scenario: a user attempts to log into a system, and the traffic passes through a firewall, a proxy, and an authentication server.

• The firewall log might record the originating IP address as src_ip.
• The proxy server might record it as client-ip.
• The identity provider might log it as source_address.

In a legacy SIEM, a security analyst looking for lateral movement from the IP 192.168.1.50 would have to write a complex, nested query accounting for all three variations: (src_ip = „192.168.1.50“ OR client-ip = „192.168.1.50“ OR source_address = „192.168.1.50“). If a new log source is added tomorrow with the field originating_ip, the query instantly becomes outdated and blind to the new system.

With UDM, the prebuilt parsers take care of this translation dynamically upon ingestion. Regardless of what the vendor called it, the IP address is mapped to a single, standardized field: principal.ip.

One name. One field. Regardless of where the log came from. The platform takes care of standardizing hostnames, normalizing user IDs, and converting all timestamps to an absolute, comparable format.

The Hard Part: Custom Applications and Uncharted Territory

However, enterprise environments are rarely built entirely out of off-the-shelf components. Every large organization runs custom applications, internal microservices, and proprietary business logic that simply do not match any known vendor format.

For these sources, the prebuilt parser catalog ends, and the real engineering work begins.

The clearest example in our recent deployment involved custom application logs generated by internal microservices running inside a massive Kubernetes cluster. When these logs arrived in Google SecOps, they came as raw JSON payloads. But there was a catch: while the Kubernetes wrapper was valid JSON, the actual security-relevant information generated by the application was buried inside a generic message field as a single, massive string of unstructured plain text.

To the SecOps platform, this string was a black box. It was unsearchable via standard schema fields, uncorrelatable with network traffic, and effectively invisible to out-of-the-box detection rules.

To solve this, we had to write a custom parser. In Google SecOps, parsers allow you to pick these raw logs apart using logic, grok patterns, and regular expressions, lifting the relevant pieces out of the generic message text and placing them into proper UDM fields.

We mapped the custom application’s internal transaction ID to network.session_id. We extracted the user’s input and mapped it to target.url. We took the application’s internal status codes (e.g., ‚Validation_Failed‘, ‚Access_Denied‘) and mapped them to security_result.action.

Before (Raw JSON Log from K8s):

{

  „timestamp“: „2024-10-14T09:15:00Z“,

  „container_name“: „internal-finance-api“,

  „message“: „WARN: Access_Denied for transaction_id=99482A from origin=10.0.5.12 path=/api/v1/transfer payload_length=4096 error=Validation_Failed“

}

After (Structured UDM Entity via Custom Parser):

{

  „metadata.event_timestamp“: „2024-10-14T09:15:00Z“,

  „target.application“: „internal-finance-api“,

  „network.session_id“: „99482A“,

  „principal.ip“: „10.0.5.12“,

  „target.url“: „/api/v1/transfer“,

  „security_result.action“: „BLOCK“,

  „security_result.description“: „Validation_Failed“

}

This is highly specialized, often tedious work. It requires a deep understanding of both the application generating the logs and the SecOps platform’s data structure. But the return on investment for doing this work is astronomical.

The Case Study: Finding the Needle in the Kubernetes Haystack

Here is exactly why that parsing effort mattered.

Before the custom parser was implemented, the application logs were just noise. Millions of lines of raw text generated every day. But once those custom fields were mapped into the UDM schema, a distinct pattern emerged that nobody—not the developers, and not the security team—had been able to see before.

A single internal principal IP was generating a steady, slow-and-low stream of rejected requests against an internal API endpoint. Each request was slightly different. One request would append a strange character string to the URL path; the next would include a malformed header; the third would attempt to insert SQL syntax into a user-agent string. Every single one of these requests failed validation, triggering an application error.

On their own, looking at the raw logs in real-time, these individual lines looked like harmless application noise or perhaps a slightly buggy internal script. But together, correlated in structured form over a 72-hour window, they painted a crystal-clear picture: they were a textbook case of someone (or an automated script) actively probing an endpoint with manipulated input, fuzzing the application, and looking for a vulnerability to exploit.

Without the custom parser mapping the data to UDM, that signal would have stayed buried in an unstructured text field forever. No traditional SIEM rule could have caught it, because there was no specific field to write a rule against.

The Regex Trap: Schema-on-Read vs. Schema-on-Write

At this point, veterans of legacy SIEMs might argue: „Why not just run a regular expression (regex) search across the raw logs to find the pattern?“

This is the fundamental difference between Schema-on-Read (parsing data when you search for it) and Schema-on-Write (parsing data the moment it is ingested).

If you try to find our API probing attack using a legacy SIEM that relies on unstructured data, you have to write a regex query that scans the full, raw text of every single log line looking for specific keywords or patterns. If you have petabytes of data, running a wildcard or regex search across raw text is incredibly slow. It consumes massive amounts of compute resources, often causing queries to time out or take hours to complete. Furthermore, it is incredibly brittle. If the development team updates the application tomorrow and slightly shifts the log format, adding an extra space or changing a delimiter, your regex breaks silently. Your detection rule stops working, and you won’t even know you are blind.

By contrast, Google SecOps utilizes a Schema-on-Write philosophy through UDM. Because the custom parser lifts the values into proper UDM columns during ingestion, the data is already structured and indexed by the time it hits the database.

Instead of writing a complex regex, you can write a clean, logical query: principal.ip = „10.0.5.12“ AND security_result.action = „BLOCK“ AND target.application = „Internal_Finance_API“.

Because you are searching indexed, structured fields, the platform can return results over petabytes of data in less than a second. You can combine these fields using real logic (AND, OR, NOT), and you can reuse these exact same variables across hundreds of rules and investigations.

This is the part of the engineering work that doesn’t show up in high-level architecture diagrams or vendor marketing brochures. But it is precisely this stage that decides whether your detection layer later operates on real, actionable data, or whether you are just endlessly fighting your own tooling.

The Result: One Unified Search Layer

Once everything is connected, parsed, and mapped into the Unified Data Model, the daily reality for security analysts fundamentally changes.

In Part 1, we described the agonizing process of a traditional investigation: A suspicious IP address meant an analyst had to open one browser tab for the Endpoint Detection and Response (EDR) tool to check the host, a second tab for the vulnerability scanner, a third for the proxy logs to see web traffic, and a fourth to check the user’s Active Directory behavior. They then had to manually copy-paste timestamps and IP addresses into Notepad to stitch the narrative together.

After thorough onboarding and UDM normalization, that same investigation becomes one single query.

Because the host log, the firewall log, the EDR alert, and the custom application log all describe that IP address using the exact same principal.ip field, all the events are automatically correlated. When an analyst clicks on an IP address in Google SecOps, the platform instantly visualizes a consolidated timeline. It shows the user logging in via Okta, moving through the Palo Alto firewall, executing a command intercepted by CrowdStrike, and triggering a validation error in the custom Kubernetes app—all in one chronological view.

The pieces no longer need stitching. They are already speaking the same language.

Looking Ahead: Building on the Foundation

Normalization is the unsung hero of security operations. It is largely invisible work; nobody builds a shiny dashboard to celebrate a perfectly normalized data model. Yet, every single detection rule, every machine learning anomaly trigger, and every automated SOAR response in the rest of this series assumes this foundation is rock solid. Get it wrong, and everything built on top inherits the mistake. Get it right, and you transform a mountain of noisy data into a highly tuned intelligence engine.

Now that the data is finally structured, indexed, and speaking one universal language, we can move from defense to offense.

In Part 3 of our series, we will build exactly on this UDM foundation and dive deep into the world of Detection Engineering. The question shifts from „How do we read the data?“ to „How do we turn this data into high-fidelity alerts that catch real threats?“ That is where Google’s YARA-L rule language comes into play, where we can write lightning-fast rules across petabytes of historical data, and where the real work of separating the signal from the noise truly begins.