Skip to main content
Version: 2.0.0 (Latest)

Aggregation & Correlation

This page covers stateful, time-bounded PDL: metrics computed over a timespan, optionally split by group_by and filtered with where. That includes classic aggregations (count, sum, avg, …) and correlation-style constructs such as temporal.

Per-event parsing and eval live under Expressions. Function-by-function aggregation examples with sample JSON outputs are in Aggregation functions.

Also useful: PDL Quick Reference · Reference

Common syntax pattern​

Many windowed statements follow this idea (exact keyword order may vary by function; timespan is always required for aggregates):

<function> [<function-specific-args>] timespan=<value><unit> [group_by <fields>] [where <query>]

Examples:

count timespan=5m group_by field1 where field3 > 100
distinct_count(myfield) timespan=30s
temporal(ordered=true) [ padasRule="internal_error" || padasRule="new_network_connection" ] timespan=1m group_by internal_ip, remote_ip

Shared parameters​

Windowed operators evaluate events that fall into the same logical window. group_by emits one result row per distinct key combination inside that window; where applies a query expression so only matching events contribute to the aggregate.

Typical keyword roles​

KeywordRequiredRoleExample
timespanYes (aggregates)Window length: integer + s, m, h, or dtimespan=5m, timespan=1h
group_byNoSplit metrics by one or more fieldsgroup_by field1, field2
whereNoRestrict which events are counted / summed / …where field1 > 100

The function name (e.g. count, temporal) always leads the statement; some functions accept extra arguments before timespan (see Reference).

Aggregation functions (summary)​

PDL provides counting and statistical reducers over windows. All of them honour timespan, and typically support group_by and where as above.

Counting​

  • count — Events in the window (optionally count(field) when the field must exist).
  • distinct_count(field) / dc(field) — Cardinality of distinct values.

Statistical​

  • avg, median, min, max, variance, stddev — Numeric summaries over the window.

Worked examples and illustrative padas output shapes: Aggregation functions.

Correlation functions​

Temporal​

Description​

Temporal correlation statement checks for all the events matching the expression array within the time frame defined. If the boolean value ordered is set to true, then all the events are expected to occur in the given order. The result may also contain count of events for each group specified by group_by separately.

Syntax and Functions​

... | temporal(<ordered-param>) [ <expression> || <expression> || ... ] <timespan-param> <group-by-clause> <where-clause>

Ordered parameter: order is assigned either true or false as value (e.g. ordered=true) to specify whether the events are expected to match expression array order. Expression array: The array consists of one or more expressions separated by double-pipe || character (e.g. [ field1="valu*" || field3 < 100 AND field4=false>])

Temporal Examples​

The following table provides examples of available functionality based on the following JSON value:

JSON InputExpressionExpected Output
  {"field1":"value1", "field2":99}
  {"field1":"value1", "field2":124}
  {"field1":"value2", "field2":123}
  {"field1":"value2", "field2":125}
  temporal [ field1?="value" || field2 < 100 ] timespan=2m
  {
    "padasRule": "myrule8",
    "padasTemporal": {
      "result": {
        "field1?=\"value\"": [
          {
            "field1": "value1",
            "field2": 99
          },
          {
            "field1": "value1",
            "field2": 124
          },
          {
            "field1": "value2",
            "field2": 123
          },
          {
            "field1": "value2",
            "field2": 125
          },
        ],
        "field2<100": [
          {
            "field1": "value1",
            "field2": 99
          }
        ]
      }
    }
  }
  {"field1":"sometext", "field2":90}
  {"field1":"value1", "field2":124}
  {"field1":"value2", "field2":95}
  {"field1":"value2", "field2":123}
  temporal(ordered=true) [ field1?="value" || field2 < 100 ] timespan=2m group_by field2
    {
      "padasRule": "myrule9",
      "padasTemporal": {
        "groupBy": {
          "field2": 95
        },
        "result": {
          "field2<100": [
            {
              "field1": "value2",
              "field2": 95
            }
          ],
          "field1?=\"value\"": [
            {
              "field1": "value2",
              "field2": 95
            }
          ]
        }
      }
    }