Instant Logs

Instant Logs allows Cloudflare customers to access a live stream of the traffic for their domain from the Cloudflare dashboard or from a command-line interface (CLI). Seeing data in real time allows you to investigate an attack, troubleshoot, debug or test out changes made to your network. Instant Logs is lightweight, simple to use and does not require any additional setup.

Filters and columns from the Logs view carry over into the live stream. Filters that cannot be streamed display a message instead of being silently dropped.

Create a session by sending a POST request to the Instant Logs job endpoint with the following parameters:

• Fields - List any field available in the HTTP Requests dataset.

• Sample - The sample parameter is the sample rate of the records set by the client: "sample": 1 is 100% of records "sample": 10 is 10% and so on.

• Filters - Use filters to drill down into specific events. Filters consist of three parts: key, operator and value.

All supported operators can be found in the Filters page.

Below we have three examples of filters:

# Filter when client IP country is not Canada:
"filter": "{\"where\":{\"and\":[{\"key\":\"ClientCountry\",\"operator\":\"neq\",\"value\":\"ca\"}]}}"

# Filter when the status code returned from Cloudflare is either 200 or 201:
"filter": "{\"where\":{\"and\":[{\"key\":\"EdgeResponseStatus\",\"operator\":\"in\",\"value\":[200,201]}]}}"

# Filter when the request path contains "/static" and the request hostname is "example.com":
"filter": "{\"where\":{\"and\":[{\"key\":\"ClientRequestPath\",\"operator\":\"contains\",\"value\":\"/static\"}, {\"where\":{\"and\":[{\"key\":\"ClientRequestHost\",\"operator\":\"eq\",\"value\":\"example.com\"}]}}"

Example request using cURL:

curl "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/edge/jobs" \
	--request POST \
	--header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
	--json '{
		"fields": "ClientIP,ClientRequestHost,ClientRequestMethod,ClientRequestURI,EdgeEndTimestamp,EdgeResponseBytes,EdgeResponseStatus,EdgeStartTimestamp,RayID",
		"sample": 100,
		"filter": "",
		"kind": "instant-logs"
	}'

Response:

The response will include a new field called destination_conf. The value of this field is your unique WebSocket address that will receive messages from Cloudflare's global network.

{
  "errors": [],
  "messages": [],
  "result": {
    "id": <JOB_ID>,
    "fields": "ClientIP,ClientRequestHost,ClientRequestMethod,ClientRequestURI,EdgeEndTimestamp,EdgeResponseBytes,EdgeResponseStatus,EdgeStartTimestamp,RayID",
    "sample": 100,
    "filter": "",
    "destination_conf": "wss://logs.cloudflare.com/instant-logs/ws/sessions/<SESSION_ID>",
    "kind": "instant-logs"
  },
  "success": true
}

Using a CLI utility like Websocat ↗︎, you can connect to the WebSocket and start immediately receiving logs.

websocat wss://logs.cloudflare.com/instant-logs/ws/sessions/<SESSION_ID>

Response:

Once connected to the websocket, you will receive messages of line-delimited JSON.

Now that you have a connection to Cloudflare's websocket and are receiving logs from Cloudflare's global network, you can start slicing and dicing the logs. A handy tool for this is Angle Grinder ↗︎. Angle Grinder lets you apply filtering, transformations and aggregations on stdin with first class JSON support. For example, to get the number of visitors from each country you can sum the number of events by the ClientCountry field.

websocat wss://logs.cloudflare.com/instant-logs/ws/sessions/<SESSION_ID> | agrind '* | json | sum(sampleInterval) by ClientCountry'

Response:

For the moment, HTTP Requests is the only dataset supported. In the future, we will expand to other datasets.

You can download the table of logs that appears in the dashboard, in JSON format via the Export button.

Instant Logs has three limits set in place:

• Only one active Instant Logs session per zone.
• Maximum session time is 60 minutes.
• If you stop listening to a socket for more than five minutes.

If either of these limits are reached, the logs stream will automatically stop.

If you have any feature requests or notice any bugs, share your feedback directly with us by joining the Cloudflare Developers community on Discord ↗︎.