CDN Raw Log

Question

Is there a way to get raw logs of my CDN resource?

Answer

We do support raw logs of your own CDN resources to be delivered to your log server.  The raw log feature is enabled on per user account basis (not per cdn resource).

There are 2 different types of delivery:

(a) Syslog

The HTTP log format (V8) for syslog is:

$time_local<TAB>$edge_server_id<TAB>$remote_addr<TAB>$request_method<TAB>$http_host<TAB>$request_uri<TAB>$status<TAB>$bytes_sent<TAB>$upstream_http_x_cache<TAB>$upstream_response_time<TAB>$request_time<TAB>$tcpinfo_rtt<TAB>$for_operator<TAB>$resource_id<TAB>$server_name<TAB>$server_addr<TAB>$http_user_agent<TAB>$http_referer<TAB>$http_range<TAB>$body_bytes_sent<TAB>$publisher_id<TAB>$tls_version<TAB>$placeholder1<TAB>$placeholder2<TAB>$placeholder3<TAB>$placeholder4<TAB>$placeholder5<TAB>$placeholder6<TAB>$placeholder7<TAB>$placeholder8<TAB>V8

Example of HTTP log (V8) log:

00:00:00 +0000 123456789 1.2.3.5 GET test-site.com /testfile.txt 200 1086 HIT 0.000 0.002 110972 234567891 345678912 test-site.com 1.2.3.4 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36 https://www.test-site.com/test-page - 671 456789123 TLSv1.3 - - - - - - - - V8

The STREAM log format for syslog is:

date<TAB>time<TAB>edge-id<TAB>type<TAB>x-app<TAB>x-severity<TAB>x-category<TAB>x-event<TAB>date<TAB>time<TAB>c-client-id<TAB>c-ip<TAB>c-port<TAB>cs-bytes<TAB>sc-bytes<TAB>x-duration<TAB>x-sname<TAB>x-stream-id<TAB>x-spos<TAB>sc-stream-bytes<TAB>cs-stream-bytes<TAB>x-file-size<TAB>x-file-length<TAB>x-ctx<TAB>x-comment<TAB>x-sname-query

(b) SFTP and FTP:

The HTTP log format for sftp and ftp is in Combined Log Format:

$remote_addr - - [$time_local] "$request_method $request_uri HTTP/1.1" $http_status $bytes_sent "$http_referer" "$http_user_agent"

Example of HTTP log:

1.2.3.4 - - [02/Jan/2020:07:59:42 +0000] "GET /test-file.txt HTTP/1.1" 200 1000 "https://www.test-site.com/test-page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"

The STREAM log format for sftp and ftp is:

x-app<TAB>x-severity<TAB>x-category<TAB>x-event<TAB>date<TAB>time<TAB>c-client-id<TAB>c-ip<TAB>c-port<TAB>cs-bytes<TAB>sc-bytes<TAB>x-duration<TAB>x-sname<TAB>x-stream-id<TAB>x-spos<TAB>sc-stream-bytes<TAB>cs-stream-bytes<TAB>x-file-size<TAB>x-file-length<TAB>x-ctx<TAB>x-comment<TAB>x-sname-query

Our FTP raw log will be uploaded to client's FTP server every 10 minutes, and it will be removed from our server once the upload is successful. Our log server will store up to 50GB if the FTP is unreachable. However, for end user, our log server will only store up to 5GB only.

You can may insert the FTP IP address or the FTP hostname with subdirectory.

Raw log parameter:

• $upstream_response_time

Time taken for upstream to complete sending a http response from upstream to edge between the first bytes and the log write after the last bytes were sent to the edge server. It can contain multiple time taken and it is separated by commas and colons

• $upstream_http_x_cache

Value of X-Cache HTTP response header field from upstream server

• $request_time

Time elapsed between the first bytes were read from the client and the log write after the last bytes were sent to the client

• $tcpinfo_rtt

Round trip time ("ping") estimate of the tcp socket, in microseconds.

• $http_referer

HTTP request header field that identifies the address of the webpage (i.e. the URI or IRI) that is linked to the resource being requested

• $for_operator

Buyer (if any) that subscribe to the edge

To enable this feature (on a user basis), you can find the "raw log" settings at the "List CDN resource" page on CP.

If you would like to request for this feature on all the CDN resources of all the users created in CP, please send in a support ticket, and provide your log server info together with the user account which you wish to enable raw logs feature for:
(a) For FTP/SFTP , state your ftp hostname, username and password, and state it is FTP or SFTP
(b) For Syslog , state your syslog server hostname, port, TCP or UDP