Skip to main content

Logs Uploader

Logs Uploader exports CDN request and cache logs to a configured destination in batches at regular intervals. When Origin Shielding is enabled and the policy includes those logs, the export also contains requests processed by mid-tier cache servers.

To export logs, create a policy that defines what to export, connect a target that receives the logs, then create a configuration that links them to CDN resources.

Accounts that already use the hourly Raw Logs export to Amazon Simple Storage Service (S3), File Transfer Protocol (FTP), or SSH File Transfer Protocol (SFTP) can keep that configuration. Use Logs Uploader for new setups. A new Amazon S3 destination uses the S3 Amazon target. To migrate an existing Raw Logs export, contact CDB Support.

Logs Uploader must be enabled for the account before the configuration UI is available. Contact CDB Support or use Contact support on the Logs Uploader page in the CDB Technical Web Portal.

Create a policy

A policy selects the log fields, the file name format, and how often files rotate. The Time interval is 5–60 minutes and defaults to 5. If no requests arrive during an interval and Include empty logs is selected, Logs Uploader still sends an empty batch. For a file-based target, the resulting empty log file is about 20 bytes.

In the Customer Portal, open CDN → Logs → Logs Uploader, then Policies, and click Add policy.

1. Enter a name.

2. Leave Include empty logs cleared to skip empty intervals, or select it to confirm delivery when there is no traffic.

3. Select Include logs from origin shielding when Origin Shielding is enabled and those mid-tier requests should appear in the export.

4. Select the log fields. The field table lists each field and its meaning.

5. Set delimiter, separator, file name format, time interval, and file size limit. The file size limit applies before compression — a compressed file can be up to 99% smaller.

6. Click Add policy.

To change an existing policy, open it by name. The same settings apply; click Save changes when done.

Create a target

A target is the destination. Authentication complete means Logs Uploader can reach and authenticate to that destination — it does not mean a batch has arrived.

On the Targets tab, click Add target, enter a name, and configure one destination:

CDB Object Storage

  1. Keep S3 CDB selected in Receive logs using.
  2. In Storage, select an existing Object Storage or create one from the list.
  3. In Bucket, select a bucket or create one from the list.
  4. Optionally, enter an existing Folder inside the bucket.

Access keys are not entered for CDB Object Storage. Only Standard storage is supported when creating storage from this form.

Alibaba Cloud OSS

  1. Enter the Access Key ID and Secret Access Key.
  2. Enter the cloud region and an existing bucket.
  3. Optionally, specify an existing folder inside the bucket.

Alibaba documents OSS AccessKey pair creation.

Alibaba Cloud SLS

  1. Enter the Region (such as cn-hangzhou).
  2. Enter the Project name (3–63 characters; lowercase letters, digits, and hyphens).
  3. Enter the Logstore name (3–36 characters; lowercase letters, digits, hyphens, and underscores).
  4. Enter the Access Key ID and Secret Access Key.
  5. Optionally, enter the Endpoint to override region.log.aliyuncs.com, and a Topic (up to 128 characters).

Alibaba documents SLS AccessKey pair creation and project setup in the SLS quick start.

Amazon S3

  1. Enter the Access Key ID and Secret Access Key.
  2. Enter the region and an existing bucket.
  3. Optionally, specify an existing folder inside the bucket.

Generic S3-compatible object storage

  1. Select HTTP or HTTPS and specify the Endpoint.
  2. Enter the Access Key ID and Secret Access Key.
  3. Enter the storage region and an existing bucket.
  4. Optionally, specify an existing folder inside the bucket.

FTP transmits credentials and log data without transport encryption. Use SFTP or HTTPS when logs contain IP addresses, cookies, or request URLs.

  1. Enter the Hostname, Login, and Password.
  2. Optionally, specify an existing destination directory.
  1. Enter the Hostname and Login.
  2. Authenticate with a Password, or with a Private Key and Passphrase.
  3. Optionally, specify an existing destination folder.

Plain HTTP transmits log data without transport encryption. Use HTTPS when logs contain IP addresses, cookies, or request URLs.

  1. Select POST or PUT, then HTTP or HTTPS, and specify the Endpoint.
  2. Add custom headers if the receiver requires them.
  3. Turn on Authentication for token-based header auth, and Use compression to gzip the JSON body (up to 99% smaller).

Click Add target. A green Authentication complete status confirms the destination is reachable. Click the refresh icon next to the status to run the check again.

To change an existing target, open it by name. The same connection settings apply.

Create a configuration

A configuration links one policy and one target to CDN resources and starts delivery.

On the Configurations tab, click Add configuration.

1. Enter a name.

2. Cover All Resources (including resources created later) or select Specific Resource(s) from the list.

3. Select the policy and the target.

4. Leave Enable configuration on to start delivery, or turn it off to save the setup inactive.

5. Click Add configuration.

The Status column repeats the linked target's authentication state. To change an existing configuration, open it by name — the same resource, policy, and target settings apply.

Verify delivery

Generate traffic to a CDN resource covered by the configuration. After one full time interval, check the destination for a delivered batch and confirm the expected fields. For a file-based target, check for a new file. For HTTP(S), confirm the configured POST or PUT request. For SLS, check the configured logstore. Authentication complete is not proof that a batch arrived — see Troubleshooting if the destination stays empty.

Disable or delete a setup

Use ⋯ on a configuration to Disable delivery without deleting the setup, or Enable it again. The same switch is in the configuration form.

Delete the configuration first. A policy or target that is still linked to a configuration cannot be deleted. Then delete the unused target or policy from ⋯ on its row.

Export CDN resource logs by creating a policy, a target, and a configuration that links them. Logs Uploader must be enabled for the account — a disabled account returns HTTP 403. A CDN resource ID is visible in the Customer Portal under CDN → CDN Resources, or from the List CDN resources API call.

An API token is required.

export GCORE_API_KEY="{YOUR_API_KEY}"
export CDN_RESOURCE_ID="{YOUR_RESOURCE_ID}"

Create a Logs Uploader configuration

Create a policy (rotate_interval_minutes is 5–60), an HTTP target, and a configuration for one CDN resource. Other storage_type values include s3_gcore, s3_amazon, s3_oss, s3_other, ftp, sftp, and sls. An HTTP receiver must return 2xx, or Logs Uploader retries the batch. Target validation starts asynchronously — the first status is in_progress.

import os

from gcore import CDB

client = CDB()
resource_id = int(os.environ["CDN_RESOURCE_ID"])

# Step 1. Create a policy
policy = client.cdn.logs_uploader.policies.create(
    name="cdn-json-export",
    fields=["cname", "status", "host"],
    include_empty_logs=False,
    rotate_interval_minutes=5,
    format_type="json",
)
print(policy.id, policy.name)

# Step 2. Create an HTTP target
target = client.cdn.logs_uploader.targets.create(
    name="http-collector",
    storage_type="http",
    config={
        "content_type": "json",
        "upload": {
            "url": "https://logs.example.com/cdn",
            "method": "POST",
        },
    },
)
print(target.id, target.storage_type)

# Step 3. Create a configuration
config = client.cdn.logs_uploader.configs.create(
    name="export-store",
    policy=policy.id,
    target=target.id,
    enabled=True,
    for_all_resources=False,
    resources=[resource_id],
)
print(config.id, config.enabled, config.policy, config.target)
package main

import (
    "context"
    "fmt"
    "os"
    "strconv"

    gcore "github.com/G-Core/gcore-go"
    "github.com/G-Core/gcore-go/cdn"
)

func main() {
    client := gcore.NewClient()
    ctx := context.Background()
    resourceID, err := strconv.ParseInt(os.Getenv("CDN_RESOURCE_ID"), 10, 64)
    if err != nil {
        panic(err)
    }

    // Step 1. Create a policy
    policy, err := client.CDN.LogsUploader.Policies.New(ctx, cdn.LogsUploaderPolicyNewParams{
        Name:                  gcore.String("cdn-json-export"),
        Fields:                []string{"cname", "status", "host"},
        IncludeEmptyLogs:      gcore.Bool(false),
        RotateIntervalMinutes: gcore.Int(5),
        FormatType:            cdn.LogsUploaderPolicyNewParamsFormatTypeJson,
    })
    if err != nil {
        panic(err)
    }
    fmt.Println(policy.ID, policy.Name)

    // Step 2. Create an HTTP target
    target, err := client.CDN.LogsUploader.Targets.New(ctx, cdn.LogsUploaderTargetNewParams{
        Name:        gcore.String("http-collector"),
        StorageType: cdn.LogsUploaderTargetNewParamsStorageTypeHTTP,
        Config: cdn.LogsUploaderTargetNewParamsConfigUnion{
            OfHTTPConfig: &cdn.LogsUploaderTargetNewParamsConfigHTTPConfig{
                Upload: cdn.LogsUploaderTargetNewParamsConfigHTTPConfigUpload{
                    URL:    "https://logs.example.com/cdn",
                    Method: "POST",
                },
                ContentType: "json",
            },
        },
    })
    if err != nil {
        panic(err)
    }
    fmt.Println(target.ID, target.StorageType)

    // Step 3. Create a configuration
    config, err := client.CDN.LogsUploader.Configs.New(ctx, cdn.LogsUploaderConfigNewParams{
        Name:            "export-store",
        Policy:          policy.ID,
        Target:          target.ID,
        Enabled:         gcore.Bool(true),
        ForAllResources: gcore.Bool(false),
        Resources:       []int64{resourceID},
    })
    if err != nil {
        panic(err)
    }
    fmt.Println(config.ID, config.Enabled)
}
POLICY_ID=$(curl -sS -X POST "https://api.cdb-staging.cdn.orange.com/cdn/logs_uploader/policies" \
  -H "Authorization: APIKey $GCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "cdn-json-export",
    "fields": ["cname", "status", "host"],
    "include_empty_logs": false,
    "rotate_interval_minutes": 5,
    "format_type": "json"
  }' | python -c "import json,sys; print(json.load(sys.stdin)['id'])")

TARGET_ID=$(curl -sS -X POST "https://api.cdb-staging.cdn.orange.com/cdn/logs_uploader/targets" \
  -H "Authorization: APIKey $GCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "http-collector",
    "storage_type": "http",
    "config": {
      "content_type": "json",
      "upload": {
        "url": "https://logs.example.com/cdn",
        "method": "POST"
      }
    }
  }' | python -c "import json,sys; print(json.load(sys.stdin)['id'])")

curl -X POST "https://api.cdb-staging.cdn.orange.com/cdn/logs_uploader/configs" \
  -H "Authorization: APIKey $GCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"export-store\",
    \"policy\": $POLICY_ID,
    \"target\": $TARGET_ID,
    \"enabled\": true,
    \"for_all_resources\": false,
    \"resources\": [$CDN_RESOURCE_ID]
  }"

Each create call returns HTTP 201. The configuration body includes policy, target, enabled, and resources.

{
  "id": 434,
  "name": "export-store",
  "enabled": true,
  "for_all_resources": false,
  "resources": [12345],
  "policy": 370,
  "target": 501
}

Generate traffic to the CDN resource, then wait one rotate_interval_minutes period and check the destination. A target status of in_progress means destination validation is still running. Authentication complete confirms that the destination is reachable and authentication succeeded, but it does not mean that a batch has arrived. Available policy field names come from GET /docs/cdn/logs_uploader/policies/fields.

List policies, targets, and configs

List each object type, or retrieve one configuration by ID.

from gcore import CDB

client = CDB()

for policy in client.cdn.logs_uploader.policies.list(limit=10):
    print(policy.id, policy.name)
for target in client.cdn.logs_uploader.targets.list(limit=10):
    print(target.id, target.name, target.storage_type)
for config in client.cdn.logs_uploader.configs.list(limit=10):
    print(config.id, config.name, config.enabled)

config = client.cdn.logs_uploader.configs.get(434)
print(config.id, config.policy, config.target)
package main

import (
    "context"
    "fmt"

    gcore "github.com/G-Core/gcore-go"
    "github.com/G-Core/gcore-go/cdn"
)

func main() {
    client := gcore.NewClient()
    ctx := context.Background()

    policies, err := client.CDN.LogsUploader.Policies.List(ctx, cdn.LogsUploaderPolicyListParams{})
    if err != nil {
        panic(err)
    }
    for _, policy := range policies.Results {
        fmt.Println(policy.ID, policy.Name)
    }

    targets, err := client.CDN.LogsUploader.Targets.List(ctx, cdn.LogsUploaderTargetListParams{})
    if err != nil {
        panic(err)
    }
    for _, target := range targets.Results {
        fmt.Println(target.ID, target.Name, target.StorageType)
    }

    configs, err := client.CDN.LogsUploader.Configs.List(ctx, cdn.LogsUploaderConfigListParams{})
    if err != nil {
        panic(err)
    }
    for _, config := range configs.Results {
        fmt.Println(config.ID, config.Name, config.Enabled)
    }

    config, err := client.CDN.LogsUploader.Configs.Get(ctx, 434)
    if err != nil {
        panic(err)
    }
    fmt.Println(config.ID, config.Policy, config.Target)
}
curl -X GET "https://api.cdb-staging.cdn.orange.com/cdn/logs_uploader/policies?limit=10" \
  -H "Authorization: APIKey $GCORE_API_KEY"

curl -X GET "https://api.cdb-staging.cdn.orange.com/cdn/logs_uploader/targets?limit=10" \
  -H "Authorization: APIKey $GCORE_API_KEY"

curl -X GET "https://api.cdb-staging.cdn.orange.com/cdn/logs_uploader/configs?limit=10" \
  -H "Authorization: APIKey $GCORE_API_KEY"

curl -X GET "https://api.cdb-staging.cdn.orange.com/cdn/logs_uploader/configs/434" \
  -H "Authorization: APIKey $GCORE_API_KEY"

Each list request returns a paginated object:

{
  "count": 1,
  "next": null,
  "previous": null,
  "results": [
    {
      "id": 434,
      "name": "export-store",
      "enabled": true,
      "policy": 370,
      "target": 501
    }
  ]
}

Disable a configuration

Set enabled to false to stop delivery without deleting the configuration. Set enabled to true to start it again.

from gcore import CDB

client = CDB()

config = client.cdn.logs_uploader.configs.update(434, enabled=False)
print(config.id, config.enabled)
package main

import (
    "context"
    "fmt"

    gcore "github.com/G-Core/gcore-go"
    "github.com/G-Core/gcore-go/cdn"
)

func main() {
    client := gcore.NewClient()
    ctx := context.Background()

    config, err := client.CDN.LogsUploader.Configs.Update(ctx, 434, cdn.LogsUploaderConfigUpdateParams{
        Enabled: gcore.Bool(false),
    })
    if err != nil {
        panic(err)
    }
    fmt.Println(config.ID, config.Enabled)
}
curl -X PATCH "https://api.cdb-staging.cdn.orange.com/cdn/logs_uploader/configs/434" \
  -H "Authorization: APIKey $GCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

The API returns HTTP 200:

{
  "id": 434,
  "name": "export-store",
  "enabled": false
}

Delete a configuration, target, and policy

Delete the configuration first. A delete request against a policy or target that is still assigned returns HTTP 400. After the configuration is gone, delete the target and the policy. A successful delete returns HTTP 204.

from gcore import CDB

client = CDB()

client.cdn.logs_uploader.configs.delete(434)
client.cdn.logs_uploader.targets.delete(501)
client.cdn.logs_uploader.policies.delete(370)
package main

import (
    "context"

    gcore "github.com/G-Core/gcore-go"
)

func main() {
    client := gcore.NewClient()
    ctx := context.Background()

    if err := client.CDN.LogsUploader.Configs.Delete(ctx, 434); err != nil {
        panic(err)
    }
    if err := client.CDN.LogsUploader.Targets.Delete(ctx, 501); err != nil {
        panic(err)
    }
    if err := client.CDN.LogsUploader.Policies.Delete(ctx, 370); err != nil {
        panic(err)
    }
}
curl -X DELETE "https://api.cdb-staging.cdn.orange.com/cdn/logs_uploader/configs/434" \
  -H "Authorization: APIKey $GCORE_API_KEY"

curl -X DELETE "https://api.cdb-staging.cdn.orange.com/cdn/logs_uploader/targets/501" \
  -H "Authorization: APIKey $GCORE_API_KEY"

curl -X DELETE "https://api.cdb-staging.cdn.orange.com/cdn/logs_uploader/policies/370" \
  -H "Authorization: APIKey $GCORE_API_KEY"

A successful delete returns HTTP 204 with an empty body.

Export CDN request logs for one CDN resource with the Terraform provider. Apply creates a policy, an HTTP target, and a configuration that posts JSON batches to that target on the policy interval. Logs Uploader must already be enabled for the account, or apply returns HTTP 403.

Create the logs export

The policy selects the fields and the JSON format. The target posts each batch to an HTTP endpoint that returns a 2xx status, because any other status is retried. The configuration attaches both to the CDN resource already declared in the same file, so the export covers that resource.

resource "gcore_cdn_logs_uploader_policy" "json" {
  name                    = "cdn-json-export"
  fields                  = ["cname", "status", "host"]
  include_empty_logs      = false
  rotate_interval_minutes = 5
  format_type             = "json"
}

resource "gcore_cdn_logs_uploader_target" "http" {
  name         = "http-collector"
  storage_type = "http"

  config = {
    content_type = "json"
    upload = {
      url    = "https://logs.example.com/cdn"
      method = "POST"
    }
  }
}

resource "gcore_cdn_logs_uploader_config" "store" {
  name              = "export-store"
  policy            = gcore_cdn_logs_uploader_policy.json.id
  target            = gcore_cdn_logs_uploader_target.http.id
  enabled           = true
  for_all_resources = false
  resources         = [gcore_cdn_resource.main.id]
}

Check the destination

Generate traffic to the CDN resource. After one rotate interval, the HTTP endpoint receives a POST whose body contains the selected fields. A reachable target confirms contact with the endpoint, so check the destination for the batch itself.

Stop delivery

Set enabled to false and run terraform apply to stop delivery while the policy and the target stay in place. Set enabled to true and run terraform apply to start delivery again.

resource "gcore_cdn_logs_uploader_policy" "json" {
  name                    = "cdn-json-export"
  fields                  = ["cname", "status", "host"]
  include_empty_logs      = false
  rotate_interval_minutes = 5
  format_type             = "json"
}

resource "gcore_cdn_logs_uploader_target" "http" {
  name         = "http-collector"
  storage_type = "http"

  config = {
    content_type = "json"
    upload = {
      url    = "https://logs.example.com/cdn"
      method = "POST"
    }
  }
}

resource "gcore_cdn_logs_uploader_config" "store" {
  name              = "export-store"
  policy            = gcore_cdn_logs_uploader_policy.json.id
  target            = gcore_cdn_logs_uploader_target.http.id
  enabled           = false
  for_all_resources = false
  resources         = [gcore_cdn_resource.main.id]
}

Remove the logs export

Remove the three resource blocks and run terraform apply. Terraform deletes the Logs Uploader configuration first, then the policy and the target, and leaves the CDN resource in place.

# Remove or comment out these blocks:
# resource "gcore_cdn_logs_uploader_policy" "json" {
#   name                    = "cdn-json-export"
#   fields                  = ["cname", "status", "host"]
#   include_empty_logs      = false
#   rotate_interval_minutes = 5
#   format_type             = "json"
# }
#
# resource "gcore_cdn_logs_uploader_target" "http" {
#   name         = "http-collector"
#   storage_type = "http"
#
#   config = {
#     content_type = "json"
#     upload = {
#       url    = "https://logs.example.com/cdn"
#       method = "POST"
#     }
#   }
# }
#
# resource "gcore_cdn_logs_uploader_config" "store" {
#   name              = "export-store"
#   policy            = gcore_cdn_logs_uploader_policy.json.id
#   target            = gcore_cdn_logs_uploader_target.http.id
#   enabled           = true
#   for_all_resources = false
#   resources         = [gcore_cdn_resource.main.id]
# }
terraform apply

Log schema and field definitions

The exported log schema includes standard CDN request fields, internal status codes, and a sample log line.

WAAP log fields

When a CDN resource has WAAP enabled, Logs Uploader appends WAAP security event fields to each request log line. CDN access data and the matching WAAP verdict share one entry — no separate file and no extra Logs Uploader setup. When a policy override suppresses a finding, the entry includes policy_override with the override ID, target type, and target references. The field is omitted when no override matched.

WAAP fields appear only when WAAP Status is Active on CDN → CDN resources. The toggle is in that resource's Security settings. After WAAP is enabled, the next exported batch includes the new fields at the end of the line, so positional parsers keep working. Parsers that reject unknown columns must accept the WAAP fields first. The v field in HTTP(S) payloads increments when the schema changes.

DDoS protection logs are not exported through Logs Uploader. Only per-request WAAP security events are included. DDoS attack analytics remain available through the WAAP API and dashboard.

Troubleshooting

Authentication complete confirms that Logs Uploader can reach the destination. It does not mean a batch has arrived.

Batches are delivered at the end of the policy Time interval (minimum 5 minutes). Wait at least one full interval after generating CDN traffic.

If the covered resources receive no requests during an interval and Include empty logs is off, no batch is delivered. Turn the option on in the policy to confirm delivery without traffic. For a file-based target, an empty log file is about 20 bytes.

For S3, FTP, SFTP, and SLS, the bucket, directory, or logstore must already exist — Logs Uploader does not create missing paths. The credentials must allow writes: s3:PutObject (and often s3:GetBucketLocation) on S3, write access to the FTP/SFTP directory, or Write on an SLS logstore.

An HTTP(S) receiver must return 2xx before the request times out. Any other status is a delivery failure and is retried. If compression is on, the receiver must accept Content-Encoding: gzip.

A configuration limited to Specific Resources only exports those resources. All Resources includes resources created later.

Delivery error details are not shown in the portal. Contact CDB Support for delivery logs for a target or time range.