9 // Cloudflare Tunnel

What is Cloudflare Tunnel?

Cloudflare Tunnel exposes a service running behind a private network (an AWS VM, a home server, etc.) to the public internet through Cloudflare's edge — without opening any inbound ports. The VM runs the cloudflared daemon, which makes an outbound connection to Cloudflare; the tunnel terminates inside the Cloudflare account and traffic is routed to a hostname on a configured domain.

Overview

Cloudflare Tunnel is used here to make the Dagster webserver (running on an EC2 instance) reachable to n8n Cloud so that Pipeline 3 in 8 // n8n can launch a Dagster job via the GraphQL API. n8n Cloud's dynamic egress IPs never need to be whitelisted because the tunnel is an outbound connection initiated by the VM, not an inbound one accepted by it.

The end result: https://dagster.example.com resolves through Cloudflare to the Dagster UI/GraphQL endpoint, with no public ports opened on the VM. The hostname is protected by Cloudflare Access, so requests are rejected unless they include the CF-Access-Client-Id and CF-Access-Client-Secret headers from the n8n service token created in Cloudflare Zero Trust.

Prerequisites

A Cloudflare account with a domain whose nameservers point at Cloudflare.
SSH access to the AWS VM running Dagster
Dagster running on localhost:3000 on the VM (default port).

This guide uses the following placeholder values:

Placeholder	Meaning
`dagster.example.com`	Public hostname routed through Cloudflare Tunnel
`example.com`	Domain managed by Cloudflare
`dagster`	Subdomain for the Dagster hostname
`dagster-tunnel`	Cloudflare Tunnel name
`<tunnel-uuid>`	Tunnel credentials UUID generated by Cloudflare
`<repository-location>`	Dagster repository location name
`<repository-name>`	Dagster repository name
`<dagster-job-name>`	Dagster job to launch from n8n

Setup

1. Install cloudflared on the VM

For ARM64 VMs, use the linux-arm64 binary. For x86_64 VMs, use the matching linux-amd64 binary.

sudo curl -L \
  https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-arm64 \
  -o /usr/local/bin/cloudflared

sudo chmod +x /usr/local/bin/cloudflared

cloudflared --version

2. Authenticate with Cloudflare

cloudflared tunnel login

This prints a URL. Open it in a local browser, log in to Cloudflare, and pick the relevant domain from the list. cloudflared writes a cert to ~/.cloudflared/cert.pem on the VM.

3. Create the Tunnel

cloudflared tunnel create dagster-tunnel

This prints a tunnel UUID and writes ~/.cloudflared/<tunnel-uuid>.json (the tunnel credentials). The UUID goes into the config file in step 6.

4. Route DNS

cloudflared tunnel route dns dagster-tunnel dagster.example.com

This adds a CNAME from dagster.example.com → the tunnel automatically. Verify in the Cloudflare dashboard under DNS that a new proxied (orange-cloud) CNAME entry was added.

5. Move Credentials to a System Location

Both cert.pem and the tunnel credentials JSON live in the user's ~/.cloudflared/ directory by default, but the systemd service runs as a different user that can't read /home/ubuntu/. Create the system config directory and copy both files there.

sudo mkdir -p /etc/cloudflared
sudo cp /home/ubuntu/.cloudflared/cert.pem /etc/cloudflared/
sudo cp /home/ubuntu/.cloudflared/*.json /etc/cloudflared/
sudo chmod 600 /etc/cloudflared/cert.pem /etc/cloudflared/*.json

The chmod 600 locks both files to owner read/write only — they're credentials and should be treated like SSH keys.

6. Write the Tunnel Config

Write the YAML pointing at the credentials file from step 5 and routing the hostname to Dagster's port.

sudo tee /etc/cloudflared/config.yml > /dev/null << EOF
tunnel: dagster-tunnel
credentials-file: /etc/cloudflared/<tunnel-uuid>.json

ingress:
  - hostname: dagster.example.com
    service: http://localhost:3000
  - service: http_status:404
EOF

Verify the config:

sudo cat /etc/cloudflared/config.yml

The credentials-file: line should read /etc/cloudflared/<tunnel-uuid>.json. The trailing http_status:404 rule is mandatory since Cloudflare's tunnel ingress always requires a catch-all.

7. Install as a `systemd` Service

sudo cloudflared service install

sudo systemctl status cloudflared

Look for active (running) in the status output. The service is set to start on boot automatically.

If the service fails to start, the most common diagnostic is:

sudo journalctl -u cloudflared -n 30 --no-pager

8. Verify

From a machine outside the VM:

curl -fsSI https://dagster.example.com/server_info

A 200 OK response with the x-dagster-call-counts header confirms the tunnel is forwarding traffic to Dagster end-to-end. This check assumes Cloudflare Access has not been configured yet. At this stage, the tunnel is public to the internet. It will be secured in Locking Down the Tunnel.

Calling the Dagster GraphQL API

Once the tunnel is live, the Dagster GraphQL endpoint is reachable at https://dagster.example.com/graphql. The examples in this section assume Cloudflare Access has not been configured yet. After the tunnel is locked down, include the service token headers shown in Locking Down the Tunnel.

Discovering Jobs

To enumerate available repositories, locations, and jobs:

curl -fsS https://dagster.example.com/graphql \
  -H "Content-Type: application/json" \
  -d '{"query":"{ workspaceOrError { ... on Workspace { locationEntries { name locationOrLoadError { ... on RepositoryLocation { repositories { name jobs { name } } } } } } } }"}' \
  | jq '.data.workspaceOrError.locationEntries[] | {location: .name, repos: .locationOrLoadError.repositories | map({name, jobs: .jobs | map(.name)})}'

The output's location and repos[].name values are used in the launchRun mutation below.

Field	Value
`repositoryLocationName`	`<repository-location>`
`repositoryName`	`<repository-name>`

Launching a Job

The launchRun mutation in this Dagster version takes executionParams: ExecutionParams!, with the selector nested inside. The selector field for the job name is pipelineName. Dagster's GraphQL retains the older "pipeline" terminology even for asset jobs.

curl -fsS https://dagster.example.com/graphql \
  -H "Content-Type: application/json" \
  -d '{
    "query": "mutation LaunchRun($p: ExecutionParams!) { launchRun(executionParams: $p) { __typename ... on LaunchRunSuccess { run { runId status } } ... on PythonError { message } ... on InvalidSubsetError { message } ... on RunConfigValidationInvalid { errors { message } } } }",
    "variables": {
      "p": {
        "selector": {
          "repositoryLocationName": "<repository-location>",
          "repositoryName": "<repository-name>",
          "pipelineName": "<dagster-job-name>"
        }
      }
    }
  }' | jq .

Expected response on success:

{
  "data": {
    "launchRun": {
      "__typename": "LaunchRunSuccess",
      "run": {
        "runId": "<run-uuid>",
        "status": "QUEUED"
      }
    }
  }
}

The run will then appear in the Dagster UI under Runs, transition through QUEUED → STARTED → SUCCESS, and on completion fire the existing run-status webhook into n8n's Pipeline 1 for Discord notification.

Locking Down the Tunnel

The tunnel hostname should not be left publicly reachable. Cloudflare Access can require a service token before traffic reaches Dagster, which lets n8n call the GraphQL endpoint without exposing the UI or API to the open internet.

1. Open Zero Trust

From the main Cloudflare dashboard, select the account and use the left-hand menu to open Zero Trust.

If this is the first Zero Trust setup for the account, Cloudflare asks for a team name. A project or account name is sufficient, then the Free plan can be selected.

2. Create the Service Credential

In the Zero Trust dashboard:

Go to Access controls → Service credentials → Service Tokens.
Create a service token named n8n-dagster-trigger.
Copy both generated values:
- CF-Access-Client-Id
- CF-Access-Client-Secret

Warning

Cloudflare only shows the client secret once. Store it in the n8n HTTP Request node credentials or another secure secret store before leaving the page.

3. Create the Access Policy

In Access → Applications, start a new self-hosted application and create a policy with:

Field	Value
Policy name	`n8n service token`
Action	`Service Auth`
Rule type	`Include`
Selector	`Service Token`
Value	`n8n-dagster-trigger`

If the Value dropdown has no service token options, create the service token first, then refresh the application page.

4. Create the Application

Configure the self-hosted application:

Field	Value
Application type	`Self-hosted`
Subdomain	`dagster`
Domain	`example.com`
Path	leave blank
Browser-based RDP, SSH, or VNC sessions	off
Policy	`n8n service token`

Leaving Path blank protects the whole dagster.example.com hostname. That is intentional: both the Dagster UI and /graphql endpoint should be behind Access.

5. Test Access

Without the service token headers, Cloudflare should block the request:

curl -fsS https://dagster.example.com/graphql

Expected result: 403 or a Cloudflare Access login/block response.

With the service token headers, Cloudflare should forward the request to Dagster:

curl -fsS \
  -H "CF-Access-Client-Id: <client-id>" \
  -H "CF-Access-Client-Secret: <client-secret>" \
  https://dagster.example.com/graphql

Expected result: a Dagster response. A 400 with No GraphQL query found in the request is a successful Access test because it proves the request reached Dagster and only failed because no GraphQL body was sent.

Cloudflare validates these headers at the edge before forwarding to Dagster. No VM-side changes are required.

Wiring n8n to the Locked-Down Tunnel

The n8n HTTP Request node that closes Pipeline 3's loop uses the same launchRun mutation body from Launching a Job. Settings:

Field	Value
Method	`POST`
URL	`https://dagster.example.com/graphql`
Authentication	None
Send Headers	enabled, with `Content-Type: application/json`, `CF-Access-Client-Id`, and `CF-Access-Client-Secret`
Send Body	enabled
Body Content Type	JSON
Specify Body	Using JSON
Response → Response Format	JSON

The Cloudflare Access headers use the service token values created in Locking Down the Tunnel:

Header	Value
`CF-Access-Client-Id`	the service token client ID
`CF-Access-Client-Secret`	the service token client secret

Body:

{
  "query": "mutation LaunchRun($p: ExecutionParams!) { launchRun(executionParams: $p) { __typename ... on LaunchRunSuccess { run { runId } } ... on PythonError { message } ... on InvalidSubsetError { message } ... on RunConfigValidationInvalid { errors { message } } } }",
  "variables": {
    "p": {
      "selector": {
        "repositoryLocationName": "<repository-location>",
        "repositoryName": "<repository-name>",
        "pipelineName": "<dagster-job-name>"
      }
    }
  }
}

A successful execution shows data.launchRun.__typename = "LaunchRunSuccess" plus a runId in the n8n output panel, and a fresh run appears in the Dagster UI within seconds.

Related: n8n | AWS