← Back to posts How We Fixed Turbo Remote Cache 502s by Bypassing a Turborepo Client Bug
The Problem
This is a sequel to Ditching Nx for Turborepo. If you haven’t read that one, the short version: we migrated our monorepo from Nx to Turborepo and deployed a self-hosted remote cache on Lambda. That cache had a problem.
Our CI pipeline was taking 30+ minutes from merge to production. The single biggest bottleneck: CDK synth running from scratch on every CI stage because the Turbo remote cache silently failed on large artifacts.
The symptom was a non-fatal warning:
WARNING failed to contact remote cache: HTTP status server error (502 Bad Gateway)Turbo proceeded without caching, so CI still worked - just slowly. CDK synth ran 3 times per merge (PR checks, staging, production) at 4-5 minutes each. That’s 13 minutes of wasted compute on every single merge.
Root Cause: Lambda Function URL 6MB Limit
Our Turbo remote cache ran the turborepo-remote-cache package on a Lambda behind a Function URL. Lambda Function URLs have a hard 6MB request payload limit in buffered mode. Our CDK synth outputs (CloudFormation templates + bundled Lambda assets) compressed to 189MB for one app and 626MB for another. That’s not normal for CDK output. The bloat came from bundling Lambda assets inside the synth output rather than packaging them separately, and we later fixed this with a CDK synth/deploy split (a topic for its own post). But at the time, these were the artifact sizes we had to deal with.
The 502 occurred at the Function URL layer before Lambda was even invoked. The Lambda itself was perfectly healthy: zero errors, zero throttles, all invocations completing in 100-700ms.
The Obvious Fix That Didn’t Work: S3 Presigned URLs
The natural solution: don’t proxy artifact data through Lambda. Generate S3 presigned URLs and let Turbo upload directly to S3.
Turborepo supports this via its --preflight flag. The flow:
- Turbo sends OPTIONS to the cache server
- Server returns a presigned S3 URL in the
Locationheader - Turbo follows the URL and uploads directly to S3
We implemented this. It failed with SignatureDoesNotMatch.
After hours of debugging (AWS SDK checksum injection, query parameter forwarding, different presigning strategies), we traced the issue to Turborepo’s source code.
The Turborepo Client Bug
In crates/turborepo-api-client/src/lib.rs, the put_artifact and get_artifact functions:
1. request_url = preflight_response.location // presigned S3 URL
2. request_builder = api_request(method, request_url)
3. request_builder = add_team_params(request_builder, team_id, team_slug) // APPENDS ?slug=teamadd_team_params() is called unconditionally - even after the URL has been replaced with a presigned S3 URL. It appends slug=team to the presigned URL. S3 includes all query parameters in signature verification. The appended parameter invalidates the signature.
This is a confirmed client-side bug. No issue has been filed (as of May 2026). It affects all presigned URL implementations when preflight is enabled.
This also means existing open-source solutions (EloB/turborepo-remote-cache-lambda, gpdenny/turborepo-s3-cache) likely don’t work with current Turbo versions despite appearing functional in their READMEs.
The Solution: CloudFront as an Auth Gateway
Since we can’t use S3 presigned URLs (Turbo breaks them) and we can’t proxy through Lambda (6MB limit), we needed a third path: CloudFront + CloudFront Function + Origin Access Control.
The architecture:
Turbo -> OPTIONS -> Lambda (preflight)
|
Returns HMAC-signed CloudFront URL
|
Turbo -> PUT -> CloudFront
|
CloudFront Function:
- Validates HMAC token
- Strips ALL query params (including Turbo's slug)
- Forwards clean request to S3
|
S3 via Origin Access ControlThe key insight: CloudFront Functions can strip query parameters before the request reaches S3. Turbo can append whatever it wants - the CloudFront Function removes it all. S3 never sees the extra params.
How It Works
Lambda (preflight only):
- Receives OPTIONS from Turbo
- Validates Bearer token
- Generates an HMAC:
SHA256(secret, "{method}:{path}:{expires}") - Returns a CloudFront URL with the HMAC token as a query param
CloudFront Function (viewer-request, ~50 lines ES5):
- Validates the HMAC token
- Checks expiry
- Verifies HTTP method matches signed method
- Strips ALL query parameters
- Forwards clean request to S3
CloudFront Distribution:
- S3 origin with Origin Access Control (supports GET, PUT, DELETE)
- Caching disabled (pass-through to S3)
- No payload size limit (CloudFront handles up to 64GB)
Lessons Learned Along the Way
CloudFront Function Gotchas
Three things we hit that aren’t obvious from the docs:
request.methodis read-only. It’s the only read-only field on the request object. Attempting to set it returnsFunctionValidationError(503). We originally tried to set the method from our signed param - instead we validate it matches.- Response body uses
data, notvalue. The response body format is{ encoding: "text", data: "..." }. Usingvalueinstead ofdatacausesFunctionValidationErrorwith the unhelpful message “body data is missing.” Fn::Subuses${VarName}syntax. We injected the HMAC secret into the function source at deploy time via CloudFormationFn::Sub. The placeholder must be${__HMAC_SECRET__}, not__HMAC_SECRET__. Without the${}, CloudFormation passes the string through unchanged.
Cost
The entire CloudFront setup adds ~$0.50/month. CloudFront Function invocations are free (first 10M/month). Data transfer for CI cache traffic is negligible.
Results
| Metric | Before | After |
|---|---|---|
| CDK synth (6 domains) | 4-5 minutes | 4 seconds |
| Build backend job | 4-5 minutes | 57 seconds |
| 502 warnings | Every run | Zero |
| Merge-to-production pipeline | ~35 minutes | ~25 minutes |
The CDK synth cache hit prints >>> FULL TURBO - all 6 domains replayed from cache in under 4 seconds total.
Should You File the Turbo Bug?
Yes. The add_team_params issue is straightforward to fix upstream: skip the call when preflight was used (the team context was already baked into the preflight request). Until it’s fixed, the CloudFront gateway pattern works as a robust workaround.
Tech Stack
- AWS CDK 2.x (TypeScript)
- CloudFront + CloudFront Function (JS 2.0 runtime)
- S3 + Origin Access Control
- Lambda Function URL (preflight only)
- Turborepo 2.9.x with
TURBO_PREFLIGHT=true
How We Fixed Our Broken Build Cache (By Working Around a Bug We Found in the Tool)
The Problem Nobody Warned Us About
This is a sequel to Ditching Nx for Turborepo. The short version of that story: we switched our monorepo tooling and deployed a self-hosted remote cache server. That cache had a problem.
Our CI pipeline was taking over 30 minutes from code merge to production deployment. The single biggest culprit: a step called CDK synth, which turns our infrastructure code into deployment templates. This step was running from scratch every single time, because the cache was silently failing.
The failure showed up as a warning message that was easy to miss:
WARNING failed to contact remote cache: HTTP status server error (502 Bad Gateway)The build still completed. Everything still worked. Just slowly. This step ran three times per deployment (once for pull request checks, once for staging, once for production), at four to five minutes each. That’s 13 minutes of avoidable waiting on every single code merge.
Why the Cache Was Failing
Our cache server ran as a serverless Lambda function. Lambda functions accessed via a direct URL have a hard limit: they can only receive requests up to 6 megabytes in size.
Our infrastructure templates, which the cache was supposed to store, compressed down to 189 megabytes for one app and 626 megabytes for another. These were unusually large because of how we were packaging things at the time (a separate story), but that was what we were working with.
The error happened before our cache server even saw the request. The thing sitting in front of Lambda rejected the upload at the network level. Our actual server was perfectly healthy: zero errors in the logs, all requests completing in under a second. The problem was architectural.
The Obvious Fix That Didn’t Work
The natural solution to “don’t send large files through Lambda” is to upload them directly to S3 storage instead. The caching tool (Turborepo) has a built-in feature for this: the cache server can tell the tool “here’s a direct link to S3, upload there instead.” The tool then bypasses the cache server entirely for the actual file upload.
We built this. We spent a day on it. It failed with a cryptic signature error from AWS.
After more debugging than I care to admit, we traced the problem to the caching tool’s own source code.
A Bug in Turborepo
When Turborepo receives a direct S3 upload link from the cache server, it’s supposed to use that link as-is. Instead, it appends extra query parameters to the URL, including a team identifier.
This matters because S3 direct upload links work through a signature mechanism: the link is cryptographically signed for a specific URL. Any modification to the URL, even adding a harmless-looking parameter, invalidates the signature. S3 rejects the upload.
Turborepo was breaking its own feature.
As of when we hit this (May 2026), no fix existed and no issue had been filed. This also means existing open-source self-hosted cache solutions that advertise direct S3 upload support likely don’t work with current versions of Turborepo, despite appearing functional in their documentation.
The Solution We Built
Since we couldn’t upload directly to S3 (Turborepo breaks the URLs) and couldn’t proxy through our server (Lambda’s size limit), we needed a third path. We used AWS CloudFront as an intermediary.
Here’s how it works:
When the caching tool asks to upload something, our cache server generates a time-limited authorisation token and returns a CloudFront URL with that token attached. The tool then uploads directly to CloudFront.
At the CloudFront layer, a small piece of code runs before the request reaches S3. This code validates the authorisation token, checks it hasn’t expired, verifies the upload method is what was originally authorised, and then strips all query parameters off the URL before forwarding it to S3.
That last step is the key insight. Turborepo appends its team parameter to the URL. Our CloudFront code strips all parameters. S3 receives a clean URL with no unexpected additions. The signature verification passes. The upload succeeds.
CloudFront has no meaningful size limit for uploads (it handles up to 64 gigabytes). The entire setup costs about 50 pence per month.
Things That Were Harder Than Expected
Building the CloudFront layer involved a few surprises that aren’t obvious from the documentation:
The request method field in CloudFront’s code environment is read-only. We wanted to use it to verify the upload method matches what was signed. You can read it but cannot set it, which is fine for our purposes but caused some confusion when we initially tried to manipulate it.
The format for returning a response body from CloudFront code uses a specific field name (data) rather than the more intuitive value. Using the wrong field name produces an unhelpful error message.
Injecting our authorisation secret into the CloudFront code at deployment time required a specific placeholder syntax. Without the right format, the deployment template passes the placeholder through unchanged instead of substituting the actual secret value.
None of these are major obstacles, but each one cost us time to diagnose.
The Results
| What | Before | After |
|---|---|---|
| Infrastructure code compilation (6 apps) | 4-5 minutes each | 4 seconds total |
| Backend build job | 4-5 minutes | 57 seconds |
| Cache errors | Every run | Zero |
| Total merge-to-production time | About 35 minutes | About 25 minutes |
When the cache is working properly, all six of our infrastructure apps replay from cache in under four seconds total. A process that used to consume a significant chunk of every CI run now barely registers.
Should Someone Fix This in Turborepo?
Yes. The fix in the tool itself would be straightforward: stop appending team parameters to the URL after it’s been replaced with a direct upload link. The team information was already included in the original request to the cache server before the redirect happened.
Until that fix exists, the CloudFront approach works reliably. We’ve been running it since and haven’t seen a single cache failure.