`--file` treats query params in HTTP URLs as path assignments

[mamachanko]

What steps did you take:

ytt the bootstrapper resources I am given by Tanzu Platform in form of a HTTP URL which contains query params

What happened:

ytt -f "https://example.com/?id=123&source=hello"

ytt: Error: Expected file 'https://example.com/?id=123&source=hello' to only have single '=' sign to for relative path assignment

What did you expect:

ytt fetches the file and templates it.

Anything else you would like to add:

I can workaround the issue with, say

http "https://example.com/?id=123&source=hello" | ytt -f-

Environment:

• ytt version (use ytt --version): 0.50.0
• OS (e.g. from /etc/os-release): macOS

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[100mik]

Nice catch! We should prioritise this fix sooner rather than later. But it is handy that we have a workaround.
I will mark this as a good first issue in case someone is looking to lend a hand 🙌🏼

[ernstvonoelsen]

@100mik @renuy

here's how I tested my suggested solution:

$ git branch
  develop
* issue-928
$
$
$
$ ./hack/build.sh
++ git describe --tags
++ grep -Eo '[0-9]+\.[0-9]+\.[0-9]+'
+ LATEST_GIT_TAG=0.51.0
+ VERSION=0.51.0
+ export CGO_ENABLED=0
+ CGO_ENABLED=0
+ LDFLAGS='-X carvel.dev/ytt/pkg/version.Version=0.51.0'
+ rm -f website/generated.go
++ go list ./...
++ grep -v yaml.v2
+ go fmt carvel.dev/ytt/cmd/ytt carvel.dev/ytt/cmd/ytt-lambda-website carvel.dev/ytt/pkg carvel.dev/ytt/pkg/cmd carvel.dev/ytt/pkg/cmd/template carvel.dev/ytt/pkg/cmd/ui carvel.dev/ytt/pkg/experiments carvel.dev/ytt/pkg/filepos carvel.dev/ytt/pkg/files carvel.dev/ytt/pkg/orderedmap carvel.dev/ytt/pkg/schema carvel.dev/ytt/pkg/spell carvel.dev/ytt/pkg/template carvel.dev/ytt/pkg/template/core carvel.dev/ytt/pkg/texttemplate carvel.dev/ytt/pkg/validations carvel.dev/ytt/pkg/version carvel.dev/ytt/pkg/website carvel.dev/ytt/pkg/workspace carvel.dev/ytt/pkg/workspace/datavalues carvel.dev/ytt/pkg/workspace/ref carvel.dev/ytt/pkg/yamlfmt carvel.dev/ytt/pkg/yamlmeta carvel.dev/ytt/pkg/yamltemplate carvel.dev/ytt/pkg/yttlibrary carvel.dev/ytt/pkg/yttlibrary/overlay carvel.dev/ytt/pkg/yttlibraryext carvel.dev/ytt/pkg/yttlibraryext/toml carvel.dev/ytt/test/e2e carvel.dev/ytt/test/filetests
pkg/website/generated.go
+ go mod vendor
+ go mod tidy
+ ./hack/generate-website-assets.sh
+ rm -f pkg/website/generated.go
+ cd pkg/website
+ go run ./../../cmd/ytt -f . -f ../../examples/playground/basics -f ../../examples/playground/overlays -f ../../examples/playground/getting-started --file-mark 'alt-example**/*:type=data' --file-mark 'example**/*:type=data' --file-mark generated.go.txt:exclusive-for-output=true --dangerous-emptied-output-directory ../../tmp/
creating: ../../tmp/generated.go.txt
+ mv tmp/generated.go.txt pkg/website/generated.go
+ go build '-ldflags=-X carvel.dev/ytt/pkg/version.Version=0.51.0' -trimpath -o ytt ./cmd/ytt/...
+ ./ytt version
ytt version 0.51.0
+ export GOOS=linux GOARCH=amd64
+ GOOS=linux
+ GOARCH=amd64
+ go build '-ldflags=-X carvel.dev/ytt/pkg/version.Version=0.51.0' -trimpath -o ./tmp/ytt ./cmd/ytt/...
+ go build '-ldflags=-X carvel.dev/ytt/pkg/version.Version=0.51.0' -trimpath -o ./tmp/main ./cmd/ytt-lambda-website/...
+ cd tmp
+ chmod +x main ytt
+ rm -f ytt-lambda-website.zip
+ zip ytt-lambda-website.zip main ytt
  adding: main (deflated 47%)
  adding: ytt (deflated 47%)
+ echo SUCCESS
SUCCESS
$
$
$
$ python3 -m http.server --directory yaml_resources/ &
[1] 4309
$
$ ./ytt --file http://localhost:8000/test.yaml?hello=world
127.0.0.1 - - [05/Dec/2024 05:42:06] "GET /test.yaml?hello=world HTTP/1.1" 200 -
hello: world
foo:
- bar1
- bar2
$

where

$ cat yaml_resources/test.yaml 
# comment
hello: world
foo:
  - bar1
  - bar2

is the content of the fetched template file which indeed is processed correctly.

So from the above protocol, one can deduce that the server receives the full request, including the query params. Calculating an HTTPSource.relPath with explicitly cropped query params does hence not affect the source caching/fetching.

[vsolano9]

Heads-up for anyone picking this up from the good first issue label: this appears to have been fixed already.

PR #934 ("Issue 928") is merged — merge commit faba705f, merged 2025-01-30, touching pkg/files/file.go, pkg/files/sources.go and pkg/files/file_test.go.

I checked that the commit actually shipped rather than assuming the merge was enough. GitHub's compare API reports faba705f as ahead_by: 0 against both develop and the latest release tag v0.55.1, which means it is an ancestor of both — so the fix is in develop and in every release from v0.55.1 back to whichever one first carried it.

Stated honestly: I verified that the fix is present, by commit ancestry and by reading the diff. I did not build the binary and re-run the original --file reproduction, so I can't personally confirm the behaviour is correct now — a maintainer may want to do that before closing.

The reason for the comment is just that an open issue labelled good first issue and carvel accepted is exactly what a new contributor picks up, and the work is already done.