Add upstream `arrow-rs-object-store` support for `AWS_REQUEST_PAYER=requester`

[ljstrnadiii]

Background

Recent commits in arrow-rs-object-store have support for AWS_REQUEST_PAYER=requester see apache/arrow-rs-object-store#669 for more information.

Goals

"Ideally we'd go commit-by-commit so that we ensure we expose all underlying features through to obstore" @kylebarron

This implies there may be additional features that we need to support here (locked commit vs. upstream commit):
apache/arrow-rs-object-store@6259202...719a29d

Team

(I am happy to help in any way here).

Milestones

Project Organization

apache/arrow-rs-object-store#669

[kylebarron]

Thanks for opening this.

The primary task here is updating obstore while making sure we reflect every one of the upstream commits as it makes sense. Like if one of the upstream PRs added a new config, we should ensure we expose that and document it similarly.

Ideally we would go about this as:

• (maybe in a new issue? just titled something like "update obstore to latest upstream") create a task list of all upstream commits since the previous upstream commit we track. something like:
  • fix(memory): fix out-of-order multipart upload for in-memory store (fix(memory): fix out-of-order multipart upload for in-memory store apache/arrow-rs-object-store#683)
  • etc
• We'd first have to go back and figure out what's the earliest commit that's really changed since the last obstore release (the most recent obstore release has missed some earlier upstream changes)
• Then go through all those upstream commits and figure out whether we need to update code here to reflect the upstream meaning.