Skip to content

Commit 8ddc4c5

Browse files
StopMakingThatBigFaceStopMakingThatBigFace
authored
feat: add transport features (mTLS, DNS, multipart, zstd, charset) (#5)
* feat: add transport features (mTLS, DNS, multipart, zstd, charset) * chore: bump deps * ci: add dependency audit and upstream sync
1 parent ff90247 commit 8ddc4c5

39 files changed

Lines changed: 2683 additions & 354 deletions

‎.github/dependabot.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
version: 2
2+
updates:
3+
- package-ecosystem: npm
4+
directory: /
5+
schedule:
6+
interval: weekly
7+
day: monday
8+
time: "09:00"
9+
timezone: Europe/Moscow
10+
open-pull-requests-limit: 1
11+
versioning-strategy: auto
12+
labels:
13+
- dependencies
14+
ignore:
15+
- dependency-name: "*"
16+
update-types:
17+
- version-update:semver-major
18+
groups:
19+
npm-minor-and-patch:
20+
applies-to: version-updates
21+
patterns:
22+
- "*"
23+
update-types:
24+
- minor
25+
- patch
26+
npm-security:
27+
applies-to: security-updates
28+
patterns:
29+
- "*"

‎.github/workflows/dependency-audit.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
name: 🔐 Dependency Audit
2+
3+
on:
4+
schedule:
5+
- cron: "0 5 * * 1"
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: read
10+
11+
jobs:
12+
audit:
13+
runs-on: ubuntu-latest
14+
15+
steps:
16+
- name: Checkout repository
17+
uses: actions/checkout@v4
18+
19+
- name: Setup Node.js
20+
uses: actions/setup-node@v4
21+
with:
22+
node-version-file: package.json
23+
cache: npm
24+
25+
- name: Install dependencies
26+
run: npm ci
27+
28+
- name: Run npm audit
29+
run: npm audit --audit-level=moderate

‎.github/workflows/wreq-upstream.yml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
name: 📦 Sync wreq Upstream
2+
3+
on:
4+
schedule:
5+
- cron: "15 5 * * 1"
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: write
10+
pull-requests: write
11+
12+
concurrency:
13+
group: monitor-wreq-upstream
14+
cancel-in-progress: false
15+
16+
jobs:
17+
bump-wreq:
18+
runs-on: ubuntu-latest
19+
20+
steps:
21+
- name: Checkout repository
22+
uses: actions/checkout@v4
23+
with:
24+
fetch-depth: 0
25+
26+
- name: Setup Node.js
27+
uses: actions/setup-node@v4
28+
with:
29+
node-version-file: package.json
30+
cache: npm
31+
32+
- name: Setup Rust
33+
uses: dtolnay/rust-toolchain@stable
34+
35+
- name: Check crates.io for upstream wreq releases
36+
run: node ./scripts/update-wreq-upstream.mjs
37+
38+
- name: Detect dependency changes
39+
id: changes
40+
run: |
41+
if git diff --quiet -- rust/Cargo.toml rust/Cargo.lock; then
42+
echo "changed=false" >> "$GITHUB_OUTPUT"
43+
else
44+
echo "changed=true" >> "$GITHUB_OUTPUT"
45+
fi
46+
47+
- name: Install dependencies
48+
if: steps.changes.outputs.changed == 'true'
49+
run: npm ci
50+
51+
- name: Run test suite
52+
if: steps.changes.outputs.changed == 'true'
53+
run: npm test
54+
55+
- name: Create pull request
56+
if: steps.changes.outputs.changed == 'true'
57+
uses: peter-evans/create-pull-request@v7
58+
with:
59+
branch: chore/update-wreq-upstream
60+
delete-branch: true
61+
commit-message: "chore: bump upstream wreq crates"
62+
title: "chore: bump upstream wreq crates"
63+
body: |
64+
Automated upstream bump for the Rust crates.
65+
66+
This workflow checks `crates.io` for the latest published `max_version` of:
67+
- `wreq`
68+
- `wreq-util`
69+
labels: |
70+
dependencies
71+
add-paths: |
72+
rust/Cargo.toml
73+
rust/Cargo.lock

‎README.md‎

Lines changed: 113 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,8 @@ TLS and HTTP/2 fingerprinting is actively used by major bot protection and WAF p
4545
npm install node-wreq
4646
```
4747

48+
Node.js 20+ is required.
49+
4850
## contents
4951

5052
#### ⚡   **[quick start](#quick-start)**
@@ -57,7 +59,7 @@ npm install node-wreq
5759
#### 📊   **[observability](#observability)**
5860
#### 🚨   **[error handling](#errors)**
5961
#### 🔌   **[websockets](#websockets)**
60-
#### 🧪   **[networking / transport knobs](#networking)** — TLS, HTTP/1, HTTP/2 options; header ordering.
62+
#### 🧪   **[networking / transport knobs](#networking)** — TLS, HTTP/1, HTTP/2 options; header ordering, mTLS and custom CAs; DNS controls.
6163

6264
## ⚡ <a id="quick-start"></a>quick start
6365

@@ -137,6 +139,24 @@ const response = await fetch('https://api.example.com/items', {
137139
console.log(await response.json());
138140
```
139141

142+
### upload `FormData`
143+
144+
`FormData` request bodies work like `fetch`: the multipart boundary and `content-type` header are generated automatically.
145+
146+
```ts
147+
const formData = new FormData();
148+
149+
formData.append('alpha', '1');
150+
formData.append('upload', new File(['hello'], 'hello.txt', { type: 'text/plain' }));
151+
152+
const response = await fetch('https://api.example.com/upload', {
153+
method: 'POST',
154+
body: formData,
155+
});
156+
157+
console.log(await response.json());
158+
```
159+
140160
### build a `Request` first
141161

142162
```ts
@@ -697,10 +717,44 @@ const response = await fetch('https://httpbin.org/anything', {
697717
});
698718
```
699719

720+
If you want to bypass env/system proxy detection for a specific request, use `proxy: false`:
721+
722+
```ts
723+
await fetch('https://example.com', {
724+
proxy: false,
725+
});
726+
```
727+
700728
### disable default browser-like headers
701729

702730
By default, `node-wreq` may apply profile-appropriate default headers.
703731

732+
`disableDefaultHeaders: true` disables those browser/profile preset headers only.
733+
734+
That means it turns off headers injected by the selected browser emulation, such as:
735+
736+
- `user-agent`
737+
- `accept`
738+
- `accept-language`
739+
- `sec-ch-ua`
740+
- `sec-ch-ua-mobile`
741+
- `sec-ch-ua-platform`
742+
- `sec-fetch-dest`
743+
- `sec-fetch-mode`
744+
- `sec-fetch-site`
745+
- `priority`
746+
747+
The exact set varies by profile.
748+
749+
It does **not** disable protocol or transport-level headers that may still appear automatically, such as:
750+
751+
- `host`
752+
- `accept-encoding` when `compress` is enabled
753+
- `content-length` when the request body requires it
754+
- `content-type` generated by the runtime for bodies like `FormData`
755+
756+
It also does not remove headers you set explicitly yourself.
757+
704758
If you want full manual control:
705759

706760
```ts
@@ -713,34 +767,25 @@ await fetch('https://example.com', {
713767
});
714768
```
715769

716-
### exact header order
717-
718-
Use tuples when header order matters:
770+
For example, with `browser: 'chrome_137'`, the default request would normally include Chrome-like `sec-ch-*`, `sec-fetch-*`, `user-agent`, `accept`, and `accept-language` headers. With `disableDefaultHeaders: true`, those browser preset headers are skipped, while transport headers like `host` and `accept-encoding` may still be present.
719771

720-
```ts
721-
await fetch('https://example.com', {
722-
headers: [
723-
['x-lower', 'one'],
724-
['X-Mixed', 'two'],
725-
],
726-
});
727-
```
772+
### exact header order
728773

729-
### exact original header names on the wire
774+
Use tuples when header order matters.
730775

731-
Use this only if you really need exact casing / spelling preservation:
776+
Tuple headers also preserve the original header names exactly as you wrote them on the wire:
732777

733778
```ts
734779
await fetch('https://example.com', {
735-
disableDefaultHeaders: true,
736-
keepOriginalHeaderNames: true,
737780
headers: [
738781
['x-lower', 'one'],
739782
['X-Mixed', 'two'],
740783
],
741784
});
742785
```
743786

787+
For example, this will preserve both the tuple order and the exact `x-lower` / `X-Mixed` casing you passed.
788+
744789
### lower-level transport tuning
745790

746791
If a browser preset gets you close but not all the way there:
@@ -767,14 +812,66 @@ Use these only when:
767812
- you are comparing transport behavior
768813
- you want to debug fingerprint mismatches
769814

815+
### mTLS and custom CAs
816+
817+
Use `tlsIdentity` for client certificate authentication and `ca` for a custom trust store:
818+
819+
```ts
820+
import { fetch } from 'node-wreq';
821+
import { readFileSync } from 'node:fs';
822+
823+
await fetch('https://mtls.example.com', {
824+
tlsIdentity: {
825+
cert: readFileSync('./client-cert.pem'),
826+
key: readFileSync('./client-key.pem'),
827+
},
828+
ca: {
829+
cert: readFileSync('./ca.pem'),
830+
includeDefaultRoots: false,
831+
},
832+
});
833+
```
834+
835+
PKCS#12 / PFX identities are also supported:
836+
837+
```ts
838+
await fetch('https://mtls.example.com', {
839+
tlsIdentity: {
840+
pfx: readFileSync('./client-identity.p12'),
841+
passphrase: 'secret',
842+
},
843+
ca: {
844+
cert: readFileSync('./ca.pem'),
845+
includeDefaultRoots: false,
846+
},
847+
});
848+
```
849+
770850
### compression
771851

772852
Compression is enabled by default.
773853

854+
That includes `gzip`, `br`, `deflate`, and `zstd` response decoding when the server supports them.
855+
774856
Disable it if you need stricter control over response handling:
775857

776858
```ts
777859
await fetch('https://example.com/archive', {
778860
compress: false,
779861
});
780862
```
863+
864+
### DNS controls
865+
866+
Use `dns.hosts` to pin hostnames to specific IPs, or `dns.servers` to send lookups through specific nameservers:
867+
868+
```ts
869+
await fetch('https://api.internal.test/health', {
870+
dns: {
871+
servers: ['1.1.1.1', '8.8.8.8'],
872+
hosts: {
873+
'api.internal.test': ['127.0.0.1'],
874+
},
875+
},
876+
});
877+
```

0 commit comments

Comments
 (0)