Ed25519 verify accepts non-canonical public key encoding

[py-thok]

TweetNaCl.js Ed25519 verification accepts non-canonical public key encodings where the encoded y-coordinate is >= p.

Tested version: tweetnacl-js 1.0.3

Code path:

• tweetnacl-js/nacl.js:328 (unpack25519) parses y and clears sign bit but does not enforce y < p
• tweetnacl-js/nacl.js:792 (unpackneg) uses unpack25519
• tweetnacl-js/nacl.js:830 (crypto_sign_open) calls unpackneg(q, pk) during verify

According to RFC 8032, Section 5.1.3, out-of-bounds public key encodings must be explicitly rejected. Accepting them can lead to signature verification inconsistencies across different implementations and may break certain security policies that rely on canonical encoding assumptions.

The following code demonstrates that an out-of-bounds public key can still be accepted:

#!/usr/bin/env node
'use strict';
const path = require('path');
const nacl = require(path.join(__dirname, '..', '..', 'tweetnacl-js', 'nacl.js'));
const P = (1n << 255n) - 19n;

function decodeY(pk) {
  const y = Buffer.from(pk);
  y[31] &= 0x7f;
  let out = 0n;
  for (let i = 31; i >= 0; i--) out = (out << 8n) + BigInt(y[i]);
  return out;
}

const pkCanonical = Buffer.from('01' + '00'.repeat(31), 'hex');
const pkNonCanonical = Buffer.from('ee' + 'ff'.repeat(30) + '7f', 'hex'); // p+1
const sig = Buffer.alloc(64);
sig[0] = 1;
const msg = Buffer.from('ed25519-canonical-y-check');

const yCanonical = decodeY(pkCanonical);
const yNonCanonical = decodeY(pkNonCanonical);
const okCanonical = nacl.sign.detached.verify(msg, sig, pkCanonical);
const okNonCanonical = nacl.sign.detached.verify(msg, sig, pkNonCanonical);

console.log('canonical_y<p         =', yCanonical < P);
console.log('noncanonical_y<p      =', yNonCanonical < P);
console.log('verify(canonical_pk)  =', okCanonical);
console.log('verify(noncanonical)  =', okNonCanonical);

if (okNonCanonical && yNonCanonical >= P) {
  console.log('\n non-canonical y (>=p) key was accepted by tweetnacl-js verify path.');
  process.exit(0);
}

console.error('\n expected non-canonical key acceptance behavior not observed.');
process.exit(1);

Execution Results:

canonical_y<p         = true
noncanonical_y<p      = false
verify(canonical_pk)  = true
verify(noncanonical)  = true

 non-canonical y (>=p) key was accepted by tweetnacl-js verify path.

[dchest]

Thanks, see discussion in #253

[py-thok]

Thanks, see discussion in #253

Thank you for your work on tweetnacl-js! I have actually seen that issue. While that thread discusses the missing S < L bounds check in the signature, the issue I am reporting is a separate problem related to the public key, which also violates RFC 8032.

[dchest]

Right, sorry, I've re-opened this to keep track of this too. #253 discusses implementing RFC 8032 in general as opposed to the original djb's Ed25519. I guess I'll create a single issue for this — #271 — to keep track and make this a subissue.