Skip to content

Commit 8db8046

Browse files
committed
Fix public suffix matching for IDN labels unassigned in IDNA2003
Problem: java.net.IDN enforces the Unicode 3.2 repertoire (IDNA2003) by default. A Public Suffix List label whose script was assigned to Unicode after 3.2 - for example Balinese (Unicode 5.0), the script behind the real PSL entry xn--9tfky.id - is "unassigned" under that repertoire. IDN.toUnicode silently fails to decode such a label instead of throwing, so DnsUtils.normalizeUnicode and PublicSuffixMatcher.getDomainRoot never see the label in the Unicode form the PSL rules are keyed by, and the matcher falls through to treating it as an ordinary registrable label one level short of the real suffix. This is a real, exploitable defect: a cookie set with Domain=xn--9tfky.id by one tenant under that suffix is accepted by another, unrelated tenant under the same suffix, defeating the Public Suffix List's whole purpose on the cookie path. The same decode is shared with DefaultHostnameVerifier (matchDNSName/matchCN normalize through the same DnsUtils.normalizeUnicode before calling PublicSuffixMatcher.verifyInternal), so a non-wildcard TLS identity for the bare suffix itself is wrongly accepted too. Solution: Pass IDN.ALLOW_UNASSIGNED to the IDN.toUnicode/toASCII calls in DnsUtils.normalizeUnicode and PublicSuffixMatcher.getDomainRoot. DnsUtils.normalizeUnicode is the fix that actually matters: it is the normalization verify() and DefaultHostnameVerifier both go through. getDomainRoot has no production caller in this module today, but is public API and should stay consistent with verify() for external callers. A separate, pre-existing defect was found while testing this: a wildcard SAN such as *.xn--9tfky.id (or, identically, a plain ASCII *.id) is accepted by verifyInternal for a host anywhere under that suffix, because the literal '*' is not stripped before the suffix lookup. It reproduces identically with or without IDN involved, so it is unrelated to this fix and intentionally left alone here; resolveDomainRoot is not touched. Tests: - TestPublicSuffixMatcher#testUnassignedCodePoint: getDomainRoot, matches and verify against a crafted PSL rule whose Unicode label is unassigned under 3.2 (xn--9tfky.test / Balinese, added to the suffixlistmatcher.txt fixture), in both ACE and Unicode form, plus the same verify() check against the real, live Mozilla Public Suffix List entry (xn--9tfky.id). - TestDefaultHostnameVerifier#testMatchDNSNameUnassignedCodePoint: matchDNSName, through the real normalize-then-verifyInternal call path (not matchIdentity directly, which does not normalize), rejects a non-wildcard certificate for the bare suffix, matching existing ASCII behavior for a control such as gov.uk. - TestRFC6265CookieSpec#testNoArgSpecsRejectUnassignedCodePointSuffix: the real, no-arg RFC6265LaxSpec/RFC6265StrictSpec (which wire the default PublicSuffixMatcher), same pattern as the existing testNoArgSpecsUsePublicSuffixMatcher - a cookie for the bare suffix does not match a different tenant under it, while a cookie for a genuine registrable domain under the suffix still matches its siblings. All three were run RED against unpatched master (production sources reverted to the upstream master blob, in an otherwise identical worktree) and GREEN with the fix, deterministically (3 repeated runs, no sockets, no timing). Full httpclient5 module: 1169/1169. httpclient5-testing is untouched by this revision (an earlier, larger version of this change added integration-server tests there; they duplicated what the CookieSpec-level test already covers and were removed). Compatibility: Observable change: a non-wildcard TLS identity, or a cookie Domain, exactly equal to a PSL suffix whose label was previously silently left undecoded is no longer treated as an ordinary registrable domain. This is a correction, not a new restriction: it brings these labels in line with how any other multi-label suffix (assigned or ASCII) already behaves. Reporter: Jonas Mandalunes (attribution as given). (cherry picked from commit 1ee8fb3) Cherry-picked to 5.6.x
1 parent 76e678a commit 8db8046

6 files changed

Lines changed: 123 additions & 3 deletions

File tree

‎httpclient5/src/main/java/org/apache/hc/client5/http/psl/PublicSuffixMatcher.java‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -141,12 +141,14 @@ public String getDomainRoot(final String domain, final DomainType expectedType)
141141
String normalized = DnsUtils.normalize(domain);
142142
final boolean punyCoded = normalized.contains("xn-");
143143
if (punyCoded) {
144-
normalized = IDN.toUnicode(normalized);
144+
// See DnsUtils.normalizeUnicode: some PSL labels are unassigned under the
145+
// Unicode 3.2 repertoire IDN enforces by default and still need to decode.
146+
normalized = IDN.toUnicode(normalized, IDN.ALLOW_UNASSIGNED);
145147
}
146148
final DomainRootInfo match = resolveDomainRoot(normalized, expectedType);
147149
String domainRoot = match != null ? match.root : null;
148150
if (domainRoot != null && punyCoded) {
149-
domainRoot = IDN.toASCII(domainRoot);
151+
domainRoot = IDN.toASCII(domainRoot, IDN.ALLOW_UNASSIGNED);
150152
}
151153
return domainRoot;
152154
}

‎httpclient5/src/main/java/org/apache/hc/client5/http/utils/DnsUtils.java‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,11 @@ public static String normalizeUnicode(final String s) {
8484
}
8585
String decoded;
8686
try {
87-
decoded = IDN.toUnicode(s);
87+
// Unassigned code points must still decode: the Public Suffix List contains
88+
// labels (for example Balinese, Unicode 5.0) that are unassigned under the
89+
// Unicode 3.2 repertoire IDN enforces by default, and are stored in the PSL
90+
// in decoded form.
91+
decoded = IDN.toUnicode(s, IDN.ALLOW_UNASSIGNED);
8892
} catch (final IllegalArgumentException ignore) {
8993
decoded = s;
9094
}

‎httpclient5/src/test/java/org/apache/hc/client5/http/impl/cookie/TestRFC6265CookieSpec.java‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,10 @@
3434
import org.apache.hc.client5.http.cookie.CommonCookieAttributeHandler;
3535
import org.apache.hc.client5.http.cookie.Cookie;
3636
import org.apache.hc.client5.http.cookie.CookieOrigin;
37+
import org.apache.hc.client5.http.cookie.CookieSpec;
3738
import org.apache.hc.client5.http.cookie.MalformedCookieException;
39+
import org.apache.hc.client5.http.psl.PublicSuffixMatcher;
40+
import org.apache.hc.client5.http.psl.PublicSuffixMatcherLoader;
3841
import org.apache.hc.core5.http.Header;
3942
import org.apache.hc.core5.http.message.BasicHeader;
4043
import org.junit.jupiter.api.Assertions;
@@ -44,6 +47,41 @@
4447

4548
class TestRFC6265CookieSpec {
4649

50+
/**
51+
* xn--9tfky.id (ᬩᬮᬶ.id) is a real Mozilla Public Suffix List entry whose
52+
* Unicode label is unassigned under the Unicode 3.2 repertoire java.net.IDN enforces by
53+
* default. Without IDN.ALLOW_UNASSIGNED the matcher never decodes it, so it falls
54+
* through to treating it as an ordinary registrable domain instead of a public suffix.
55+
* Uses the PSL-aware CookieSpec wiring exactly as HttpClientBuilder sets it up by
56+
* default (RFC6265CookieSpecFactory with PublicSuffixMatcherLoader.getDefault()).
57+
*/
58+
@Test
59+
void testRejectUnassignedCodePointSuffix() throws Exception {
60+
final CookieOrigin evilTenant = new CookieOrigin("evil-tenant.xn--9tfky.id", 80, "/", false);
61+
final CookieOrigin otherTenant = new CookieOrigin("other-tenant.xn--9tfky.id", 80, "/", false);
62+
final CookieOrigin legitOrigin = new CookieOrigin("foo.legit.xn--9tfky.id", 80, "/", false);
63+
final CookieOrigin legitSibling = new CookieOrigin("bar.legit.xn--9tfky.id", 80, "/", false);
64+
final PublicSuffixMatcher publicSuffixMatcher = PublicSuffixMatcherLoader.getDefault();
65+
final List<CookieSpec> cookieSpecs = Arrays.asList(
66+
new RFC6265CookieSpecFactory(RFC6265CookieSpecFactory.CompatibilityLevel.RELAXED, publicSuffixMatcher)
67+
.create(null),
68+
new RFC6265CookieSpecFactory(RFC6265CookieSpecFactory.CompatibilityLevel.STRICT, publicSuffixMatcher)
69+
.create(null));
70+
71+
for (final CookieSpec cookieSpec : cookieSpecs) {
72+
// A cookie for the bare public suffix itself must not reach another tenant under it.
73+
final Cookie suffixCookie = cookieSpec.parse(
74+
new BasicHeader("Set-Cookie", "name=value; Domain=xn--9tfky.id"), evilTenant).get(0);
75+
Assertions.assertFalse(cookieSpec.match(suffixCookie, otherTenant));
76+
77+
// A cookie for a genuine registrable domain under the suffix is still a valid
78+
// cookie domain and reaches siblings under it, like any ordinary domain.
79+
final Cookie legitCookie = cookieSpec.parse(
80+
new BasicHeader("Set-Cookie", "name=value; Domain=legit.xn--9tfky.id"), legitOrigin).get(0);
81+
Assertions.assertTrue(cookieSpec.match(legitCookie, legitSibling));
82+
}
83+
}
84+
4785
@Test
4886
void testParseCookieBasics() throws Exception {
4987
final CommonCookieAttributeHandler h1 = Mockito.mock(CommonCookieAttributeHandler.class);

‎httpclient5/src/test/java/org/apache/hc/client5/http/psl/TestPublicSuffixMatcher.java‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,40 @@ void testVerifyUnicode() {
203203
Assertions.assertTrue(matcher.verify("foo.xn--h-2fa.no"));
204204
}
205205

206+
/**
207+
* ᬩᬮᬶ.test (A-label xn--9tfky.test) exercises a public suffix whose
208+
* Unicode label is unassigned under Unicode 3.2, unlike hå.no above whose å
209+
* has been assigned since Unicode 1.0. {@link java.net.IDN#toUnicode(String)} without
210+
* {@link java.net.IDN#ALLOW_UNASSIGNED} silently fails to decode such a label, so the
211+
* matcher must still recognise it as the public suffix it is, in both the ACE and the
212+
* Unicode form, and a genuine subdomain under it must still be a valid cookie domain.
213+
* Confirmed against the crafted fixture and, for verify(), against the real, unmodified
214+
* Mozilla Public Suffix List too (rule ᬩᬮᬶ.id, A-label xn--9tfky.id).
215+
*/
216+
@Test
217+
void testUnassignedCodePoint() {
218+
Assertions.assertEquals("q.xn--9tfky.test", matcher.getDomainRoot("q.xn--9tfky.test"));
219+
Assertions.assertEquals("q.ᬩᬮᬶ.test", matcher.getDomainRoot("q.ᬩᬮᬶ.test"));
220+
Assertions.assertNull(matcher.getDomainRoot("xn--9tfky.test"));
221+
Assertions.assertNull(matcher.getDomainRoot("ᬩᬮᬶ.test"));
222+
223+
Assertions.assertTrue(matcher.matches(".xn--9tfky.test"));
224+
Assertions.assertTrue(matcher.matches(".ᬩᬮᬶ.test"));
225+
226+
// The bare public suffix, in either form, must never be a valid cookie domain.
227+
Assertions.assertFalse(matcher.verify("xn--9tfky.test"));
228+
Assertions.assertFalse(matcher.verify("ᬩᬮᬶ.test"));
229+
// A genuine registrable domain under it is still allowed, in either form.
230+
Assertions.assertTrue(matcher.verify("evil-tenant.xn--9tfky.test"));
231+
Assertions.assertTrue(matcher.verify("evil-tenant.ᬩᬮᬶ.test"));
232+
233+
// Same, against the real, live Mozilla Public Suffix List, not just the fixture.
234+
Assertions.assertFalse(pslMatcher.verify("xn--9tfky.id"));
235+
Assertions.assertFalse(pslMatcher.verify("ᬩᬮᬶ.id"));
236+
Assertions.assertTrue(pslMatcher.verify("evil-tenant.xn--9tfky.id"));
237+
Assertions.assertTrue(pslMatcher.verify("evil-tenant.ᬩᬮᬶ.id"));
238+
}
239+
206240
private void checkPublicSuffix(final String input, final String expected) {
207241
Assertions.assertEquals(expected, pslMatcher.getDomainRoot(input));
208242
}

‎httpclient5/src/test/java/org/apache/hc/client5/http/ssl/TestDefaultHostnameVerifier.java‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -455,6 +455,44 @@ void testMatchDNSName() throws Exception {
455455
publicSuffixMatcher);
456456
}
457457

458+
/**
459+
* matchDNSName normalizes the host and the SAN with DnsUtils.normalizeUnicode before
460+
* calling matchIdentity, which in turn calls PublicSuffixMatcher.verifyInternal directly,
461+
* bypassing PublicSuffixMatcher.verify. ᬩᬮᬶ.test (A-label xn--9tfky.test)
462+
* is a public suffix whose Unicode label is unassigned under Unicode 3.2, so this exercises
463+
* the same decode path as the cookie bug, but on the TLS hostname verification path.
464+
*/
465+
@Test
466+
void testMatchDNSNameUnassignedCodePoint() throws Exception {
467+
// A genuine subdomain under the suffix is a valid identity, in ACE or Unicode form.
468+
DefaultHostnameVerifier.matchDNSName(
469+
"evil-tenant.xn--9tfky.test",
470+
Collections.singletonList(SubjectName.DNS("evil-tenant.xn--9tfky.test")),
471+
publicSuffixMatcher);
472+
DefaultHostnameVerifier.matchDNSName(
473+
"evil-tenant.ᬩᬮᬶ.test",
474+
Collections.singletonList(SubjectName.DNS("evil-tenant.ᬩᬮᬶ.test")),
475+
publicSuffixMatcher);
476+
// A non-wildcard certificate for the bare public suffix itself must be rejected,
477+
// in ACE or Unicode form, exactly as it already is for an ASCII multi-label
478+
// suffix such as gov.uk (verified below as a control, same shape, no IDN involved).
479+
Assertions.assertThrows(SSLException.class, () ->
480+
DefaultHostnameVerifier.matchDNSName(
481+
"xn--9tfky.test",
482+
Collections.singletonList(SubjectName.DNS("xn--9tfky.test")),
483+
publicSuffixMatcher));
484+
Assertions.assertThrows(SSLException.class, () ->
485+
DefaultHostnameVerifier.matchDNSName(
486+
"ᬩᬮᬶ.test",
487+
Collections.singletonList(SubjectName.DNS("ᬩᬮᬶ.test")),
488+
publicSuffixMatcher));
489+
Assertions.assertThrows(SSLException.class, () ->
490+
DefaultHostnameVerifier.matchDNSName(
491+
"gov.uk",
492+
Collections.singletonList(SubjectName.DNS("gov.uk")),
493+
publicSuffixMatcher));
494+
}
495+
458496
@Test
459497
void testMatchIdentity() {
460498
// Test 1: IDN matching punycode

‎httpclient5/src/test/resources/suffixlistmatcher.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,4 +57,8 @@ gov.uk
5757
no
5858
hå.no
5959

60+
// unicode, Balinese script (Unicode 5.0, unassigned under Unicode 3.2 / IDNA2003)
61+
test
62+
ᬩᬮᬶ.test
63+
6064
// ===END ICANN DOMAINS===

0 commit comments

Comments
 (0)