ENS Name Normalization

It might be helpful to divide this problem into 2 separate components:

  1. Normalization: Technical details for converting a domain into a labelhash.
  2. User experience: What a UI shows the user during ENS interactions.

I consider this distinction similar to the one between UTS#46 (protocol level normalization) and Internationalized Domain Names (IDN) in Google Chrome (user experience guidelines). The normalization piece is largely fixed and immutable, while the user experience can be tweaked depending on the application and/or context.

What do you think about breaking this out into 2 separate goals?

  1. Definition of a strict and precisely defined protocol which converts a unicode string into a labelhash. The protocol should be as simple as possible, and any changes should be backwards compatible.
  2. Recommend standard usability guidelines across platforms (these guidelines can exist on a spectrum depending on the application, and would be amenable to change in the future).

Additionally, I think the “holy grail” for normalization would be an on-chain normalization implementation. Even if it’s only economical with eth_call, and even if it doesn’t completely implement UTS46/IDNA2008; it would allow for an unambiguous version of ENS normalization. If we’re taking the time to firm up normalization requirements, I think we should consider whether this is actually feasible.

From EIP137:

<domain> ::= <label> | <domain> "." <label>
<label> ::= any valid string label per [UTS46](https://unicode.org/reports/tr46/)

I interpret this to mean that . shouldn’t be part of any string that is UTS46 normalized, so UTS46 stop rules would not apply.

Careful that we don’t fall into a false comparison here. The alternative isn’t to break existing names; it’s also possible to just ignore NV8 (the status quo). In my estimation, the proposed “branched normalization” procedure carries a steep cost (almost impractically steep, IMO):

  • doubles the complexity of the ENS normalization process (emoji rules + everything else rules)
  • doubles the complexity of the emoji normalization/validation process (old + new registrations)
  • requires every client maintain a list of old registrations

With stated benefits of:

  • Normalized emoji are fully-qualified
  • ContextJ doesn’t break emoji
  • Compliance with NV8 (does this convey any practical benefit?)
  • Small handful of previously disallowed emoji are now allowed.

I think these same benefits can be achieved more surgically with a few choice additions/deletions of individual IDNA2008 rules and some additional “user display” logic. I’m not completely against your “branched normalization” approach (I am a self-described ENS/emoji enthusiast, after all :slight_smile: ), but I’d be very cautious. The devil is in the details, which I don’t think are fully fleshed out yet.

3 Likes

Agreed. I suggested a display name which I think helps address the 99% situation— you transcribe or copy/paste a name the visual appearance after validation being the same as what you typed—is a good test. ie. Normalize, Lookup, Normalize again, Compare. I also think knowing the users intention per name is valuable.

For normalization, I think the IDNA 2003 rules are too random: strict on things that should be separate and transparent the things that are obviously malicious. I only realized NV8 wasn’t being used after working through Bidi and ContextO. I blame the Unicode spec.

I also prefer only period as the label separator but UTS #46: Unicode IDNA Compatibility Processing

I think improvements that benefit the user are the best, so not being mislead by spoofed names and giving users confidence that names > addrs, should be the focus. NV8 is definitely a huge improvement for textual names. I think emoji are a manageable subset of Unicode. Deciding which emoji and non-emoji pictographs are both valid-and-unique is a one-time deal and feasible, hopefully with some community input.


This is where my library is at currently:

  • IDNATestV2: a few tests fail now that I’ve disallowed alternative stops. I’m also playing with some of the emoji and pictograph rules.

  • I implemented NFC myself because String.normalize fails some tests: NormalizationTest

  • I’ve moved the live demo to github hosting so I can iterate faster without publishing to npm: ENS Resolver Edit: fixed url

  • I think the core functions are pretty literal but it’s still a WIP: Tokenized IDNAContextJ+OUTS-46 w/BidiUTS-51 Emoji Logic

  • The Bidi, Context, IDNA, NFC, and Emoji units are all independent.

I think the following outputs are useful to look at:

2 Likes

I think it’s worth setting some objectives for any change to the normalisation function. In my mind they would go something like this:

  1. A new function must not result in previously valid labels normalising to a different normalised representation, unless it can be demonstrated that there are a negligible number of names affected, and the benefit from the change outweighs the effect on those names. When considering if impact is negligible, the number of names, whether they resolve to anything, and whether they appear to be in active use should all be considered.
  2. A new function may result in previously valid labels becoming invalid only if it can be demonstrated that the affected names are abusive or deceptive (eg, names containing non-meaningful ZWJs).
  3. Where a new function affects the normalisation of an existing name under (1), ENS should register the new normalisation and configure it as a duplicate of the previous name where possible. Where this is not possible, ENS should refund the user any registration costs, and make best efforts to make the user aware of the upcoming change.
  4. Where a new function makes previously valid labels invalid, and there are affected names that aren’t clearly abusive, ENS should refund those users their registration fees, and make best efforts to make the user aware of the upcoming change.
  5. When choosing between simplicity of the normalisation function and preserving existing registrations, preserving existing registrations should be given priority.
  6. Wherever possible, the normalised representation should visually match the most common or familiar form that users will enter or display the name in.
  7. Any normalisation function should avoid introducing visually identical inputs that resolve to different normalised forms (and thus namehashes). Wherever practical, inputs should either all normalise to the same label, or alternate representations should be made invalid.
6 Likes

I think those objectives seem reasonable. And I agree the decisions should hinge on concrete numbers and the damage caused to existing names.

1 Like

Well, I was just about to post that I had it all figured out, and then I was curious what would happen if I only gave my library UTS-51 rules and no UTS-46 rules (this should be a perfect emoji parser), and I ran into an issue with ZWJ when testing it:

💩💩 => valid
💩{200D}{200D}💩 => invalid
💩{200D}💩 => valid, but different than above
// invisible poo joiner!

// note: these are separate and already registered:
😵‍💫 = {1F635}{200D}{1F4AB}
😵💫 = {1F635}{1F4AB} 

// and even weirder, since "1" is an emoji
// this is also valid:
1{200D}1

I don’t see anyway of differentiating emoji_zwj_sequence (legal ZWJ sequences) from whats called the RGI_Emoji_ZWJ_Sequence (recommended for general interchange) without pinning the allowable sequences to a whitelisted set of permutations: https://adraffy.github.io/ens-normalize.js/build/unicode-raw/emoji-zwj-sequences.txt

Additionally, how should we treat deliberately text-styled emoji like 💩︎ = 💩{FE0E}?

// note: FE0E and FE0F are ignored

// (A) these should be the same
// (text styling without a joiner)
// each is 2 glyphs
😵︎💫 = {1F635}{FE0E}{1F4AB}
😵︎💫️ = {1F635}{FE0E}{1F4AB}{FE0F}
😵💫︎ = {1F635}{1F4AB}{FE0E}
😵︎💫︎ = {1F635}{FE0E}{1F4AB}{FE0E}

// (B) these should be the same (ZWJ sequence)
// but different from above:
// each is 1 glyph
😵‍💫 = {1F635}{200D}{1F4AB}
😵️‍💫 = {1F635}{FE0F}{200D}{1F4AB}

// but with deliberate text styling,
// the ZWJ sequence is supposed to terminate!
// (how these render varies per platform)
(1) 😵︎‍💫 = {1F635}{FE0E}{200D}{1F4AB} => technically invalid
(2) 😵︎‍💫️ = {1F635}{FE0E}{200D}{1F4AB}{FE0F} => technically invalid
// this one is okay, no joiner
(3) 😵︎💫︎ = {1F635}{FE0E}{1F4AB}{FE0F} => valid 

// depending on the Unicode implementation,
// the (1) edits as either:
(Edit1) [text-styled 1F635] + [200D] + [1F4AB] // 3 glyphs
(Edit2) [text-styled 1F635 + 200D + 1F4AB]     // 1 glyph
// (Edit1) and (Edit2) should correspond to (A) or (B) above

If we use the RGI_Emoji_ZWJ_Sequence whitelist, this problem is solved because no text-styled emoji appear in those sequences. Which would imply that text-styled emoji terminate, and are equivalent to their unstyled or emoji-styled versions: 💩︎ == 💩️{FE0F} == 💩(default style).

Edit: I believe my latest version gets everything correct w/ the exception of not knowing which emoji/characters should be enabled/disabled. It correctly handles the examples above.

Characters that should probably change:

  • Currency symbols: $, ¢, £, ¥, €, ₿, ... These are disallowed, they should be allowed.
  • Checkmarks: ❌❌︎🗙🗴🗵🗶🗷✔️✔🗸🗹 These are allowed, they should be consolidated.
  • Negative Circled (Serif vs San-serif): ❻ ➏ these are allowed, they should be equivalent.
  • Double vs Single Circled Digits: ➀ vs ⓵ these are allowed, they should be equivalent.
  • Emoji Tags: 🏴󠁧󠁢󠁥󠁮󠁧󠁿 1F3F4 E0067 E0062 E0065 E006E E0067 E007F: these are allowed, they should be ignored as they hide arbitrary data.

Reasonable but Non-RGI ZWJ Sequences:

Not sure:

  • Variations of this guy: ༼つ◕o◕༽つ, ಠ‿ಠ, └།๑益๑།┘
  • Blocks: ▁▂▃▄▅▆▇█ If these are allowed, why is _ disallowed? And what’s this? 🗕
  • Math Symbols: many seem very cool and unique, but need individually reviewed.
  • Abstract shapes

These lists are incomplete.

Edit: I’ve computed the 75 ZWJ sequences that are non-RGI but show up on emojipedia as JSON. I’ve also included them into the Emoji report.

4 Likes

I respect you! I want to resemble him.

2 Likes

What if we do limit it to the whitelisted set? We can expand the set over time without fear of breaking existing names.

I assume by ‘ignored’ you mean they should normalise to the non-tagged version?

Underscore should not be disallowed, I think. It’s theoretically disallowed in DNS, but in practice is used for a bunch of special-purpose stuff (dmarc, SPF etc).

1 Like

Yes.

After some thought, the presence of an emoji tag sequence should simply terminate the emoji parsing, rather than consuming the tag and ignoring it. The tag sequence would then be processed (and rejected) by IDNA 2008. Ignoring is bad because you need to differentiate Flag from Flag+TagSequence since Flag could combine with something else.

There are only (3) RGI tag sequences in the Unicode set (each following a black flag emoji.) Maybe they should be whitelisted, as 🏴󠁧󠁢󠁥󠁮󠁧󠁿.eth is currently owned (but vulnerable to spoofing on nonsupporting platforms as the tag sequence renders invisibly.)

So the whitelist logic would be:

SEQ = list of allowed complete sequences (3 RGI)
ZWJ = list of allowed ZWJ sequences (1349 RGI + 0-75 non-RGI)

  1. Find the longest SEQ that exactly matches the characters.
  2. If it exists, produce an emoji token and goto 1 (this handles the flag + tag sequences)
  3. Parse the characters according to UTS-51, where ZWJs can join if they form a whitelisted sequence.
  4. If an emoji was found, produce and emoji token and goto 1.
  5. Parse the character according to UTS-46 and goto 1.

With this logic, an unsupported ZWJ sequence will terminate before a ZWJ, which will then go through UTS-46 and get rejected by ContextJ. An unsupported SEQ sequence is just parsed normally by UTS-51 and UTS-46 (and likely rejected.)


I updated my library to support this logic. I also whitelisted the 3 RGI tag sequences and added some non-RGI ZWJ sequences as a test.

The auto-generated ZWJ and SEQ whitelists and my addition.

1 Like

Is this tied into why pfps don’t display right on emoji domains even when they are set with an avatar to display? The avatars don’t display I noticed

1 Like

Can you provide a specific example?

@mdt helped me find the answer in another thread, thank you! @raffy

It’s taking me longer than expected to review all of the Unicode characters.

To solve the emoji issue in the short term, I’ve built another variant of my library which I’m calling compat, which uses the current ENS rules (IDNA 2003 w/compat) but uses UTS-51 emoji parsing and my safe modifications (CheckBidi, ContextJ, ContextO, SEQ and ZWJ whitelist, enable a few additional emoji, enable underscore, disable alternative stops.)

Here is an updated report using this library. I’ve also included it into the emoji report.. The errors are almost exclusively names that are obviously malicious. There are 14 bidi errors.

This library is available on npm as @adraffy/ens-normalize. To access the compat build, use:

import {ens_normalize} from '@adraffy/ens-normalize/dist/ens-normalize-compat.js';

let norm = ens_normalize('rAffy💩️.eth'); // throws if invalid

If anyone is adventurous, please give this a test. It should be a straight upgrade relative to existing libraries.

The library is 37 KB. It includes its own implementation of NFC (16KB). Once I figure out why the standard JS implementation is wrong, I can potentially drop this from the payload. Additionally, if the community decides the CheckBidi isn’t necessary, that saves another 5KB.

I’m reluctant to adopt two changes in normalisation in a row. What are you reviewing all unicode characters for right now, and how long do you anticipate it taking?

My goal above was simply to release something that could be used today and possibly get some developer feedback. I believe the functionality of the library is correct, I just don’t what features/characters should be enabled/disabled. I didn’t want to prevent anyone from experimenting with it, and be stuck with my opinionated incomplete whitelist.

Compat is essentially the least-strict version possible. Any names it rejects are a consequence of correct emoji parsing (or the build settings above.) I figure all additional changes (that breaks registered names) requires community input.

  • The trade-offs for CheckBidi are: an increase in implementation complexity and a strict reduction in reachable names. I think this is good for the end user. One mixed-bidi use-case are a few of those Unicode creatures: ༼つ◕o◕༽つ which is likely by accident. Whereas satoshi܂ seems malicious. Additionally, since I hoist emoji out of this validation, emoji correctly mix with RTL labels.

  • The trade-offs for Context checks are: an increase in implementation complexity that both enables and disables names. I think this is good for the end-user as it lets various languages to be represented accurately (allows ZWJ where appropriate) and moves some script-specific ambiguities. This is part of UTS-46.

  • The trade-offs for Combing Marks are: an increase in implementation complexity and a strict reduction in names. I don’t think I’ve seen this error in a registered name. This is part of UTS-46.

  • The trade-offs for reducing the label separators to just period are: reduction in implementation complexity. I think this is good for the end-user as the other stops seem malicious. Also being able to split an unnormalized name into labels on just period is a nice property to have. I think you (Nick) agreed earlier in this thread. This is different than UTS-46 but matches EIP-137.

The rest is just dealing with IDNA 2008 being far stricter than 2003.

The emoji and pictographs are all in the emoji report. I’ve been working through the characters that exist in registered names but are invalid according to IDNA 2008, which usually involves going on a tangent to investigate the character and its neighbors. I’ve also been looking at existing homoglyph attack research for ideas.

My plan was to create a similar report to the emoji one, which shows various categories of Unicode under the different build versions, where ranges get condenses to keep it manageable.


An potential idea for how to allow these strange glyphs is to create a “safe” charset, that can’t co-mingle with the exotic glyphs. For example, allow ༼つ◕o◕༽つ but disallow ༼つ◕o◕༽つa because “a” would be safe. The obvious negative would be the increase in implementation complexity.

1 Like

I investigated the NormalizationTest issue a bit more. I compared my library to Python (unicodedata), Mathematica (CharacterNormalize), and various JS engines (Node, Chrome, Safari, Brave, Firefox) and realized its a total shitshow. My library and the latest version of Firefox are the only ones that pass the test. Pinning NFC to a specific Unicode spec appears to be the right choice.

I made a simple report that compares allowed characters (valid or mapped, minus emoji) between ENS0 (current) and IDNA 2008. I overlaid it with with my current-but-incomplete whitelist so (green = whitelist as-is, purple = whitelist mapped). These are the characters that need review and any input would be very helpful: IDNA: ENS0 vs 2008 Edit: I added the number of times each character shows up in a register name in brackets, eg. § [5], means 5 registered names use this character.

There also is a much larger list (the full disallowed list minus this list) that potentially contains characters that should of been enabled in the first place. However, this list too large for an HTML report. For example, underscore (_).

I added an additional 25K registered labels to the comparison reports. Also, this service displays the last 1024 registered labels with normalization applied: Recent ENS Names

2 Likes

UTS-39 and Chromium documentation discuss some good stuff, but they’re both even more restrictive than IDNA 2008.

UTS-39 references a tool which displays confusables. For example, x vs х is crazy dangerous. It shows up in a 14 registered names so far: some surrounded by the appropriate script and a few that are certainly malicious. Applying a confusable-like mapping might be a good idea but it will brick some names. Some of these are already handled by IDNA 2008.

To reduce implementation complexity, rather then relying on many public Unicode files, ENS could supply a singular table that combines UTS-51 (+SEQ/ZWJ), UTS-46, and UTS-39 which makes implementation relatively straight forward.

Edit: Here is a first attempt at a visualization of the confusables relative to IDNA 2008.

The way to read this is: “o” is a confusable category. There are 75 characters that are confusable with it. According to IDNA 2008, they correspond to 42 separate entities after normalization is applied. The largest group has 14 characters that map to o. The next largest group has 7 characters that map to ه, etc. Groups of one are just shown as a single element (without the count and black arrow) to save space. The color codes match the rest of the reports: green = valid, purple = mapped, red = disallowed.

The scary thing would be any groups that map to similar yet distinct characters. For the image above, 14 map to o (6F) and 5 map to ο (3BF). There’s very little difference between “Latin Letter O” and “Greek Omicron”.

2 Likes

I made some improvements to the Confusables. I also added a visual breakdown of the Scripts, with emoji removed. I also included the name if you hover over a character.


I am currently thinking about a Script-based approach to address homograph attacks, as my attempts to whitelist have not been very successful with so many characters. First, each script can be independently reduced to a non-confusable set. Then, each script can specify which scripts it can mix with. By reducing the problem from all characters, to just scripts that should be combined together, the script-script confusable surface is many orders of magnitude smaller.

A few of the script groupings are too sloppy. So I suggest creating a few artificial categories, like Emoji, Digits (that span all scripts), Symbols (split from Common), etc. and merge a few that get used frequently together (Han/Katakana/Hiragana). Common/Latin/Greek/Cyrillic scripts should get collapsed using an extremely aggressive version of the confusables, so there’s absolutely 0 confusables with ASCII-like characters.

Then, based on the labels registered so far, determine what kind of script-base rule permits the most names. eg “Latin|Emoji|Digits|Symbols” is a valid recipe.

I’ve computed some tallies that show what types of scripts show up in labels using the following process: map each character of a label to a script, aaαa → {Latin,Latin,Greek,Latin} then collect:

For the sorted case, you can see most labels aren’t that diverse:

Since 371K labels solely use the Latin script, I think starting with Emoji, Digits (0-9) and Latin (A-Z) and building up, using the confusable mapping to allow more characters, until all of those names are accepted is a good starting point. And then grow from there.

1 Like

I have nowhere near the expertise and knowledge that you do with this, but I have a question: would it be useful to disallow any names ending or beginning with ZWJ, plus disallowing consecutive ZWJ in name?. It seems like this would get rid of a lot of scam names or cut them way back?

So the rule would be a name can’t begin or end with a ZWJ or have more than one ZWJ within the name (I’m assuming consecutive ZWJs aren’t used for any words in other languages or in so-called ASCII art). Is this thinking in the right direction or helpful?

1 Like