is this hyphen behavior finalized / certain? asking as I am seeing -name --name -name- --name-- name- name-- registrations already and don’t want to enable something on ensvision which is not approved or guaranteed for being allowed in the future.
we try our best to stay in line with ENS registration policies to avoid confusion between the services and what names will be valid but people will start asking about these soon from seeing the registration bots.
I agree, some more information/clarity would be helpful as there are many people now registering domains with characters that are expected to be normalized/validated and some that I believe are not (according to the resolver).
this concerns leading and trailing hyphens, the leading underscore, the dollar sign, the ability to use the euro and pound symbols with letters, the “@” symbol, the ethereum symbol, the bitcoin symbol, and more.
If raffy could provide a list of all the invalid domains along with some information explaining some of the intricate characteristics for different characters/symbols, that would be very helpful to avoid anyone else wasting their eth on invalid domains that will stay invalid after the normalization/validation.
Not my call but I think the registrant should be made aware of the DNS incompatibilities, otherwise, I see no issue.
The only new one is $
Combining marks have been discussed a few times in this thread (#143, #347, etc.)
In Unicode 14 Normalization Tests, 61 59A 316 1DFA 316 62 is supposed to normalize to 61 1DFA 316 316 59A 62. The characters 1DFA 316 316 59A are all combining marks. However, 1DFA was added in Unicode 14. In Unicode 13, this string normalizes to 61 316 59A 1DFA 316 62 which is different.
I’m proposing that we limit combining marks to one mark per non-mark character when the name is in NFD form. In other words: no combining mark can touch another combining mark. This avoids any issue with combining mark ordering. If in the future, a previously valid character is deemed a combining mark, the name is invalid (instead of potentially having a new normalization.) The spec should define the explicit list of approved combining marks.
There are 813 names with 2+ combining marks. There is 1 name with a leading combining mark.
While we’re addressing CM, we should also enforce that each label cannot start with a combining mark (this is part of UTS-46 but wasn’t previously enforced) and potentially enforce that a combining mark cannot follow an emoji.
This change would: reduce the confusable attack surface (a large category of spoofs involve carefully placed dot-like combining marks) and greatly expand support for system-level Unicode libraries (otherwise, most normalize implementations will required a versioned Unicode implementation and cannot rely on a system-level function, like the ENSIP warns.)
However, it’s unclear if there are legitimate needs for 2+ combining marks, so I’m looking for input.
I think these are good ideas. It’ll affect a relatively small amount of names now, and we can always relax the restriction later if needed.
Another possible confusable
0000⁄0000.eth
been getting minted recently
/ is not being allowed though so maybe it will be ok?
/ [2044] is on my “I’m not sure” list. It’s used in IDNA 2003 (and 2008) mapping for fractions (like ½). So if we disallow 2044, we also need to remove 20 mappings.
As of recently, there are about 250 names with 2044.
@raffy trivial:
import {ens_beautify, ens_normalize} from "@adraffy/ens_normalize";
will not resolve ens_beautify, but:
import {ens_beautify, ens_normalize} from "@adraffy/ens_normalize/src/lib";
will. Tested on Angular 2.0 v13.4
I incorporated these changes into the ENSIP and updated the corresponding libraries.
Damage report: ens_normalize (1.6.0) vs eth-ens-namehash (2.0.15) [1463921 labels] @ 2022-09-01T09:50:38.018Z
There’s some Hebrew, Devanagari, Thai names that are collateral damage from the CM change. It’s unclear if these are legitimate names or if they’re just decorated with superfluous CM.
w/r/t Unicode NF support: Node 14 with Unicode 13 now runs the validation tests successfully. Node 12 with Unicode 11 has one 1 error (on a decomposable codepoint that was added in Unicode 13.)
To implement the Emoji + CM rule, the emoji.json now includes single-codepoint default text-presentation emoji (they used to be in chars.json and were considered characters.) Ultimately, I think this is an improvement, because they are emoji, but it potentially breaks the assumption that all emoji are safe – instead, now all emoji of 2+ codepoints are safe.
I also removed the following CMs which appear to be underscore-like:
cm-tally.json
320 (x̠) Combining Minus Sign Below
332 (x̲) Combining Low Line
333 (x̳) Combining Double Low Line
347 (x͇) Combining Equals Sign Below
FE2B (x︫) Combining Macron Left Half Below
FE2C (x︬) Combining Macron Right Half Below
FE2D (x︭) Combining Conjoining Macron Below
Also, josiahadams brought up an issue about substring matching, which requires partial normalization. I added some comments to the ENSIP about fragments and included an example which uses ens_normalize_fragment() which is nearly-equivalent to the Processing step in the ENSIP. For example, this would let you prepare the fragment [303 39 FE0F 20E3] and "aa--" (which both fail full normalization.)
ENS Rego Bot isn’t picking up on this and just showing 0.eth
Had to go through ENSvisions recent regos to get how it’s being done
Another one:
I wasn’t aware specifically of Horizontal Scan Line-# but I’m aware there are many symbols that require opinionated review.
I agree ⎺ ⎻ ⎼ ⎽ ⏤ ⎯ [23BA 23BB 23BC 23BD 23E4 23AF] need mapped to hyphen or disallowed. They should of been part of the hyphen mapping. Edit I say we (1) either disallow all 6, (2) disallow the first 4 and map the last 2 to hyphen, (3) map all to hyphen.
The policy should probably be: there’s one underscore, there’s many things that map to hyphen, or it’s disallowed.
IMO, the following nearby codepoints could be disallowed too:
⎀⎁⎂⎃ — Latin-like⌿⍀⍳⍸⍴⍷⍵⍹⍺⍶ — APL Letter Like⍡⍢⍣⍤⍥⍨⍩ — APL + Double Dot⍫⍬⍭ — APL + Tidle⍪⍮⍘⍙⍚⍛⍜ — APL + Hyphen/Underscore⎜⎟⎢⎥⎨⎪⎬⎮⎸⎹⍿ — Vertical Baraa⏜aa⏝aa⏞aa⏟aa⏠aa⏡aa – Rotated Brackets (ignore the a’s)I mapped all 6 of those characters to hyphen (option 3.) While it’s best to disallow characters so they can be revisited later (mapping should be used sparingly), I see no future for alternative hyphens and mapping is much better than the alternative.
I left the others unchanged. They’re low frequency usage at the moment JSON.
The latest version of ens_normalize.js includes NF 14.0.0 ending any Unicode issues. I was able to run 100% validation tests (unmodified) on Node 11 (which is on Unicode 11.) I also removed globalThis.atob() dependency (h/t makoto)
There’s now an ens_emoji() function which returns all of the emoji sequences.
I was incorrect here. There are some confusable emoji that eventually need to be reviewed regarding if some kind of warning is appropriate:
🚴♂🚴🏻🚴 Gender:[1F6B4 200D 2642] vs Skin:[1F6B4 1F3FB] vs Singular:[1F6B4]🇺🇸 🇺🇲 US:[1F1FA 1F1F8] vs UM:[1F1FA 1F1F2]For the flag cases, the beautifier now inserts 200B between regional indicators, which prevents them collapsing into flags. This makes US vs UM obvious and works for all similar flag-related issues.
Delta Report: ens_normalize (1.6.1) vs ens_normalize (1.6.3) [1473529 labels] @ 2022-09-09T08:57:10.840Z
U+0027 disallowed but U+2018 & U+2019 allowed
These are going to be a confusable to each other let alone ‘
Thanks @Theth.eth. Agreed, a bunch of that punctuation block should be disallowed.
Disallowed
2000 ( ) EN QUAD2001 ( ) EM QUAD2002 ( ) EN SPACE2003 ( ) EM SPACE2004 ( ) THREE-PER-EM SPACE2005 ( ) FOUR-PER-EM SPACE2006 ( ) SIX-PER-EM SPACE2007 ( ) FIGURE SPACE2008 ( ) PUNCTUATION SPACE2009 ( ) THIN SPACE200A ( ) HAIR SPACE200C () ZERO WIDTH NON-JOINER200D () ZERO WIDTH JOINER200E () LEFT-TO-RIGHT MARK200F () RIGHT-TO-LEFT MARK2017 (‗) DOUBLE LOW LINE2024 (․) ONE DOT LEADER2025 (‥) TWO DOT LEADER2026 (…) HORIZONTAL ELLIPSIS2028 ( ) LINE SEPARATOR2029 ( ) PARAGRAPH SEPARATOR202A (<U+202A>) LEFT-TO-RIGHT EMBEDDING202B (<U+202B>) RIGHT-TO-LEFT EMBEDDING202C (<U+202C>) POP DIRECTIONAL FORMATTING202D (<U+202D>) LEFT-TO-RIGHT OVERRIDE202E (?) RIGHT-TO-LEFT OVERRIDE202F ( ) NARROW NO-BREAK SPACE203C (‼) DOUBLE EXCLAMATION MARK203E (‾) OVERLINE2047 (⁇) DOUBLE QUESTION MARK2048 (⁈) QUESTION EXCLAMATION MARK2049 (⁉) EXCLAMATION QUESTION MARK205F ( ) MEDIUM MATHEMATICAL SPACE2061 () FUNCTION APPLICATION2062 () INVISIBLE TIMES2063 () INVISIBLE SEPARATOR2065 () undefined2066 (<U+2066>) LEFT-TO-RIGHT ISOLATE2067 (<U+2067>) RIGHT-TO-LEFT ISOLATE2068 (<U+2068>) FIRST STRONG ISOLATE2069 (<U+2069>) POP DIRECTIONAL ISOLATE206A () INHIBIT SYMMETRIC SWAPPING206B () ACTIVATE SYMMETRIC SWAPPING206C () INHIBIT ARABIC FORM SHAPING206D () ACTIVATE ARABIC FORM SHAPING206E () NATIONAL DIGIT SHAPES206F () NOMINAL DIGIT SHAPESIgnored
200B () ZERO WIDTH SPACE2060 () WORD JOINER2064 () INVISIBLE PLUSMapped
2010 (‐) HYPHEN → 2D (-) HYPHEN-MINUS
2011 (‑) NON-BREAKING HYPHEN → 2D (-) HYPHEN-MINUS
2012 (‒) FIGURE DASH → 2D (-) HYPHEN-MINUS
2013 (–) EN DASH → 2D (-) HYPHEN-MINUS
2014 (—) EM DASH → 2D (-) HYPHEN-MINUS
2015 (―) HORIZONTAL BAR → 2D (-) HYPHEN-MINUS
2033 (″) DOUBLE PRIME → [2032 2032]2034 (‴) TRIPLE PRIME → [2032 2032 2032]2036 (‶) REVERSED DOUBLE PRIME → [2035 2035]2037 (‷) REVERSED TRIPLE PRIME → [2035 2035 2035]2057 (⁗) QUADRUPLE PRIME → [2032 2032 2032 2032]Valid
2016 (‖) DOUBLE VERTICAL LINE2018 (‘) LEFT SINGLE QUOTATION MARK2019 (’) RIGHT SINGLE QUOTATION MARK201A (‚) SINGLE LOW-9 QUOTATION MARK201B (‛) SINGLE HIGH-REVERSED-9 QUOTATION MARK201C (“) LEFT DOUBLE QUOTATION MARK201D (”) RIGHT DOUBLE QUOTATION MARK201E („) DOUBLE LOW-9 QUOTATION MARK201F (‟) DOUBLE HIGH-REVERSED-9 QUOTATION MARK2020 (†) DAGGER2021 (‡) DOUBLE DAGGER2022 (•) BULLET2023 (‣) TRIANGULAR BULLET2027 (‧) HYPHENATION POINT2030 (‰) PER MILLE SIGN2031 (‱) PER TEN THOUSAND SIGN2032 (′) PRIME2035 (‵) REVERSED PRIME2038 (‸) CARET2039 (‹) SINGLE LEFT-POINTING ANGLE QUOTATION MARK203A (›) SINGLE RIGHT-POINTING ANGLE QUOTATION MARK203B (※) REFERENCE MARK203D (‽) INTERROBANG203F (‿) UNDERTIE2040 (⁀) CHARACTER TIE2041 (⁁) CARET INSERTION POINT2042 (⁂) ASTERISM2043 (⁃) HYPHEN BULLET → 2D (-) HYPHEN-MINUS
2044 (⁄) FRACTION SLASH2045 (⁅) LEFT SQUARE BRACKET WITH QUILL2046 (⁆) RIGHT SQUARE BRACKET WITH QUILL204A (⁊) TIRONIAN SIGN ET204B (⁋) REVERSED PILCROW SIGN204C (⁌) BLACK LEFTWARDS BULLET204D (⁍) BLACK RIGHTWARDS BULLET204E (⁎) LOW ASTERISK204F (⁏) REVERSED SEMICOLON2050 (⁐) CLOSE UP2051 (⁑) TWO ASTERISKS ALIGNED VERTICALLY2052 (⁒) COMMERCIAL MINUS SIGN2053 (⁓) SWUNG DASH2054 (⁔) INVERTED UNDERTIE2055 (⁕) FLOWER PUNCTUATION MARK2056 (⁖) THREE DOT PUNCTUATION2058 (⁘) FOUR DOT PUNCTUATION2059 (⁙) FIVE DOT PUNCTUATION205A (⁚) TWO DOT PUNCTUATION205B (⁛) FOUR DOT MARK205C (⁜) DOTTED CROSS205D (⁝) TRICOLON205E (⁞) VERTICAL FOUR DOTS⁂⁜※ seem cool, I’m not sure about •, disallow the rest?Frequencies: JSON
Edit: If we disallow ‿ and ⁔ we lose a lot of faces like ◕‿◕.
• would be a confusable for various braille letters
there is also this which could be a confusable for braille:
Would be good if there was a way to ban everything, but then select the ones you want, instead of the other way round
I tried this but you run into the same issues. Many symbols just need manually reviewed.
I updated my choices above and included a report. Most of damage are names with apostroph-like characters or leading bullets.
Are you meaning the fraction slash not to be in bold?
Would that not be a confusable with the forward slash on a keyboard which isn’t allowed
Unicode 15 officially released. I don’t know if we want to include emojis now or wait until there’s platform support (I say now that they’re official.) I’ll also check what other differences show up when I switch to the latest Unicode data files.
Platform support takes years, Windows still doesn’t support Unicode 14. I say just include them now so we don’t have to release a new version of our libraries and then get ethers/metamask and everyone else to update again etc.
2019 (’) RIGHT SINGLE QUOTATION MARK should be valid.Allows for names and surnames to be used on Ethereum NAME Service. (O’Brien, O’Donnell, Kevin O’Leary, etc