On-chain ENS Domain Normalization

royalfork · June 10, 2022, 8:36pm

Way back in December, I made the following comment in our now infamous ENS Name Normalization thread:

After taking some time to look into it, I believe that an onchain implementation of “ENS Name Normalization” (as proposed by @raffy in Draft ENSIP - Standardization of ENS Name Normalization) is both technically feasible and financially economic within a mainnet contract.

As a small proof of concept, I built github.com/royalfork/ens-ascii-normalizer, which you can try out on etherscan. ENSNormalizeAscii contains the following functions:

namehash(string domain) returns (string, bytes32) returns the normalized domain, and the namehash node of the normalized domain (reverting if domain is invalid).
owner(string domain) returns (address) uses namehash to return the domain’s current ENS owner.

Screen Shot 2022-06-10 at 3.24.35 PM767×697 60.5 KB

Note: This contract only supports ASCII-only domains. If a domain contains non-ASCII characters, namehash/lookup will revert.

Moving to full unicode support:

utf8 encoding/decoding incurs additional complexity and computing cost over just ASCII domains. In preliminary tests, validation of a simple 6 letter unicode domain cost about 200000 gas (~$5), which is probably too expensive to be called by other contracts (although this number is based on unoptimized solidity code, and could be reduced).
IDNA2008 has ~1 million “unicode” rules which would all need to be supported. These rule can be compressed (run length encoding) into a set of ~2000 “rule import” eth transactions, each costing ~1,500,000 gas. This puts the total cost of IDNA2008 rule importation at around $85,000 (at 30 gwei gas).
Emoji validation requires importation of another ~3000 rules (based on saving the full RGI set on-chain, but it might be possible to do this with less). Not sure on the best data structures for this part yet, but would estimate that emoji validation data import would cost another $20,000-$50,000 in gas (I can try and firm up a better estimate if needed).
Most UTS-46 mapping would be encoded into the IDNA2008 rule set, but “1 char → n char” mappings would require separate importation. There are only a few of these, but this could add another ~$10,000 to deployment costs (also, implementing unicode 1->n mappings isn’t as simple as the ascii 1->1 mappings, some inline assembly is required to make those mappings memory efficient).
Once on-chain, the rules would be customizable, so an “owner” could allow/disallow certain characters and/or change how context-based validation works (emoji validation is implemented via context-based validation).

Not sure if ENSAsciiNormalizer is the first mainnet “string → owner” contract for ENS, but I find it nice to perform direct on-chain domain lookup/validation without the need for intermediary client-side code (it’s also nice to check for ZWJ characters directly on etherscan). If there’s any interest in a “full unicode normalization” contract (and we can collectively stomach the ~$200,000 in fees it would take us to get there), I can put together a more detailed RFP. Would be pretty cool to implement something like this, and it could vastly simplify 3rd party integrations (instead of asking etherscan, metamask, opensea, ethers.js, everyone else to “update their code” everytime ENS normalization rules are updated, we can just point them to our easy-to-use on-chain normalization function and be done with it).

nick.eth · June 11, 2022, 8:47am

Onchain normalisation would be really useful to have; it would mean that implementations can start by using it, and transition to doing it in the client library for efficiency later, if desired.

Gas efficiency for normalisation needn’t be the top concern; normally there’d be no reason to do the normalisation inside a transaction - instead, clients can call the normalisation function to get the namehash, and pass that in to the transaction.

The costs to upload the tables will be substantial, but I think that with the preprocessing and compression work @raffy has done, it’s probably viable for a much smaller sum.

royalfork · June 25, 2022, 8:50pm

I extended this into a proof-of-concept for emoji+ascii normalization, and deployed on ropsten here: https://ropsten.etherscan.io/address/0x31096216008a6bc55ba6434488d14c86bdcf4283#code.

This contract maintains a list of all canonical/allowed emoji sequences, and only allows emoji within that set. ZWJ characters are only allowed within emoji sequences, and are only allowed in those emoji sequences that specifically use them, where they must match the allowed sequence verbatim. There’s also some special handling for 0xFE0F such that only fully-qualified emoji are returned as “normalized”, but the minimally-qualified, without 0xFE0F, is used to compute the namehash/labelhash (meaning this is fully backwards-compatible with existing emoji registrations).

This PoC implementation wastes a lot of storage (with expensive “nested maps”), but even in its unoptimized state, saving a large, 5-char-long emoji sequence into the allowlist costs 140k gas (https://ropsten.etherscan.io/tx/0x9ecf9c9665f415e5c97511489efa5185ab3d82ec0c62b8f7a88fa34023462195), which is about $13 on mainnet (30gwei). Saving 3000 RGI emoji would cost worst case $40k, but most likely a lot less (@raffy has some ideas on how this can be substantially improved).

Note: For this deployment, I only added and to the emoji allowlist…all other emoji will revert if you try and namehash, because they aren’t in the allowlist.

5pence.eth · June 25, 2022, 10:16pm

@royalfork - Awesome work. Thanks for taking the time to share the concept and PoC it.

@slobo.eth - Could we get this on the agenda for this week’s Ecosystem WG meeting? It’d be great to discuss this synchronously.
We’re back to 7pm GMT on Monday this week, correct?

raffy · June 26, 2022, 8:26am

Here is an initial version of an efficient emoji parser library that should be a drop-in for your contract.

I deployed a contract using it here and populated it.

It requires less than 1 eth in gas to deploy and supports all Unicode 14 emoji. (If the upload mechanism is exposed, it’s also upgradeable, but since it’s so cheap, its probably better to seal it and redeploy for future Unicode updates.)

function read(Emoji storage self, uint24[] memory cps, uint256 pos, bool skipFE0F) returns (uint256) — will return the length of the next emoji token, if it exists. If skipFE0F is true, it will effectively ignore FE0F (but it will also return a non-zero result for a run of FE0F's without any emoji.)
function filter(Emoji storage self, uint24[] memory cps, bool skipFE0F) returns (uint24[]) — will replace emoji tokens with a single 0xFFFFFF.

If you use as a library, you’ll need to call upload() with the following 10KB payload, broken into (2) chunks:

Chunk 1

0x000026F901010001F3C301010001F3C401010001F3CA01010001F3CB01010001F3CC01010001F46E01010001F47001010001F47101010001F47301010001F47701010001F48101010001F48201010001F48601010001F48701010001F57501010001F64501010001F64601010001F64701010001F64B01010001F64D01010001F64E01010001F6A301010001F6B401010001F6B501010001F6B601010001F92601010001F93501010001F93701010001F93801010001F93901010001F93C01010001F93D01010001F93E01010001F9B801010001F9B901010001F9CD01010001F9CE01010001F9CF01010001F9D401010001F9D601010001F9D701010001F9D801010001F9D901010001F9DA01010001F9DB01010001F9DC01010001F9DD01010100200D00020200264002030200264202030101F3FB01040101F3FC01040101F3FD01040101F3FE01040101F3FF01040400200D00020000276401050500200D00060601F52502030601FA7902030001F3F301070700200D0008080026A702030801F30802030001F3F401090900200D000A0A0026200203090E0067000B0B0E0062000C0C0E0065000D0D0E006E000E0E0E0067000F0F0E007F06100C0E00730011110E00630012120E0074000F0C0E00770013130E006C0014140E0073000F0001F40801151500200D001616002B1B02030001F41501171700200D00181801F9BA02030001F43B01191900200D001A1A00274402030001F441011B1B00200D001C1C01F5E802030001F468011D1D00200D001E1E00269502031E00269602031E00270802031E01F33E02031E01F37302031E01F37C02031E01F39302031E01F3A402031E01F3A802031E01F3EB02031E01F3ED02031E01F4BB02031E01F4BC02031E01F52702031E01F52C02031E01F68002031E01F69202031E01F9AF02031E01F9B002031E01F9B102031E01F9B202031E01F9B302031E01F9BC02031E01F9BD02031E01F466021F1F00200D00202001F46602031E01F46702212100200D00222201F46602032201F46702031E00276400232300200D00242401F46804252401F48B00262600200D00272701F46806101E01F46800281E01F46900282800200D00292901F466042A2A00200D00202901F467042B2B00200D00221D01F3FB412C1D01F3FC412C1D01F3FD412C1D01F3FE412C1D01F3FF412C2C00200D002D2D00269502032D00269602032D00270802032D01F33E02032D01F37302032D01F37C02032D01F39302032D01F3A402032D01F3A802032D01F3EB02032D01F3ED02032D01F4BB02032D01F4BC02032D01F52702032D01F52C02032D01F68002032D01F69202032D01F9AF02032D01F9B002032D01F9B102032D01F9B202032D01F9B302032D01F9BC02032D01F9BD02032D01F91D002E2E00200D002F2F01F46800303001F3FB85313001F3FC85313001F3FD85313001F3FE85313001F3FF85312D00276400323200200D00333301F46800343401F3FB05313401F3FC05313401F3FD05313401F3FE05313401F3FF05313301F48B00353500200D00363601F46800373701F3FB07383701F3FC07383701F3FD07383701F3FE07383701F3FF07380001F46901393900200D003A3A00269502033A00269602033A00270802033A01F33E02033A01F37302033A01F37C02033A01F39302033A01F3A402033A01F3A802033A01F3EB02033A01F3ED02033A01F4BB02033A01F4BC02033A01F52702033A01F52C02033A01F68002033A01F69202033A01F9AF02033A01F9B002033A01F9B102033A01F9B202033A01F9B302033A01F9BC02033A01F9BD02033A01F466021F3A01F46702213A002764003B3B00200D003C3C01F46804253C01F46904253C01F48B003D3D00200D003E3E01F46806103E01F46906103A01F46900283901F3FB413F3901F3FC413F3901F3FD413F3901F3FE413F3901F3FF413F3F00200D00404000269502034000269602034000270802034001F33E02034001F37302034001F37C02034001F39302034001F3A402034001F3A802034001F3EB02034001F3ED02034001F4BB02034001F4BC02034001F52702034001F52C02034001F68002034001F69202034001F9AF02034001F9B002034001F9B102034001F9B202034001F9B302034001F9BC02034001F9BD02034001F91D00414100200D00424201F46800304201F46900304000276400434300200D00444401F46800344401F46900344401F48B00454500200D00464601F46800374601F46900370001F46F01474700200D00024701F3FB01484701F3FC01484701F3FD01484701F3FE01484701F3FF01480001F62E01494900200D004A4A01F4A802030001F635014B4B00200D004C4C01F4AB02030001F636014D4D00200D004E4E01F32B02030001F9D1014F4F00200D00505000269502035000269602035000270802035001F33E02035001F37302035001F37C02035001F38402035001F39302035001F3A402035001F3A802035001F3EB02035001F3ED02035001F4BB02035001F4BC02035001F52702035001F52C02035001F68002035001F69202035001F9AF02035001F9B002035001F9B102035001F9B202035001F9B302035001F9BC02035001F9BD02035001F91D00515100200D00525201F9D104254F01F3FB41534F01F3FC41534F01F3FD41534F01F3FE41534F01F3FF41535300200D00545400269502035400269602035400270802035401F33E02035401F37302035401F37C02035401F38402035401F39302035401F3A402035401F3A802035401F3EB02035401F3ED02035401F4BB02035401F4BC02035401F52702035401F52C02035401F68002035401F69202035401F9AF02035401F9B002035401F9B102035401F9B202035401F9B302035401F9BC02035401F9BD02035400276400555500200D00565601F9D100305601F48B00575700200D00585801F9D100595901F3FB87385901F3FC87385901F3FD87385901F3FE87385901F3FF87385401F91D005A5A00200D005B5B01F9D100340001F9DE01040001F9DF01040001FAF1015C5C01F3FB415D5C01F3FC415D5C01F3FD415D5C01F3FE415D5C01F3FF415D5D00200D005E5E01FAF2005F5F01F3FB83605F01F3FC83605F01F3FD83605F01F3FE83605F01F3FF8360000000A90148000000AE01480000203C0148000020490148000021940148000021950148000021960148000021970148000021980148000021990148000021A90148000021AA01480000231A01480000231B0148000023280148000023CF0148000023E90148000023EA0148000023EB0148000023EC0148000023ED0148000023EE0148000023EF0148000023F00148000023F10148000023F20148000023F30148000023F80148000023F90148000023FA0148000025AA0148000025AB0148000025B60148000025C00148000025FB0148000025FC0148000025FD0148000025FE01480000260001480000260101480000260201480000260301480000260401480000260E01480000261101480000261401480000261501480000261801480000262001480000262201480000262301480000262601480000262A01480000262E01480000262F01480000263801480000263901480000263A01480000264001480000264201480000264801480000264901480000264A01480000264B01480000264C01480000264D01480000264E01480000264F01480000265001480000265101480000265201480000265301480000265F01480000266001480000266301480000266501480000266601480000266801480000267B01480000267E01480000267F01480000269201480000269301480000269401480000269501480000269601480000269701480000269901480000269B01480000269C0148000026A00148000026A10148000026A70148000026AA0148000026AB0148000026B00148000026B10148000026BD0148000026BE0148000026C40148000026C50148000026C80148000026CE0148000026CF0148000026D10148000026D30148000026D40148000026E90148000026EA0148000026F00148000026F10148000026F20148000026F30148000026F40148000026F50148000026F70148000026F80148000026FA0148000026FD01480000270201480000270501480000270801480000270901480000270F01480000271201480000271401480000271601480000271D01480000272101480000272801480000273301480000273401480000274401480000274701480000274C01480000274E0148000027530148000027540148000027550148000027570148000027630148000027950148000027960148000027970148000027A10148000027B00148000027BF014800002934014800002935014800002B05014800002B06014800002B07014800002B1B014800002B1C014800002B50014800002B5501480000303001480000303D01480001F00401480001F0CF01480001F17001480001F17101480001F17E01480001F17F01480001F18E01480001F19101480001F19201480001F19301480001F19401480001F19501480001F19601480001F19701480001F19801480001F19901480001F19A01480001F1E601480001F1E701480001F1E801480001F1E901480001F1EA01480001F1EB01480001F1EC01480001F1ED01480001F1EE01480001F1EF01480001F1F001480001F1F101480001F1F201480001F1F301480001F1F401480001F1F501480001F1F601480001F1F701480001F1F801480001F1F901480001F1FA01480001F1FB01480001F1FC01480001F1FD01480001F1FE01480001F1FF01480001F30001480001F30101480001F30201480001F30301480001F30401480001F30501480001F30601480001F30701480001F30801480001F30901480001F30A01480001F30B01480001F30C01480001F30D01480001F30E01480001F30F01480001F31001480001F31101480001F31201480001F31301480001F31401480001F31501480001F31601480001F31701480001F31801480001F31901480001F31A01480001F31B01480001F31C01480001F31D01480001F31E01480001F31F01480001F32001480001F32101480001F32401480001F32501480001F32601480001F32701480001F32801480001F32901480001F32A01480001F32B01480001F32C01480001F32D01480001F32E01480001F32F01480001F33001480001F33101480001F33201480001F33301480001F33401480001F33501480001F33601480001F33701480001F33801480001F33901480001F33A01480001F33B01480001F33C01480001F33D01480001F33E01480001F33F01480001F34001480001F34101480001F34201480001F34301480001F34401480001F34501480001F34601480001F34701480001F34801480001F34901480001F34A01480001F34B01480001F34C01480001F34D01480001F34E01480001F34F01480001F35001480001F35101480001F35201480001F35301480001F35401480001F35501480001F35601480001F35701480001F35801480001F35901480001F35A01480001F35B01480001F35C01480001F35D01480001F35E01480001F35F01480001F36001480001F36101480001F36201480001F36301480001F36401480001F36501480001F36601480001F36701480001F36801480001F36901480001F36A01480001F36B01480001F36C01480001F36D01480001F36E01480001F36F01480001F37001480001F37101480001F37201480001F37301480001F37401480001F37501480001F37601480001F37701480001F37801480001F37901480001F37A01480001F37B01480001F37C01480001F37D01480001F37E01480001F37F01480001F38001480001F38101480001F38201480001F38301480001F38401480001F38601480001F38701480001F38801480001F38901480001F38A01480001F38B01480001F38C01480001F38D01480001F38E01480001F38F01480001F39001480001F39101480001F39201480001F39301480001F39601480001F39701480001F39901480001F39A01480001F39B01480001F39E01480001F39F01480001F3A001480001F3A101480001F3A201480001F3A301480001F3A401480001F3A501480001F3A601480001F3A701480001F3A801480001F3A901480001F3AA01480001F3AB01480001F3AC01480001F3AD01480001F3AE01480001F3AF01480001F3B001480001F3B101480001F3B201480001F3B301480001F3B401480001F3B501480001F3B601480001F3B701480001F3B801480001F3B901480001F3BA01480001F3BB01480001F3BC01480001F3BD01480001F3BE01480001F3BF01480001F3C001480001F3C101480001F3C501480001F3C601480001F3C801480001F3C901480001F3CD01480001F3CE01480001F3CF01480001F3D001480001F3D101480001F3D201480001F3D301480001F3D401480001F3D501480001F3D601480001F3D701480001F3D801480001F3D901480001F3DA01480001F3DB01480001F3DC01480001F3DD01480001F3DE01480001F3DF01480001F3E001480001F3E101480001F3E201480001F3E301480001F3E401480001F3E501480001F3E601480001F3E701480001F3E801480001F3E901480001F3EA01480001F3EB01480001F3EC01480001F3ED01480001F3EE01480001F3EF01480001F3F001480001F3F501480001F3F701480001F3F801480001F3F901480001F3FA01480001F3FB01480001F3FC01480001F3FD01480001F3FE01480001F3FF01480001F40001480001F40101480001F40201480001F40301480001F40401480001F40501480001F40601480001F40701480001F40901480001F40A01480001F40B01480001F40C01480001F40D01480001F40E01480001F40F01480001F41001480001F41101480001F41201480001F41301480001F41401480001F41601480001F41701480001F41801480001F41901480001F41A01480001F41B01480001F41C01480001F41D01480001F41E01480001F41F01480001F42001480001F42101480001F42201480001F42301480001F42401480001F42501480001F42601480001F42701480001F42801480001F42901480001F42A01480001F42B01480001F42C01480001F42D01480001F42E01480001F42F01480001F43001480001F43101480001F43201480001F43301480001F43401480001F43501480001F43601480001F43701480001F43801480001F43901480001F43A01480001F43C01480001F43D01480001F43E01480001F43F01480001F44001480001F44401480001F44501480001F45101480001F45201480001F45301480001F45401480001F45501480001F45601480001F45701480001F45801480001F45901480001F45A01480001F45B01480001F45C01480001F45D01480001F45E01480001F45F01480001F46001480001F46101480001F46201480001F46301480001F46401480001F46501480001F47901480001F47A01480001F47B01480001F47D01480001F47E01480001F47F01480001F48001480001F48401480001F48801480001F48901480001F48A01480001F48B01480001F48C01480001F48D01480001F48E01480001F49001480001F49201480001F49301480001F49401480001F49501480001F49601480001F49701480001F49801480001F49901480001F49A01480001F49B01480001F49C01480001F49D01480001F49E01480001F49F01480001F4A001480001F4A101480001F4A201480001F4A301480001F4A401480001F4A501480001F4A601480001F4A701480001F4A801480001F4A901480001F4AB01480001F4AC01480001F4AD01480001F4AE01480001F4AF01480001F4B001480001F4B101480001F4B201480001F4B301480001F4B401480001F4B501480001F4B601480001F4B701480001F4B801480001F4B901480001F4BA01480001F4BB01480001F4BC01480001F4BD01480001F4BE01480001F4BF01480001F4C001480001F4C101480001F4C201480001F4C301480001F4C401480001F4C501480001F4C601480001F4C701480001F4C801480001F4C901480001F4CA01480001F4CB01480001F4CC01480001F4CD01480001F4CE01480001F4CF01480001F4D001480001F4D101480001F4D201480001F4D301480001F4D401480001F4D501480001F4D601480001F4D701480001F4D801480001F4D901480001F4DA01480001F4DB01480001F4DC01480001F4DD01480001F4DE01480001F4DF01480001F4E001480001F4E101480001F4E201480001F4E301480001F4E401480001F4E501480001F4E601480001F4E70148

Chunk 2

0x0001F4E801480001F4E901480001F4EA01480001F4EB01480001F4EC01480001F4ED01480001F4EE01480001F4EF01480001F4F001480001F4F101480001F4F201480001F4F301480001F4F401480001F4F501480001F4F601480001F4F701480001F4F801480001F4F901480001F4FA01480001F4FB01480001F4FC01480001F4FD01480001F4FF01480001F50001480001F50101480001F50201480001F50301480001F50401480001F50501480001F50601480001F50701480001F50801480001F50901480001F50A01480001F50B01480001F50C01480001F50D01480001F50E01480001F50F01480001F51001480001F51101480001F51201480001F51301480001F51401480001F51501480001F51601480001F51701480001F51801480001F51901480001F51A01480001F51B01480001F51C01480001F51D01480001F51E01480001F51F01480001F52001480001F52101480001F52201480001F52301480001F52401480001F52501480001F52601480001F52701480001F52801480001F52901480001F52A01480001F52B01480001F52C01480001F52D01480001F52E01480001F52F01480001F53001480001F53101480001F53201480001F53301480001F53401480001F53501480001F53601480001F53701480001F53801480001F53901480001F53A01480001F53B01480001F53C01480001F53D01480001F54901480001F54A01480001F54B01480001F54C01480001F54D01480001F54E01480001F55001480001F55101480001F55201480001F55301480001F55401480001F55501480001F55601480001F55701480001F55801480001F55901480001F55A01480001F55B01480001F55C01480001F55D01480001F55E01480001F55F01480001F56001480001F56101480001F56201480001F56301480001F56401480001F56501480001F56601480001F56701480001F56F01480001F57001480001F57301480001F57601480001F57701480001F57801480001F57901480001F58701480001F58A01480001F58B01480001F58C01480001F58D01480001F5A401480001F5A501480001F5A801480001F5B101480001F5B201480001F5BC01480001F5C201480001F5C301480001F5C401480001F5D101480001F5D201480001F5D301480001F5DC01480001F5DD01480001F5DE01480001F5E101480001F5E301480001F5E801480001F5EF01480001F5F301480001F5FA01480001F5FB01480001F5FC01480001F5FD01480001F5FE01480001F5FF01480001F60001480001F60101480001F60201480001F60301480001F60401480001F60501480001F60601480001F60701480001F60801480001F60901480001F60A01480001F60B01480001F60C01480001F60D01480001F60E01480001F60F01480001F61001480001F61101480001F61201480001F61301480001F61401480001F61501480001F61601480001F61701480001F61801480001F61901480001F61A01480001F61B01480001F61C01480001F61D01480001F61E01480001F61F01480001F62001480001F62101480001F62201480001F62301480001F62401480001F62501480001F62601480001F62701480001F62801480001F62901480001F62A01480001F62B01480001F62C01480001F62D01480001F62F01480001F63001480001F63101480001F63201480001F63301480001F63401480001F63701480001F63801480001F63901480001F63A01480001F63B01480001F63C01480001F63D01480001F63E01480001F63F01480001F64001480001F64101480001F64201480001F64301480001F64401480001F64801480001F64901480001F64A01480001F68001480001F68101480001F68201480001F68301480001F68401480001F68501480001F68601480001F68701480001F68801480001F68901480001F68A01480001F68B01480001F68C01480001F68D01480001F68E01480001F68F01480001F69001480001F69101480001F69201480001F69301480001F69401480001F69501480001F69601480001F69701480001F69801480001F69901480001F69A01480001F69B01480001F69C01480001F69D01480001F69E01480001F69F01480001F6A001480001F6A101480001F6A201480001F6A401480001F6A501480001F6A601480001F6A701480001F6A801480001F6A901480001F6AA01480001F6AB01480001F6AC01480001F6AD01480001F6AE01480001F6AF01480001F6B001480001F6B101480001F6B201480001F6B301480001F6B701480001F6B801480001F6B901480001F6BA01480001F6BB01480001F6BC01480001F6BD01480001F6BE01480001F6BF01480001F6C101480001F6C201480001F6C301480001F6C401480001F6C501480001F6CB01480001F6CD01480001F6CE01480001F6CF01480001F6D001480001F6D101480001F6D201480001F6D501480001F6D601480001F6D701480001F6DD01480001F6DE01480001F6DF01480001F6E001480001F6E101480001F6E201480001F6E301480001F6E401480001F6E501480001F6E901480001F6EB01480001F6EC01480001F6F001480001F6F301480001F6F401480001F6F501480001F6F601480001F6F701480001F6F801480001F6F901480001F6FA01480001F6FB01480001F6FC01480001F7E001480001F7E101480001F7E201480001F7E301480001F7E401480001F7E501480001F7E601480001F7E701480001F7E801480001F7E901480001F7EA01480001F7EB01480001F7F001480001F90D01480001F90E01480001F91001480001F91101480001F91201480001F91301480001F91401480001F91501480001F91601480001F91701480001F92001480001F92101480001F92201480001F92301480001F92401480001F92501480001F92701480001F92801480001F92901480001F92A01480001F92B01480001F92C01480001F92D01480001F92E01480001F92F01480001F93A01480001F93F01480001F94001480001F94101480001F94201480001F94301480001F94401480001F94501480001F94701480001F94801480001F94901480001F94A01480001F94B01480001F94C01480001F94D01480001F94E01480001F94F01480001F95001480001F95101480001F95201480001F95301480001F95401480001F95501480001F95601480001F95701480001F95801480001F95901480001F95A01480001F95B01480001F95C01480001F95D01480001F95E01480001F95F01480001F96001480001F96101480001F96201480001F96301480001F96401480001F96501480001F96601480001F96701480001F96801480001F96901480001F96A01480001F96B01480001F96C01480001F96D01480001F96E01480001F96F01480001F97001480001F97101480001F97201480001F97301480001F97401480001F97501480001F97601480001F97801480001F97901480001F97A01480001F97B01480001F97C01480001F97D01480001F97E01480001F97F01480001F98001480001F98101480001F98201480001F98301480001F98401480001F98501480001F98601480001F98701480001F98801480001F98901480001F98A01480001F98B01480001F98C01480001F98D01480001F98E01480001F98F01480001F99001480001F99101480001F99201480001F99301480001F99401480001F99501480001F99601480001F99701480001F99801480001F99901480001F99A01480001F99B01480001F99C01480001F99D01480001F99E01480001F99F01480001F9A001480001F9A101480001F9A201480001F9A301480001F9A401480001F9A501480001F9A601480001F9A701480001F9A801480001F9A901480001F9AA01480001F9AB01480001F9AC01480001F9AD01480001F9AE01480001F9AF01480001F9B001480001F9B101480001F9B201480001F9B301480001F9B401480001F9B701480001F9BA01480001F9BC01480001F9BD01480001F9BE01480001F9BF01480001F9C001480001F9C101480001F9C201480001F9C301480001F9C401480001F9C501480001F9C601480001F9C701480001F9C801480001F9C901480001F9CA01480001F9CB01480001F9CC01480001F9D001480001F9E001480001F9E101480001F9E201480001F9E301480001F9E401480001F9E501480001F9E601480001F9E701480001F9E801480001F9E901480001F9EA01480001F9EB01480001F9EC01480001F9ED01480001F9EE01480001F9EF01480001F9F001480001F9F101480001F9F201480001F9F301480001F9F401480001F9F501480001F9F601480001F9F701480001F9F801480001F9F901480001F9FA01480001F9FB01480001F9FC01480001F9FD01480001F9FE01480001F9FF01480001FA7001480001FA7101480001FA7201480001FA7301480001FA7401480001FA7801480001FA7901480001FA7A01480001FA7B01480001FA7C01480001FA8001480001FA8101480001FA8201480001FA8301480001FA8401480001FA8501480001FA8601480001FA9001480001FA9101480001FA9201480001FA9301480001FA9401480001FA9501480001FA9601480001FA9701480001FA9801480001FA9901480001FA9A01480001FA9B01480001FA9C01480001FA9D01480001FA9E01480001FA9F01480001FAA001480001FAA101480001FAA201480001FAA301480001FAA401480001FAA501480001FAA601480001FAA701480001FAA801480001FAA901480001FAAA01480001FAAB01480001FAAC01480001FAB001480001FAB101480001FAB201480001FAB301480001FAB401480001FAB501480001FAB601480001FAB701480001FAB801480001FAB901480001FABA01480001FAC001480001FAC101480001FAC201480001FAD001480001FAD101480001FAD201480001FAD301480001FAD401480001FAD501480001FAD601480001FAD701480001FAD801480001FAD901480001FAE001480001FAE101480001FAE201480001FAE301480001FAE401480001FAE501480001FAE601480001FAE701480000261D01610000270A01610000270B01610000270C01610000270D01610001F38501610001F3C201610001F3C701610001F44201610001F44301610001F44601610001F44701610001F44801610001F44901610001F44A01610001F44B01610001F44C01610001F44D01610001F44E01610001F44F01610001F45001610001F46601610001F46701610001F46A01610001F46B01610001F46C01610001F46D01610001F47201610001F47401610001F47501610001F47601610001F47801610001F47C01610001F48301610001F48501610001F48F01610001F49101610001F4AA01610001F57401610001F57A01610001F59001610001F59501610001F59601610001F64C01610001F64F01610001F6C001610001F6CC01610001F90C01610001F90F01610001F91801610001F91901610001F91A01610001F91B01610001F91C01610001F91D01610001F91E01610001F91F01610001F93001610001F93101610001F93201610001F93301610001F93401610001F93601610001F97701610001F9B501610001F9B601610001F9BB01610001F9D201610001F9D301610001F9D501610001FAC301610001FAC401610001FAC501610001FAF001610001FAF201610001FAF301610001FAF401610001FAF501610001FAF601616101F3FB01486101F3FC01486101F3FD01486101F3FE01486101F3FF01480000002300620000002A0062000000300062000000310062000000320062000000330062000000340062000000350062000000360062000000370062000000380062000000390062620020E30203

The chunks are 6 byte rules for state transitions which follow this tree:

Here is a test contract that I’ve set the above contract as the emoji parser and a BasicValidator contract that implements /^[a-z09-_.]+/$ as the validator.

filter(string name, bool ignoreFEOF) should parse any UTF8 string with emojis.
validate(string name) should validate any normalized name which uses valid emoji or obeys the regex above. Note: This will fail if emoji have FE0F (which isn’t normalized). This should validate 94% of registered names as-of today.

nick.eth · June 28, 2022, 9:14am

Outstanding! Do you have an estimate of what it would take to extend this to match your entire normalisation function, not just Emoji?

raffy · June 28, 2022, 10:42am

It depends on how IDNA and NFC are implemented. If you implement them verbatim, it will be very expensive to deploy like royalfork estimated. However, there’s probably a partial mapping that would give a lot of bang for the buck.

I think there is a possibility of using NFKD → Apply the small number of differences from IDNA 2003 → NFC, but I’d need to verify that this is the same thing as IDNA 2003 → NFD → NFC. You could avoid a whole lookup table with this approach.

For IDNA, we can pack 10 uint24 per storage slot for single char mappings. There are runs of “ABCabc” and “AaBbCc” for casefolding, and runs of “ABC”, “A?B?C?”, and “A??B??C??” that map to a constant. There’s a bunch of ideas in my github.

If, for the moment, we reject combining marks and composite characters (so NFC isn’t needed) then my emoji parser + an efficient IDNA mapping would be a string → string implementation. All the effort would go into efficiently compressing the mapping.

Otherwise, I think the most important contract is an NFD/NFC implementation.

For validation:

You can naively encode the valid output character set into a bitmap for 700 storage slots. Combine that with the emoji contract above + NFC quick check and that’s a gas efficient function for determining if a name is in the normalized form. I can demo this immediately once I have an NFC implementation.

I also think a few separate simple validators for each script would give a lot of coverage (and more could be added over time). However, this approach requires a mechanism for looping through the set of validators to find one which returns true for your name.

I mentioned in the other thread, you could store the address of validator on-chain for the name you verified, making all future validations only two SLOADs, node → validator address → is valid validator address, but this requires extra infrastructure.

nick.eth · June 28, 2022, 12:43pm

This seems to be a reasonable starting point. Are there existing domains that use combining marks or composite characters?

raffy · June 29, 2022, 12:13pm

5K names have combined characters (NFC != NFD.)
171 names have two combining marks that could be reordered.

Codepoints: 2.4K CM and 13K composed.

Maybe that is a good idea, just compress the IDNA part and disallow the codepoints that need NFC. That’s 99% of names unless I’m forgetting something.

@adraffy/ens-norm-research has an improved contract that only needs 360 storage slots and no longer requires the skipFE0F option, and the code that generates the state payload.

Emoji + IDNA 2003 - (Combining Marks, Compositions) can be done with 10KB contract code, 128 slots for Valid bitmap, 450 Slots for Mappings of [1-2 CP], 175 Slots for Mappings [3+ CP], 360 slots for Emoji.

[128 + 452 + 175 + 360] → 1115 slots → ~30M + contract gas so under 1 eth to deploy.

122810 valid, 4208 mapped, 9 ignored, 3046 emoji sequences.

Since the IDNA table is stable, a huge chunk of the valid and single character mappings can be compiled directly to code.

I think worse-case per character would be, ~15 comparisons (x < A), 10 range comparisions (x >= A && x <= B), 1 sload for valid, 1 sload for small mapping, 1 sload for big mapping, and then calling the emoji function which is (1 sload per codepoint.)

Edit: Here is my first working version: https://rinkeby.etherscan.io/address/0xdb68eb6f0ab93bc7059f2752d477b45f04457eb2#readContract

Cost 32M gas to deploy and populate.

Should work for every name except those with combining marks or characters that compose (like à).

Edit 2: I deployed a more gas efficient version that avoids using uint24[]. It also has a beautify() function.

I’m seeing about 50-100K gas per name.

Neal · July 3, 2022, 2:28am

Although I have little idea what you’re on about half the time here, your commitment to see this all the way through is admirable, @raffy

raffy · July 6, 2022, 8:50am

I got a version with NFC Quick Check, which means only disordered combining marks or decomposed characters will incorrectly fail to normalize.

https://rinkeby.etherscan.io/address/0x335be342669ae015d7a87eb7c632447f9218254b
43M gas

🤼🏻‍♂️Ⅷ👩🏽‍⚕️_$🇦ß.۱۲۳👨‍👩‍👦9️⃣.ѐ🏴󠁧󠁢󠁥󠁮󠁧󠁿.eth

nick.eth · July 6, 2022, 3:15pm

How will they be handled? Do they throw an error?

Roughly how much gas does normalize cost?

raffy · July 6, 2022, 6:52pm

NFC Quick Check failures throw NotNFC().
Disallowed throw InvalidCodepoint(uint256 cp).

🤼🏻‍♂️Ⅷ👩🏽‍⚕️_$🇦ß.۱۲۳👨‍👩‍👦9️⃣.ѐ🏴󠁧󠁢󠁥󠁮󠁧󠁿.eth → 180K gas
💩💩💩.eth → 50K gas
RAFFY.ETH → 60K gas
maybe-its-time-to-stop-registering-garbage-like-this-and-buy-premiums-from-secondary-instead-such-as-3-or-4-digits-or-clean-one-word-domains-like-soy.eth → 280K gas

~~An ASCII fastpath in-front of the emoji checker would probably save a lot of gas.~~ ~~Only saves 1 sload.~~

Edit: Actually, once both emoji and valid are known, a fast path bitmap can be stored in a single uint256, that returns true for any valid character that isn’t an emoji prefix. At the moment, that’s true for 81 of 0-255. That would make DNS names much cheaper if we need this kind of optimization.

raffy.eth → 29K gas
maybe-its-time-to-stop-registering-garbage-like-this-and-buy-premiums-from-secondary-instead-such-as-3-or-4-digits-or-clean-one-word-domains-like-soy.eth → 111K gas (save 170K)

Another question is whether the contract ultimately should be adjustable. I’m not sure yet the best way to encode the large tree of codepoint ranges → codepoints (I’m using a ~700 line function.)

nick.eth · July 7, 2022, 8:46am

This sounds reasonable to me. Is there a reason we would need to implement the full NFC, in this context? Rejecting invalid normalisations seems sufficient to me.

Not bad! We shouldn’t generally have to do these in a transaction, but it’s nice to know it’d be viable if we really had to. Do you think there’s much room for gas optimisation here?

I’d be in favor of immutable contracts that can be replaced as needed; this tends to save gas anyway, and makes verificaiton much easier.

raffy · July 8, 2022, 8:21am

We should have full NFC for when someone inputs decomposed characters.

Now that QC works, I think I’ll just implement the algorithm as-is in my next release, since it will be invoked rarely.

Good to know.

nick.eth · July 8, 2022, 9:00am

Can we not simply reject such names as invalid?

royalfork · July 29, 2022, 10:34pm

We’ll have to define “cheap”, but I believe Unicode adds new characters once a year-ish, so if each deployment is >$1k (or operationally complex with multiple transactions), I’d think that easy upgradeability should be a design requirement (ie: new code points can be set to “valid” by a single transaction).

I’d say yes, for the same reason that you’d also reject labels which contain minimally-qualified emoji.

What do we think about the following top-level API?

/**
 * @notice Compute namehash of domain after ENS normalization and
 *         validation.  Reverts if domain contains disallowed 
 *         codepoints, disordered combining marks, or decomposed characters.
 * @param domain Domain to namehash.
 * @return normalized Normalized domain, to be displayed to end-users.
 * @return node Namehash of domain.
 * @return charsets Bit array of unicode character sets present in the domain.
 */
function namehash(string memory domain) public view returns (string memory normalized, bytes32 node, uint256 charsets) {

// Unicode charsets
var (
	ASCII = 1 << iota
	Latin-1
	Greek
	Cryllic
	Emoji
	...
)

This includes a charsets return value, which tells the caller which charsets are present in the domain. I think this would make ENS much safer and inclusive for international users. For example, English sites could “flag as suspicious” any domain where charsets != ASCII & Emoji, and Chinese sites could flag charsets != Chinese & ASCII, etc.

Open Questions:

This function doesn’t really play well with the existing “UniversalResolver”. Should this function operate only on labels? Or should it have tighter integration with Universal Resolver? Either way, I think users should have an on-chain function which provides direct string -> (...helpful ENS info...) functionality.
I’m not wild about the namehash function name, since this returns more than just the namehash. Any suggestions for a better name?

@raffy That is some really impressive code you have there! Very well done

raffy · July 30, 2022, 1:01am

Here are some thoughts:

Assume max label length fits in 16 bits (or w/e). The low level normalize should be string → (string norm, string normNoEmoji, uint256 labelData) where:

normNoEmoji is the same string but each emoji sequence is zero’d
eg. (4 bytes) → [0,0,0,0]
each labelData is:
- 16 bit → label length
- 240 bits → bitset of active non-emoji codepoints shifted by 14
  (2^21/240 => 14 bits)

From that, it’s easy to compute the namehash, extract out any label, compute any label hash, or quickly determine which validation could apply. eg. the basic validator (DNS + Emoji) only runs if the bitset is 0x1. normNoEmoji avoids processing emoji again during validation.

The primary normalize function should be string → (string, hash) like you describe.

I think all the charset stuff should exist in the validator, whether that’s the same contract or a different contract, I’m not sure. Most validator checks are per-label, so given (label, bitset), you can efficiently check if its valid. IIRC, only check bidi requires a full-name check.

The low level validation function would be (string, bitset) → bool like:

function validate(string label, uint256 bitset) returns (bool) {
     // any 0 byte is a previously processed emoji
     if (bitset == 1) { // only codepoints [0, 0x4000)
         // check if basic
         // check if latin non-confusable
         // check if greek w/o wholescript confusable
         // etc...
     }
     return false;
}

Or more cleanly, the validation contract could have arbitrary functions key’d by bitset + nonce, so you could register/unregister any number of validation functions, and quickly check which ones could apply by intersecting the label bitset, etc.

The primary normalize + validation function would be string → (string norm, uint256 hash, bool valid) where:

given name
(norm, normNoEmoji, labelData) = normalize(name)
start = 0
hash = 0
valid = true
// possibly apply full-name check
for [len, bitset] of labelData.reverse()
    hash = keccak(hash + keccak(norm.slice(start, len)))
    valid &= validate(normNoEmoji.slice(start, len), bitset)
    start += 1 + len
return (norm, hash, valid)

If this throws, the name is invalid
If valid is false, the user should get some kind of warning that the name is potentially unsafe (where unsafe means one or more labels satisfied 0 approved validators.)
If valid isn’t needed, use the primary normalize function instead.

royalfork · July 30, 2022, 3:39am

I haven’t thought much about what the internal API should be, but this certainly looks reasonable. Just to clarify, is the motivation behind the emoji special case (norm vs normNoEmoji) to sort out the 0xfe0f issue? (ie: 0xfe0f must be present for “input/display normalization”, but be absent for namehash calculation?)

Would this also be part of the internal API? I don’t thing the nuance around valid should be exposed to end-users.

Why wouldn’t you also throw in this case?

I think the first step is to build general consensus around a top-level public API to be used by ecosystem developers. Once we have that, I’m pretty confident we can work backwards to nail down the low-level internals.

raffy · July 30, 2022, 4:13am

Validation only needs to check if the non-emoji part of the string obeys all the Unicode rules. But you need to run the full emoji logic to skip over them, so each validator would then need efficient access to the emoji logic. Whereas, allocating a little more memory and keeping a second copy with zero’d emoji bytes means each validators can just skip over 0’s and be fully ignorant of emoji.

This lets you construct the basic validator /[a-z0-9_-]+/ (and many others) very easily.

This is my opinion but I think normalizable-but-confusing names should still work. Especially for the headless situation, eg. aрe.eth confusable can normalize and hash, but valid would be false.

There exist creative/vanity names that will never satisfy even the most relaxed script-based rules simply because they use a single-character of a script that’s confusable in other contexts.