On-chain ENS Domain Normalization

Way back in December, I made the following comment in our now infamous ENS Name Normalization thread:

After taking some time to look into it, I believe that an onchain implementation of “ENS Name Normalization” (as proposed by @raffy in Draft ENSIP - Standardization of ENS Name Normalization) is both technically feasible and financially economic within a mainnet contract.

As a small proof of concept, I built github.com/royalfork/ens-ascii-normalizer, which you can try out on etherscan. ENSNormalizeAscii contains the following functions:

  • namehash(string domain) returns (string, bytes32) returns the normalized domain, and the namehash node of the normalized domain (reverting if domain is invalid).
  • owner(string domain) returns (address) uses namehash to return the domain’s current ENS owner.

Note: This contract only supports ASCII-only domains. If a domain contains non-ASCII characters, namehash/lookup will revert.

Moving to full unicode support:

  • utf8 encoding/decoding incurs additional complexity and computing cost over just ASCII domains. In preliminary tests, validation of a simple 6 letter unicode domain cost about 200000 gas (~$5), which is probably too expensive to be called by other contracts (although this number is based on unoptimized solidity code, and could be reduced).
  • IDNA2008 has ~1 million “unicode” rules which would all need to be supported. These rule can be compressed (run length encoding) into a set of ~2000 “rule import” eth transactions, each costing ~1,500,000 gas. This puts the total cost of IDNA2008 rule importation at around $85,000 (at 30 gwei gas).
  • Emoji validation requires importation of another ~3000 rules (based on saving the full RGI set on-chain, but it might be possible to do this with less). Not sure on the best data structures for this part yet, but would estimate that emoji validation data import would cost another $20,000-$50,000 in gas (I can try and firm up a better estimate if needed).
  • Most UTS-46 mapping would be encoded into the IDNA2008 rule set, but “1 char → n char” mappings would require separate importation. There are only a few of these, but this could add another ~$10,000 to deployment costs (also, implementing unicode 1->n mappings isn’t as simple as the ascii 1->1 mappings, some inline assembly is required to make those mappings memory efficient).
  • Once on-chain, the rules would be customizable, so an “owner” could allow/disallow certain characters and/or change how context-based validation works (emoji validation is implemented via context-based validation).

Not sure if ENSAsciiNormalizer is the first mainnet “string → owner” contract for ENS, but I find it nice to perform direct on-chain domain lookup/validation without the need for intermediary client-side code (it’s also nice to check for ZWJ characters directly on etherscan). If there’s any interest in a “full unicode normalization” contract (and we can collectively stomach the ~$200,000 in fees it would take us to get there), I can put together a more detailed RFP. Would be pretty cool to implement something like this, and it could vastly simplify 3rd party integrations (instead of asking etherscan, metamask, opensea, ethers.js, everyone else to “update their code” everytime ENS normalization rules are updated, we can just point them to our easy-to-use on-chain normalization function and be done with it).

5 Likes

Onchain normalisation would be really useful to have; it would mean that implementations can start by using it, and transition to doing it in the client library for efficiency later, if desired.

Gas efficiency for normalisation needn’t be the top concern; normally there’d be no reason to do the normalisation inside a transaction - instead, clients can call the normalisation function to get the namehash, and pass that in to the transaction.

The costs to upload the tables will be substantial, but I think that with the preprocessing and compression work @raffy has done, it’s probably viable for a much smaller sum.

5 Likes

I extended this into a proof-of-concept for emoji+ascii normalization, and deployed on ropsten here: https://ropsten.etherscan.io/address/0x31096216008a6bc55ba6434488d14c86bdcf4283#code.

This contract maintains a list of all canonical/allowed emoji sequences, and only allows emoji within that set. ZWJ characters are only allowed within emoji sequences, and are only allowed in those emoji sequences that specifically use them, where they must match the allowed sequence verbatim. There’s also some special handling for 0xFE0F such that only fully-qualified emoji are returned as “normalized”, but the minimally-qualified, without 0xFE0F, is used to compute the namehash/labelhash (meaning this is fully backwards-compatible with existing emoji registrations).

This PoC implementation wastes a lot of storage (with expensive “nested maps”), but even in its unoptimized state, saving a large, 5-char-long emoji sequence into the allowlist costs 140k gas (https://ropsten.etherscan.io/tx/0x9ecf9c9665f415e5c97511489efa5185ab3d82ec0c62b8f7a88fa34023462195), which is about $13 on mainnet (30gwei). Saving 3000 RGI emoji would cost worst case $40k, but most likely a lot less (@raffy has some ideas on how this can be substantially improved).

Note: For this deployment, I only added :seven: and :policewoman:t2: to the emoji allowlist…all other emoji will revert if you try and namehash, because they aren’t in the allowlist.

2 Likes

@royalfork - Awesome work. Thanks for taking the time to share the concept and PoC it.

@slobo.eth - Could we get this on the agenda for this week’s Ecosystem WG meeting? It’d be great to discuss this synchronously.
We’re back to 7pm GMT on Monday this week, correct?

1 Like

Here is an initial version of an efficient emoji parser library that should be a drop-in for your contract.

I deployed a contract using it here and populated it.

It requires less than 1 eth in gas to deploy and supports all Unicode 14 emoji. (If the upload mechanism is exposed, it’s also upgradeable, but since it’s so cheap, its probably better to seal it and redeploy for future Unicode updates.)

  • function read(Emoji storage self, uint24[] memory cps, uint256 pos, bool skipFE0F) returns (uint256) — will return the length of the next emoji token, if it exists. If skipFE0F is true, it will effectively ignore FE0F (but it will also return a non-zero result for a run of FE0F's without any emoji.)

  • function filter(Emoji storage self, uint24[] memory cps, bool skipFE0F) returns (uint24[]) — will replace emoji tokens with a single 0xFFFFFF.

If you use as a library, you’ll need to call upload() with the following 10KB payload, broken into (2) chunks:

Chunk 1
0x000026F901010001F3C301010001F3C401010001F3CA01010001F3CB01010001F3CC01010001F46E01010001F47001010001F47101010001F47301010001F47701010001F48101010001F48201010001F48601010001F48701010001F57501010001F64501010001F64601010001F64701010001F64B01010001F64D01010001F64E01010001F6A301010001F6B401010001F6B501010001F6B601010001F92601010001F93501010001F93701010001F93801010001F93901010001F93C01010001F93D01010001F93E01010001F9B801010001F9B901010001F9CD01010001F9CE01010001F9CF01010001F9D401010001F9D601010001F9D701010001F9D801010001F9D901010001F9DA01010001F9DB01010001F9DC01010001F9DD01010100200D00020200264002030200264202030101F3FB01040101F3FC01040101F3FD01040101F3FE01040101F3FF01040400200D00020000276401050500200D00060601F52502030601FA7902030001F3F301070700200D0008080026A702030801F30802030001F3F401090900200D000A0A0026200203090E0067000B0B0E0062000C0C0E0065000D0D0E006E000E0E0E0067000F0F0E007F06100C0E00730011110E00630012120E0074000F0C0E00770013130E006C0014140E0073000F0001F40801151500200D001616002B1B02030001F41501171700200D00181801F9BA02030001F43B01191900200D001A1A00274402030001F441011B1B00200D001C1C01F5E802030001F468011D1D00200D001E1E00269502031E00269602031E00270802031E01F33E02031E01F37302031E01F37C02031E01F39302031E01F3A402031E01F3A802031E01F3EB02031E01F3ED02031E01F4BB02031E01F4BC02031E01F52702031E01F52C02031E01F68002031E01F69202031E01F9AF02031E01F9B002031E01F9B102031E01F9B202031E01F9B302031E01F9BC02031E01F9BD02031E01F466021F1F00200D00202001F46602031E01F46702212100200D00222201F46602032201F46702031E00276400232300200D00242401F46804252401F48B00262600200D00272701F46806101E01F46800281E01F46900282800200D00292901F466042A2A00200D00202901F467042B2B00200D00221D01F3FB412C1D01F3FC412C1D01F3FD412C1D01F3FE412C1D01F3FF412C2C00200D002D2D00269502032D00269602032D00270802032D01F33E02032D01F37302032D01F37C02032D01F39302032D01F3A402032D01F3A802032D01F3EB02032D01F3ED02032D01F4BB02032D01F4BC02032D01F52702032D01F52C02032D01F68002032D01F69202032D01F9AF02032D01F9B002032D01F9B102032D01F9B202032D01F9B302032D01F9BC02032D01F9BD02032D01F91D002E2E00200D002F2F01F46800303001F3FB85313001F3FC85313001F3FD85313001F3FE85313001F3FF85312D00276400323200200D00333301F46800343401F3FB05313401F3FC05313401F3FD05313401F3FE05313401F3FF05313301F48B00353500200D00363601F46800373701F3FB07383701F3FC07383701F3FD07383701F3FE07383701F3FF07380001F46901393900200D003A3A00269502033A00269602033A00270802033A01F33E02033A01F37302033A01F37C02033A01F39302033A01F3A402033A01F3A802033A01F3EB02033A01F3ED02033A01F4BB02033A01F4BC02033A01F52702033A01F52C02033A01F68002033A01F69202033A01F9AF02033A01F9B002033A01F9B102033A01F9B202033A01F9B302033A01F9BC02033A01F9BD02033A01F466021F3A01F46702213A002764003B3B00200D003C3C01F46804253C01F46904253C01F48B003D3D00200D003E3E01F46806103E01F46906103A01F46900283901F3FB413F3901F3FC413F3901F3FD413F3901F3FE413F3901F3FF413F3F00200D00404000269502034000269602034000270802034001F33E02034001F37302034001F37C02034001F39302034001F3A402034001F3A802034001F3EB02034001F3ED02034001F4BB02034001F4BC02034001F52702034001F52C02034001F68002034001F69202034001F9AF02034001F9B002034001F9B102034001F9B202034001F9B302034001F9BC02034001F9BD02034001F91D00414100200D00424201F46800304201F46900304000276400434300200D00444401F46800344401F46900344401F48B00454500200D00464601F46800374601F46900370001F46F01474700200D00024701F3FB01484701F3FC01484701F3FD01484701F3FE01484701F3FF01480001F62E01494900200D004A4A01F4A802030001F635014B4B00200D004C4C01F4AB02030001F636014D4D00200D004E4E01F32B02030001F9D1014F4F00200D00505000269502035000269602035000270802035001F33E02035001F37302035001F37C02035001F38402035001F39302035001F3A402035001F3A802035001F3EB02035001F3ED02035001F4BB02035001F4BC02035001F52702035001F52C02035001F68002035001F69202035001F9AF02035001F9B002035001F9B102035001F9B202035001F9B302035001F9BC02035001F9BD02035001F91D00515100200D00525201F9D104254F01F3FB41534F01F3FC41534F01F3FD41534F01F3FE41534F01F3FF41535300200D00545400269502035400269602035400270802035401F33E02035401F37302035401F37C02035401F38402035401F39302035401F3A402035401F3A802035401F3EB02035401F3ED02035401F4BB02035401F4BC02035401F52702035401F52C02035401F68002035401F69202035401F9AF02035401F9B002035401F9B102035401F9B202035401F9B302035401F9BC02035401F9BD02035400276400555500200D00565601F9D100305601F48B00575700200D00585801F9D100595901F3FB87385901F3FC87385901F3FD87385901F3FE87385901F3FF87385401F91D005A5A00200D005B5B01F9D100340001F9DE01040001F9DF01040001FAF1015C5C01F3FB415D5C01F3FC415D5C01F3FD415D5C01F3FE415D5C01F3FF415D5D00200D005E5E01FAF2005F5F01F3FB83605F01F3FC83605F01F3FD83605F01F3FE83605F01F3FF8360000000A90148000000AE01480000203C0148000020490148000021940148000021950148000021960148000021970148000021980148000021990148000021A90148000021AA01480000231A01480000231B0148000023280148000023CF0148000023E90148000023EA0148000023EB0148000023EC0148000023ED0148000023EE0148000023EF0148000023F00148000023F10148000023F20148000023F30148000023F80148000023F90148000023FA0148000025AA0148000025AB0148000025B60148000025C00148000025FB0148000025FC0148000025FD0148000025FE01480000260001480000260101480000260201480000260301480000260401480000260E01480000261101480000261401480000261501480000261801480000262001480000262201480000262301480000262601480000262A01480000262E01480000262F01480000263801480000263901480000263A01480000264001480000264201480000264801480000264901480000264A01480000264B01480000264C01480000264D01480000264E01480000264F01480000265001480000265101480000265201480000265301480000265F01480000266001480000266301480000266501480000266601480000266801480000267B01480000267E01480000267F01480000269201480000269301480000269401480000269501480000269601480000269701480000269901480000269B01480000269C0148000026A00148000026A10148000026A70148000026AA0148000026AB0148000026B00148000026B10148000026BD0148000026BE0148000026C40148000026C50148000026C80148000026CE0148000026CF0148000026D10148000026D30148000026D40148000026E90148000026EA0148000026F00148000026F10148000026F20148000026F30148000026F40148000026F50148000026F70148000026F80148000026FA0148000026FD01480000270201480000270501480000270801480000270901480000270F01480000271201480000271401480000271601480000271D01480000272101480000272801480000273301480000273401480000274401480000274701480000274C01480000274E0148000027530148000027540148000027550148000027570148000027630148000027950148000027960148000027970148000027A10148000027B00148000027BF014800002934014800002935014800002B05014800002B06014800002B07014800002B1B014800002B1C014800002B50014800002B5501480000303001480000303D01480001F00401480001F0CF01480001F17001480001F17101480001F17E01480001F17F01480001F18E01480001F19101480001F19201480001F19301480001F19401480001F19501480001F19601480001F19701480001F19801480001F19901480001F19A01480001F1E601480001F1E701480001F1E801480001F1E901480001F1EA01480001F1EB01480001F1EC01480001F1ED01480001F1EE01480001F1EF01480001F1F001480001F1F101480001F1F201480001F1F301480001F1F401480001F1F501480001F1F601480001F1F701480001F1F801480001F1F901480001F1FA01480001F1FB01480001F1FC01480001F1FD01480001F1FE01480001F1FF01480001F30001480001F30101480001F30201480001F30301480001F30401480001F30501480001F30601480001F30701480001F30801480001F30901480001F30A01480001F30B01480001F30C01480001F30D01480001F30E01480001F30F01480001F31001480001F31101480001F31201480001F31301480001F31401480001F31501480001F31601480001F31701480001F31801480001F31901480001F31A01480001F31B01480001F31C01480001F31D01480001F31E01480001F31F01480001F32001480001F32101480001F32401480001F32501480001F32601480001F32701480001F32801480001F32901480001F32A01480001F32B01480001F32C01480001F32D01480001F32E01480001F32F01480001F33001480001F33101480001F33201480001F33301480001F33401480001F33501480001F33601480001F33701480001F33801480001F33901480001F33A01480001F33B01480001F33C01480001F33D01480001F33E01480001F33F01480001F34001480001F34101480001F34201480001F34301480001F34401480001F34501480001F34601480001F34701480001F34801480001F34901480001F34A01480001F34B01480001F34C01480001F34D01480001F34E01480001F34F01480001F35001480001F35101480001F35201480001F35301480001F35401480001F35501480001F35601480001F35701480001F35801480001F35901480001F35A01480001F35B01480001F35C01480001F35D01480001F35E01480001F35F01480001F36001480001F36101480001F36201480001F36301480001F36401480001F36501480001F36601480001F36701480001F36801480001F36901480001F36A01480001F36B01480001F36C01480001F36D01480001F36E01480001F36F01480001F37001480001F37101480001F37201480001F37301480001F37401480001F37501480001F37601480001F37701480001F37801480001F37901480001F37A01480001F37B01480001F37C01480001F37D01480001F37E01480001F37F01480001F38001480001F38101480001F38201480001F38301480001F38401480001F38601480001F38701480001F38801480001F38901480001F38A01480001F38B01480001F38C01480001F38D01480001F38E01480001F38F01480001F39001480001F39101480001F39201480001F39301480001F39601480001F39701480001F39901480001F39A01480001F39B01480001F39E01480001F39F01480001F3A001480001F3A101480001F3A201480001F3A301480001F3A401480001F3A501480001F3A601480001F3A701480001F3A801480001F3A901480001F3AA01480001F3AB01480001F3AC01480001F3AD01480001F3AE01480001F3AF01480001F3B001480001F3B101480001F3B201480001F3B301480001F3B401480001F3B501480001F3B601480001F3B701480001F3B801480001F3B901480001F3BA01480001F3BB01480001F3BC01480001F3BD01480001F3BE01480001F3BF01480001F3C001480001F3C101480001F3C501480001F3C601480001F3C801480001F3C901480001F3CD01480001F3CE01480001F3CF01480001F3D001480001F3D101480001F3D201480001F3D301480001F3D401480001F3D501480001F3D601480001F3D701480001F3D801480001F3D901480001F3DA01480001F3DB01480001F3DC01480001F3DD01480001F3DE01480001F3DF01480001F3E001480001F3E101480001F3E201480001F3E301480001F3E401480001F3E501480001F3E601480001F3E701480001F3E801480001F3E901480001F3EA01480001F3EB01480001F3EC01480001F3ED01480001F3EE01480001F3EF01480001F3F001480001F3F501480001F3F701480001F3F801480001F3F901480001F3FA01480001F3FB01480001F3FC01480001F3FD01480001F3FE01480001F3FF01480001F40001480001F40101480001F40201480001F40301480001F40401480001F40501480001F40601480001F40701480001F40901480001F40A01480001F40B01480001F40C01480001F40D01480001F40E01480001F40F01480001F41001480001F41101480001F41201480001F41301480001F41401480001F41601480001F41701480001F41801480001F41901480001F41A01480001F41B01480001F41C01480001F41D01480001F41E01480001F41F01480001F42001480001F42101480001F42201480001F42301480001F42401480001F42501480001F42601480001F42701480001F42801480001F42901480001F42A01480001F42B01480001F42C01480001F42D01480001F42E01480001F42F01480001F43001480001F43101480001F43201480001F43301480001F43401480001F43501480001F43601480001F43701480001F43801480001F43901480001F43A01480001F43C01480001F43D01480001F43E01480001F43F01480001F44001480001F44401480001F44501480001F45101480001F45201480001F45301480001F45401480001F45501480001F45601480001F45701480001F45801480001F45901480001F45A01480001F45B01480001F45C01480001F45D01480001F45E01480001F45F01480001F46001480001F46101480001F46201480001F46301480001F46401480001F46501480001F47901480001F47A01480001F47B01480001F47D01480001F47E01480001F47F01480001F48001480001F48401480001F48801480001F48901480001F48A01480001F48B01480001F48C01480001F48D01480001F48E01480001F49001480001F49201480001F49301480001F49401480001F49501480001F49601480001F49701480001F49801480001F49901480001F49A01480001F49B01480001F49C01480001F49D01480001F49E01480001F49F01480001F4A001480001F4A101480001F4A201480001F4A301480001F4A401480001F4A501480001F4A601480001F4A701480001F4A801480001F4A901480001F4AB01480001F4AC01480001F4AD01480001F4AE01480001F4AF01480001F4B001480001F4B101480001F4B201480001F4B301480001F4B401480001F4B501480001F4B601480001F4B701480001F4B801480001F4B901480001F4BA01480001F4BB01480001F4BC01480001F4BD01480001F4BE01480001F4BF01480001F4C001480001F4C101480001F4C201480001F4C301480001F4C401480001F4C501480001F4C601480001F4C701480001F4C801480001F4C901480001F4CA01480001F4CB01480001F4CC01480001F4CD01480001F4CE01480001F4CF01480001F4D001480001F4D101480001F4D201480001F4D301480001F4D401480001F4D501480001F4D601480001F4D701480001F4D801480001F4D901480001F4DA01480001F4DB01480001F4DC01480001F4DD01480001F4DE01480001F4DF01480001F4E001480001F4E101480001F4E201480001F4E301480001F4E401480001F4E501480001F4E601480001F4E70148
Chunk 2
0x0001F4E801480001F4E901480001F4EA01480001F4EB01480001F4EC01480001F4ED01480001F4EE01480001F4EF01480001F4F001480001F4F101480001F4F201480001F4F301480001F4F401480001F4F501480001F4F601480001F4F701480001F4F801480001F4F901480001F4FA01480001F4FB01480001F4FC01480001F4FD01480001F4FF01480001F50001480001F50101480001F50201480001F50301480001F50401480001F50501480001F50601480001F50701480001F50801480001F50901480001F50A01480001F50B01480001F50C01480001F50D01480001F50E01480001F50F01480001F51001480001F51101480001F51201480001F51301480001F51401480001F51501480001F51601480001F51701480001F51801480001F51901480001F51A01480001F51B01480001F51C01480001F51D01480001F51E01480001F51F01480001F52001480001F52101480001F52201480001F52301480001F52401480001F52501480001F52601480001F52701480001F52801480001F52901480001F52A01480001F52B01480001F52C01480001F52D01480001F52E01480001F52F01480001F53001480001F53101480001F53201480001F53301480001F53401480001F53501480001F53601480001F53701480001F53801480001F53901480001F53A01480001F53B01480001F53C01480001F53D01480001F54901480001F54A01480001F54B01480001F54C01480001F54D01480001F54E01480001F55001480001F55101480001F55201480001F55301480001F55401480001F55501480001F55601480001F55701480001F55801480001F55901480001F55A01480001F55B01480001F55C01480001F55D01480001F55E01480001F55F01480001F56001480001F56101480001F56201480001F56301480001F56401480001F56501480001F56601480001F56701480001F56F01480001F57001480001F57301480001F57601480001F57701480001F57801480001F57901480001F58701480001F58A01480001F58B01480001F58C01480001F58D01480001F5A401480001F5A501480001F5A801480001F5B101480001F5B201480001F5BC01480001F5C201480001F5C301480001F5C401480001F5D101480001F5D201480001F5D301480001F5DC01480001F5DD01480001F5DE01480001F5E101480001F5E301480001F5E801480001F5EF01480001F5F301480001F5FA01480001F5FB01480001F5FC01480001F5FD01480001F5FE01480001F5FF01480001F60001480001F60101480001F60201480001F60301480001F60401480001F60501480001F60601480001F60701480001F60801480001F60901480001F60A01480001F60B01480001F60C01480001F60D01480001F60E01480001F60F01480001F61001480001F61101480001F61201480001F61301480001F61401480001F61501480001F61601480001F61701480001F61801480001F61901480001F61A01480001F61B01480001F61C01480001F61D01480001F61E01480001F61F01480001F62001480001F62101480001F62201480001F62301480001F62401480001F62501480001F62601480001F62701480001F62801480001F62901480001F62A01480001F62B01480001F62C01480001F62D01480001F62F01480001F63001480001F63101480001F63201480001F63301480001F63401480001F63701480001F63801480001F63901480001F63A01480001F63B01480001F63C01480001F63D01480001F63E01480001F63F01480001F64001480001F64101480001F64201480001F64301480001F64401480001F64801480001F64901480001F64A01480001F68001480001F68101480001F68201480001F68301480001F68401480001F68501480001F68601480001F68701480001F68801480001F68901480001F68A01480001F68B01480001F68C01480001F68D01480001F68E01480001F68F01480001F69001480001F69101480001F69201480001F69301480001F69401480001F69501480001F69601480001F69701480001F69801480001F69901480001F69A01480001F69B01480001F69C01480001F69D01480001F69E01480001F69F01480001F6A001480001F6A101480001F6A201480001F6A401480001F6A501480001F6A601480001F6A701480001F6A801480001F6A901480001F6AA01480001F6AB01480001F6AC01480001F6AD01480001F6AE01480001F6AF01480001F6B001480001F6B101480001F6B201480001F6B301480001F6B701480001F6B801480001F6B901480001F6BA01480001F6BB01480001F6BC01480001F6BD01480001F6BE01480001F6BF01480001F6C101480001F6C201480001F6C301480001F6C401480001F6C501480001F6CB01480001F6CD01480001F6CE01480001F6CF01480001F6D001480001F6D101480001F6D201480001F6D501480001F6D601480001F6D701480001F6DD01480001F6DE01480001F6DF01480001F6E001480001F6E101480001F6E201480001F6E301480001F6E401480001F6E501480001F6E901480001F6EB01480001F6EC01480001F6F001480001F6F301480001F6F401480001F6F501480001F6F601480001F6F701480001F6F801480001F6F901480001F6FA01480001F6FB01480001F6FC01480001F7E001480001F7E101480001F7E201480001F7E301480001F7E401480001F7E501480001F7E601480001F7E701480001F7E801480001F7E901480001F7EA01480001F7EB01480001F7F001480001F90D01480001F90E01480001F91001480001F91101480001F91201480001F91301480001F91401480001F91501480001F91601480001F91701480001F92001480001F92101480001F92201480001F92301480001F92401480001F92501480001F92701480001F92801480001F92901480001F92A01480001F92B01480001F92C01480001F92D01480001F92E01480001F92F01480001F93A01480001F93F01480001F94001480001F94101480001F94201480001F94301480001F94401480001F94501480001F94701480001F94801480001F94901480001F94A01480001F94B01480001F94C01480001F94D01480001F94E01480001F94F01480001F95001480001F95101480001F95201480001F95301480001F95401480001F95501480001F95601480001F95701480001F95801480001F95901480001F95A01480001F95B01480001F95C01480001F95D01480001F95E01480001F95F01480001F96001480001F96101480001F96201480001F96301480001F96401480001F96501480001F96601480001F96701480001F96801480001F96901480001F96A01480001F96B01480001F96C01480001F96D01480001F96E01480001F96F01480001F97001480001F97101480001F97201480001F97301480001F97401480001F97501480001F97601480001F97801480001F97901480001F97A01480001F97B01480001F97C01480001F97D01480001F97E01480001F97F01480001F98001480001F98101480001F98201480001F98301480001F98401480001F98501480001F98601480001F98701480001F98801480001F98901480001F98A01480001F98B01480001F98C01480001F98D01480001F98E01480001F98F01480001F99001480001F99101480001F99201480001F99301480001F99401480001F99501480001F99601480001F99701480001F99801480001F99901480001F99A01480001F99B01480001F99C01480001F99D01480001F99E01480001F99F01480001F9A001480001F9A101480001F9A201480001F9A301480001F9A401480001F9A501480001F9A601480001F9A701480001F9A801480001F9A901480001F9AA01480001F9AB01480001F9AC01480001F9AD01480001F9AE01480001F9AF01480001F9B001480001F9B101480001F9B201480001F9B301480001F9B401480001F9B701480001F9BA01480001F9BC01480001F9BD01480001F9BE01480001F9BF01480001F9C001480001F9C101480001F9C201480001F9C301480001F9C401480001F9C501480001F9C601480001F9C701480001F9C801480001F9C901480001F9CA01480001F9CB01480001F9CC01480001F9D001480001F9E001480001F9E101480001F9E201480001F9E301480001F9E401480001F9E501480001F9E601480001F9E701480001F9E801480001F9E901480001F9EA01480001F9EB01480001F9EC01480001F9ED01480001F9EE01480001F9EF01480001F9F001480001F9F101480001F9F201480001F9F301480001F9F401480001F9F501480001F9F601480001F9F701480001F9F801480001F9F901480001F9FA01480001F9FB01480001F9FC01480001F9FD01480001F9FE01480001F9FF01480001FA7001480001FA7101480001FA7201480001FA7301480001FA7401480001FA7801480001FA7901480001FA7A01480001FA7B01480001FA7C01480001FA8001480001FA8101480001FA8201480001FA8301480001FA8401480001FA8501480001FA8601480001FA9001480001FA9101480001FA9201480001FA9301480001FA9401480001FA9501480001FA9601480001FA9701480001FA9801480001FA9901480001FA9A01480001FA9B01480001FA9C01480001FA9D01480001FA9E01480001FA9F01480001FAA001480001FAA101480001FAA201480001FAA301480001FAA401480001FAA501480001FAA601480001FAA701480001FAA801480001FAA901480001FAAA01480001FAAB01480001FAAC01480001FAB001480001FAB101480001FAB201480001FAB301480001FAB401480001FAB501480001FAB601480001FAB701480001FAB801480001FAB901480001FABA01480001FAC001480001FAC101480001FAC201480001FAD001480001FAD101480001FAD201480001FAD301480001FAD401480001FAD501480001FAD601480001FAD701480001FAD801480001FAD901480001FAE001480001FAE101480001FAE201480001FAE301480001FAE401480001FAE501480001FAE601480001FAE701480000261D01610000270A01610000270B01610000270C01610000270D01610001F38501610001F3C201610001F3C701610001F44201610001F44301610001F44601610001F44701610001F44801610001F44901610001F44A01610001F44B01610001F44C01610001F44D01610001F44E01610001F44F01610001F45001610001F46601610001F46701610001F46A01610001F46B01610001F46C01610001F46D01610001F47201610001F47401610001F47501610001F47601610001F47801610001F47C01610001F48301610001F48501610001F48F01610001F49101610001F4AA01610001F57401610001F57A01610001F59001610001F59501610001F59601610001F64C01610001F64F01610001F6C001610001F6CC01610001F90C01610001F90F01610001F91801610001F91901610001F91A01610001F91B01610001F91C01610001F91D01610001F91E01610001F91F01610001F93001610001F93101610001F93201610001F93301610001F93401610001F93601610001F97701610001F9B501610001F9B601610001F9BB01610001F9D201610001F9D301610001F9D501610001FAC301610001FAC401610001FAC501610001FAF001610001FAF201610001FAF301610001FAF401610001FAF501610001FAF601616101F3FB01486101F3FC01486101F3FD01486101F3FE01486101F3FF01480000002300620000002A0062000000300062000000310062000000320062000000330062000000340062000000350062000000360062000000370062000000380062000000390062620020E30203

The chunks are 6 byte rules for state transitions which follow this tree:


Here is a test contract that I’ve set the above contract as the emoji parser and a BasicValidator contract that implements /^[a-z09-_.]+/$ as the validator.

  • filter(string name, bool ignoreFEOF) should parse any UTF8 string with emojis.

  • validate(string name) should validate any normalized name which uses valid emoji or obeys the regex above. Note: This will fail if emoji have FE0F (which isn’t normalized). This should validate 94% of registered names as-of today.

Outstanding! Do you have an estimate of what it would take to extend this to match your entire normalisation function, not just Emoji?

It depends on how IDNA and NFC are implemented. If you implement them verbatim, it will be very expensive to deploy like royalfork estimated. However, there’s probably a partial mapping that would give a lot of bang for the buck.

I think there is a possibility of using NFKD → Apply the small number of differences from IDNA 2003 → NFC, but I’d need to verify that this is the same thing as IDNA 2003 → NFD → NFC. You could avoid a whole lookup table with this approach.

For IDNA, we can pack 10 uint24 per storage slot for single char mappings. There are runs of “ABCabc” and “AaBbCc” for casefolding, and runs of “ABC”, “A?B?C?”, and “A??B??C??” that map to a constant. There’s a bunch of ideas in my github.

If, for the moment, we reject combining marks and composite characters (so NFC isn’t needed) then my emoji parser + an efficient IDNA mapping would be a string → string implementation. All the effort would go into efficiently compressing the mapping.

Otherwise, I think the most important contract is an NFD/NFC implementation.


For validation:

You can naively encode the valid output character set into a bitmap for 700 storage slots. Combine that with the emoji contract above + NFC quick check and that’s a gas efficient function for determining if a name is in the normalized form. I can demo this immediately once I have an NFC implementation.

I also think a few separate simple validators for each script would give a lot of coverage (and more could be added over time). However, this approach requires a mechanism for looping through the set of validators to find one which returns true for your name.

I mentioned in the other thread, you could store the address of validator on-chain for the name you verified, making all future validations only two SLOADs, node → validator address → is valid validator address, but this requires extra infrastructure.

1 Like

This seems to be a reasonable starting point. Are there existing domains that use combining marks or composite characters?

1 Like

5K names have combined characters (NFC != NFD.)
171 names have two combining marks that could be reordered.

Codepoints: 2.4K CM and 13K composed.

Maybe that is a good idea, just compress the IDNA part and disallow the codepoints that need NFC. That’s 99% of names unless I’m forgetting something.


@adraffy/ens-norm-research has an improved contract that only needs 360 storage slots and no longer requires the skipFE0F option, and the code that generates the state payload.


Emoji + IDNA 2003 - (Combining Marks, Compositions) can be done with 10KB contract code, 128 slots for Valid bitmap, 450 Slots for Mappings of [1-2 CP], 175 Slots for Mappings [3+ CP], 360 slots for Emoji.

[128 + 452 + 175 + 360] → 1115 slots → ~30M + contract gas so under 1 eth to deploy.

122810 valid, 4208 mapped, 9 ignored, 3046 emoji sequences.

Since the IDNA table is stable, a huge chunk of the valid and single character mappings can be compiled directly to code.

I think worse-case per character would be, ~15 comparisons (x < A), 10 range comparisions (x >= A && x <= B), 1 sload for valid, 1 sload for small mapping, 1 sload for big mapping, and then calling the emoji function which is (1 sload per codepoint.)


Edit: Here is my first working version: https://rinkeby.etherscan.io/address/0xdb68eb6f0ab93bc7059f2752d477b45f04457eb2#readContract

Cost 32M gas to deploy and populate.

image

Should work for every name except those with combining marks or characters that compose (like à).


Edit 2: I deployed a more gas efficient version that avoids using uint24[]. It also has a beautify() function.

I’m seeing about 50-100K gas per name.

Although I have little idea what you’re on about half the time here, your commitment to see this all the way through is admirable, @raffy :1st_place_medal:

1 Like