On-chain ENS Domain Normalization

raffy · June 26, 2022, 8:26am

Here is an initial version of an efficient emoji parser library that should be a drop-in for your contract.

I deployed a contract using it here and populated it.

It requires less than 1 eth in gas to deploy and supports all Unicode 14 emoji. (If the upload mechanism is exposed, it’s also upgradeable, but since it’s so cheap, its probably better to seal it and redeploy for future Unicode updates.)

function read(Emoji storage self, uint24[] memory cps, uint256 pos, bool skipFE0F) returns (uint256) — will return the length of the next emoji token, if it exists. If skipFE0F is true, it will effectively ignore FE0F (but it will also return a non-zero result for a run of FE0F's without any emoji.)
function filter(Emoji storage self, uint24[] memory cps, bool skipFE0F) returns (uint24[]) — will replace emoji tokens with a single 0xFFFFFF.

If you use as a library, you’ll need to call upload() with the following 10KB payload, broken into (2) chunks:

Chunk 1

0x000026F901010001F3C301010001F3C401010001F3CA01010001F3CB01010001F3CC01010001F46E01010001F47001010001F47101010001F47301010001F47701010001F48101010001F48201010001F48601010001F48701010001F57501010001F64501010001F64601010001F64701010001F64B01010001F64D01010001F64E01010001F6A301010001F6B401010001F6B501010001F6B601010001F92601010001F93501010001F93701010001F93801010001F93901010001F93C01010001F93D01010001F93E01010001F9B801010001F9B901010001F9CD01010001F9CE01010001F9CF01010001F9D401010001F9D601010001F9D701010001F9D801010001F9D901010001F9DA01010001F9DB01010001F9DC01010001F9DD01010100200D00020200264002030200264202030101F3FB01040101F3FC01040101F3FD01040101F3FE01040101F3FF01040400200D00020000276401050500200D00060601F52502030601FA7902030001F3F301070700200D0008080026A702030801F30802030001F3F401090900200D000A0A0026200203090E0067000B0B0E0062000C0C0E0065000D0D0E006E000E0E0E0067000F0F0E007F06100C0E00730011110E00630012120E0074000F0C0E00770013130E006C0014140E0073000F0001F40801151500200D001616002B1B02030001F41501171700200D00181801F9BA02030001F43B01191900200D001A1A00274402030001F441011B1B00200D001C1C01F5E802030001F468011D1D00200D001E1E00269502031E00269602031E00270802031E01F33E02031E01F37302031E01F37C02031E01F39302031E01F3A402031E01F3A802031E01F3EB02031E01F3ED02031E01F4BB02031E01F4BC02031E01F52702031E01F52C02031E01F68002031E01F69202031E01F9AF02031E01F9B002031E01F9B102031E01F9B202031E01F9B302031E01F9BC02031E01F9BD02031E01F466021F1F00200D00202001F46602031E01F46702212100200D00222201F46602032201F46702031E00276400232300200D00242401F46804252401F48B00262600200D00272701F46806101E01F46800281E01F46900282800200D00292901F466042A2A00200D00202901F467042B2B00200D00221D01F3FB412C1D01F3FC412C1D01F3FD412C1D01F3FE412C1D01F3FF412C2C00200D002D2D00269502032D00269602032D00270802032D01F33E02032D01F37302032D01F37C02032D01F39302032D01F3A402032D01F3A802032D01F3EB02032D01F3ED02032D01F4BB02032D01F4BC02032D01F52702032D01F52C02032D01F68002032D01F69202032D01F9AF02032D01F9B002032D01F9B102032D01F9B202032D01F9B302032D01F9BC02032D01F9BD02032D01F91D002E2E00200D002F2F01F46800303001F3FB85313001F3FC85313001F3FD85313001F3FE85313001F3FF85312D00276400323200200D00333301F46800343401F3FB05313401F3FC05313401F3FD05313401F3FE05313401F3FF05313301F48B00353500200D00363601F46800373701F3FB07383701F3FC07383701F3FD07383701F3FE07383701F3FF07380001F46901393900200D003A3A00269502033A00269602033A00270802033A01F33E02033A01F37302033A01F37C02033A01F39302033A01F3A402033A01F3A802033A01F3EB02033A01F3ED02033A01F4BB02033A01F4BC02033A01F52702033A01F52C02033A01F68002033A01F69202033A01F9AF02033A01F9B002033A01F9B102033A01F9B202033A01F9B302033A01F9BC02033A01F9BD02033A01F466021F3A01F46702213A002764003B3B00200D003C3C01F46804253C01F46904253C01F48B003D3D00200D003E3E01F46806103E01F46906103A01F46900283901F3FB413F3901F3FC413F3901F3FD413F3901F3FE413F3901F3FF413F3F00200D00404000269502034000269602034000270802034001F33E02034001F37302034001F37C02034001F39302034001F3A402034001F3A802034001F3EB02034001F3ED02034001F4BB02034001F4BC02034001F52702034001F52C02034001F68002034001F69202034001F9AF02034001F9B002034001F9B102034001F9B202034001F9B302034001F9BC02034001F9BD02034001F91D00414100200D00424201F46800304201F46900304000276400434300200D00444401F46800344401F46900344401F48B00454500200D00464601F46800374601F46900370001F46F01474700200D00024701F3FB01484701F3FC01484701F3FD01484701F3FE01484701F3FF01480001F62E01494900200D004A4A01F4A802030001F635014B4B00200D004C4C01F4AB02030001F636014D4D00200D004E4E01F32B02030001F9D1014F4F00200D00505000269502035000269602035000270802035001F33E02035001F37302035001F37C02035001F38402035001F39302035001F3A402035001F3A802035001F3EB02035001F3ED02035001F4BB02035001F4BC02035001F52702035001F52C02035001F68002035001F69202035001F9AF02035001F9B002035001F9B102035001F9B202035001F9B302035001F9BC02035001F9BD02035001F91D00515100200D00525201F9D104254F01F3FB41534F01F3FC41534F01F3FD41534F01F3FE41534F01F3FF41535300200D00545400269502035400269602035400270802035401F33E02035401F37302035401F37C02035401F38402035401F39302035401F3A402035401F3A802035401F3EB02035401F3ED02035401F4BB02035401F4BC02035401F52702035401F52C02035401F68002035401F69202035401F9AF02035401F9B002035401F9B102035401F9B202035401F9B302035401F9BC02035401F9BD02035400276400555500200D00565601F9D100305601F48B00575700200D00585801F9D100595901F3FB87385901F3FC87385901F3FD87385901F3FE87385901F3FF87385401F91D005A5A00200D005B5B01F9D100340001F9DE01040001F9DF01040001FAF1015C5C01F3FB415D5C01F3FC415D5C01F3FD415D5C01F3FE415D5C01F3FF415D5D00200D005E5E01FAF2005F5F01F3FB83605F01F3FC83605F01F3FD83605F01F3FE83605F01F3FF8360000000A90148000000AE01480000203C0148000020490148000021940148000021950148000021960148000021970148000021980148000021990148000021A90148000021AA01480000231A01480000231B0148000023280148000023CF0148000023E90148000023EA0148000023EB0148000023EC0148000023ED0148000023EE0148000023EF0148000023F00148000023F10148000023F20148000023F30148000023F80148000023F90148000023FA0148000025AA0148000025AB0148000025B60148000025C00148000025FB0148000025FC0148000025FD0148000025FE01480000260001480000260101480000260201480000260301480000260401480000260E01480000261101480000261401480000261501480000261801480000262001480000262201480000262301480000262601480000262A01480000262E01480000262F01480000263801480000263901480000263A01480000264001480000264201480000264801480000264901480000264A01480000264B01480000264C01480000264D01480000264E01480000264F01480000265001480000265101480000265201480000265301480000265F01480000266001480000266301480000266501480000266601480000266801480000267B01480000267E01480000267F01480000269201480000269301480000269401480000269501480000269601480000269701480000269901480000269B01480000269C0148000026A00148000026A10148000026A70148000026AA0148000026AB0148000026B00148000026B10148000026BD0148000026BE0148000026C40148000026C50148000026C80148000026CE0148000026CF0148000026D10148000026D30148000026D40148000026E90148000026EA0148000026F00148000026F10148000026F20148000026F30148000026F40148000026F50148000026F70148000026F80148000026FA0148000026FD01480000270201480000270501480000270801480000270901480000270F01480000271201480000271401480000271601480000271D01480000272101480000272801480000273301480000273401480000274401480000274701480000274C01480000274E0148000027530148000027540148000027550148000027570148000027630148000027950148000027960148000027970148000027A10148000027B00148000027BF014800002934014800002935014800002B05014800002B06014800002B07014800002B1B014800002B1C014800002B50014800002B5501480000303001480000303D01480001F00401480001F0CF01480001F17001480001F17101480001F17E01480001F17F01480001F18E01480001F19101480001F19201480001F19301480001F19401480001F19501480001F19601480001F19701480001F19801480001F19901480001F19A01480001F1E601480001F1E701480001F1E801480001F1E901480001F1EA01480001F1EB01480001F1EC01480001F1ED01480001F1EE01480001F1EF01480001F1F001480001F1F101480001F1F201480001F1F301480001F1F401480001F1F501480001F1F601480001F1F701480001F1F801480001F1F901480001F1FA01480001F1FB01480001F1FC01480001F1FD01480001F1FE01480001F1FF01480001F30001480001F30101480001F30201480001F30301480001F30401480001F30501480001F30601480001F30701480001F30801480001F30901480001F30A01480001F30B01480001F30C01480001F30D01480001F30E01480001F30F01480001F31001480001F31101480001F31201480001F31301480001F31401480001F31501480001F31601480001F31701480001F31801480001F31901480001F31A01480001F31B01480001F31C01480001F31D01480001F31E01480001F31F01480001F32001480001F32101480001F32401480001F32501480001F32601480001F32701480001F32801480001F32901480001F32A01480001F32B01480001F32C01480001F32D01480001F32E01480001F32F01480001F33001480001F33101480001F33201480001F33301480001F33401480001F33501480001F33601480001F33701480001F33801480001F33901480001F33A01480001F33B01480001F33C01480001F33D01480001F33E01480001F33F01480001F34001480001F34101480001F34201480001F34301480001F34401480001F34501480001F34601480001F34701480001F34801480001F34901480001F34A01480001F34B01480001F34C01480001F34D01480001F34E01480001F34F01480001F35001480001F35101480001F35201480001F35301480001F35401480001F35501480001F35601480001F35701480001F35801480001F35901480001F35A01480001F35B01480001F35C01480001F35D01480001F35E01480001F35F01480001F36001480001F36101480001F36201480001F36301480001F36401480001F36501480001F36601480001F36701480001F36801480001F36901480001F36A01480001F36B01480001F36C01480001F36D01480001F36E01480001F36F01480001F37001480001F37101480001F37201480001F37301480001F37401480001F37501480001F37601480001F37701480001F37801480001F37901480001F37A01480001F37B01480001F37C01480001F37D01480001F37E01480001F37F01480001F38001480001F38101480001F38201480001F38301480001F38401480001F38601480001F38701480001F38801480001F38901480001F38A01480001F38B01480001F38C01480001F38D01480001F38E01480001F38F01480001F39001480001F39101480001F39201480001F39301480001F39601480001F39701480001F39901480001F39A01480001F39B01480001F39E01480001F39F01480001F3A001480001F3A101480001F3A201480001F3A301480001F3A401480001F3A501480001F3A601480001F3A701480001F3A801480001F3A901480001F3AA01480001F3AB01480001F3AC01480001F3AD01480001F3AE01480001F3AF01480001F3B001480001F3B101480001F3B201480001F3B301480001F3B401480001F3B501480001F3B601480001F3B701480001F3B801480001F3B901480001F3BA01480001F3BB01480001F3BC01480001F3BD01480001F3BE01480001F3BF01480001F3C001480001F3C101480001F3C501480001F3C601480001F3C801480001F3C901480001F3CD01480001F3CE01480001F3CF01480001F3D001480001F3D101480001F3D201480001F3D301480001F3D401480001F3D501480001F3D601480001F3D701480001F3D801480001F3D901480001F3DA01480001F3DB01480001F3DC01480001F3DD01480001F3DE01480001F3DF01480001F3E001480001F3E101480001F3E201480001F3E301480001F3E401480001F3E501480001F3E601480001F3E701480001F3E801480001F3E901480001F3EA01480001F3EB01480001F3EC01480001F3ED01480001F3EE01480001F3EF01480001F3F001480001F3F501480001F3F701480001F3F801480001F3F901480001F3FA01480001F3FB01480001F3FC01480001F3FD01480001F3FE01480001F3FF01480001F40001480001F40101480001F40201480001F40301480001F40401480001F40501480001F40601480001F40701480001F40901480001F40A01480001F40B01480001F40C01480001F40D01480001F40E01480001F40F01480001F41001480001F41101480001F41201480001F41301480001F41401480001F41601480001F41701480001F41801480001F41901480001F41A01480001F41B01480001F41C01480001F41D01480001F41E01480001F41F01480001F42001480001F42101480001F42201480001F42301480001F42401480001F42501480001F42601480001F42701480001F42801480001F42901480001F42A01480001F42B01480001F42C01480001F42D01480001F42E01480001F42F01480001F43001480001F43101480001F43201480001F43301480001F43401480001F43501480001F43601480001F43701480001F43801480001F43901480001F43A01480001F43C01480001F43D01480001F43E01480001F43F01480001F44001480001F44401480001F44501480001F45101480001F45201480001F45301480001F45401480001F45501480001F45601480001F45701480001F45801480001F45901480001F45A01480001F45B01480001F45C01480001F45D01480001F45E01480001F45F01480001F46001480001F46101480001F46201480001F46301480001F46401480001F46501480001F47901480001F47A01480001F47B01480001F47D01480001F47E01480001F47F01480001F48001480001F48401480001F48801480001F48901480001F48A01480001F48B01480001F48C01480001F48D01480001F48E01480001F49001480001F49201480001F49301480001F49401480001F49501480001F49601480001F49701480001F49801480001F49901480001F49A01480001F49B01480001F49C01480001F49D01480001F49E01480001F49F01480001F4A001480001F4A101480001F4A201480001F4A301480001F4A401480001F4A501480001F4A601480001F4A701480001F4A801480001F4A901480001F4AB01480001F4AC01480001F4AD01480001F4AE01480001F4AF01480001F4B001480001F4B101480001F4B201480001F4B301480001F4B401480001F4B501480001F4B601480001F4B701480001F4B801480001F4B901480001F4BA01480001F4BB01480001F4BC01480001F4BD01480001F4BE01480001F4BF01480001F4C001480001F4C101480001F4C201480001F4C301480001F4C401480001F4C501480001F4C601480001F4C701480001F4C801480001F4C901480001F4CA01480001F4CB01480001F4CC01480001F4CD01480001F4CE01480001F4CF01480001F4D001480001F4D101480001F4D201480001F4D301480001F4D401480001F4D501480001F4D601480001F4D701480001F4D801480001F4D901480001F4DA01480001F4DB01480001F4DC01480001F4DD01480001F4DE01480001F4DF01480001F4E001480001F4E101480001F4E201480001F4E301480001F4E401480001F4E501480001F4E601480001F4E70148

Chunk 2

0x0001F4E801480001F4E901480001F4EA01480001F4EB01480001F4EC01480001F4ED01480001F4EE01480001F4EF01480001F4F001480001F4F101480001F4F201480001F4F301480001F4F401480001F4F501480001F4F601480001F4F701480001F4F801480001F4F901480001F4FA01480001F4FB01480001F4FC01480001F4FD01480001F4FF01480001F50001480001F50101480001F50201480001F50301480001F50401480001F50501480001F50601480001F50701480001F50801480001F50901480001F50A01480001F50B01480001F50C01480001F50D01480001F50E01480001F50F01480001F51001480001F51101480001F51201480001F51301480001F51401480001F51501480001F51601480001F51701480001F51801480001F51901480001F51A01480001F51B01480001F51C01480001F51D01480001F51E01480001F51F01480001F52001480001F52101480001F52201480001F52301480001F52401480001F52501480001F52601480001F52701480001F52801480001F52901480001F52A01480001F52B01480001F52C01480001F52D01480001F52E01480001F52F01480001F53001480001F53101480001F53201480001F53301480001F53401480001F53501480001F53601480001F53701480001F53801480001F53901480001F53A01480001F53B01480001F53C01480001F53D01480001F54901480001F54A01480001F54B01480001F54C01480001F54D01480001F54E01480001F55001480001F55101480001F55201480001F55301480001F55401480001F55501480001F55601480001F55701480001F55801480001F55901480001F55A01480001F55B01480001F55C01480001F55D01480001F55E01480001F55F01480001F56001480001F56101480001F56201480001F56301480001F56401480001F56501480001F56601480001F56701480001F56F01480001F57001480001F57301480001F57601480001F57701480001F57801480001F57901480001F58701480001F58A01480001F58B01480001F58C01480001F58D01480001F5A401480001F5A501480001F5A801480001F5B101480001F5B201480001F5BC01480001F5C201480001F5C301480001F5C401480001F5D101480001F5D201480001F5D301480001F5DC01480001F5DD01480001F5DE01480001F5E101480001F5E301480001F5E801480001F5EF01480001F5F301480001F5FA01480001F5FB01480001F5FC01480001F5FD01480001F5FE01480001F5FF01480001F60001480001F60101480001F60201480001F60301480001F60401480001F60501480001F60601480001F60701480001F60801480001F60901480001F60A01480001F60B01480001F60C01480001F60D01480001F60E01480001F60F01480001F61001480001F61101480001F61201480001F61301480001F61401480001F61501480001F61601480001F61701480001F61801480001F61901480001F61A01480001F61B01480001F61C01480001F61D01480001F61E01480001F61F01480001F62001480001F62101480001F62201480001F62301480001F62401480001F62501480001F62601480001F62701480001F62801480001F62901480001F62A01480001F62B01480001F62C01480001F62D01480001F62F01480001F63001480001F63101480001F63201480001F63301480001F63401480001F63701480001F63801480001F63901480001F63A01480001F63B01480001F63C01480001F63D01480001F63E01480001F63F01480001F64001480001F64101480001F64201480001F64301480001F64401480001F64801480001F64901480001F64A01480001F68001480001F68101480001F68201480001F68301480001F68401480001F68501480001F68601480001F68701480001F68801480001F68901480001F68A01480001F68B01480001F68C01480001F68D01480001F68E01480001F68F01480001F69001480001F69101480001F69201480001F69301480001F69401480001F69501480001F69601480001F69701480001F69801480001F69901480001F69A01480001F69B01480001F69C01480001F69D01480001F69E01480001F69F01480001F6A001480001F6A101480001F6A201480001F6A401480001F6A501480001F6A601480001F6A701480001F6A801480001F6A901480001F6AA01480001F6AB01480001F6AC01480001F6AD01480001F6AE01480001F6AF01480001F6B001480001F6B101480001F6B201480001F6B301480001F6B701480001F6B801480001F6B901480001F6BA01480001F6BB01480001F6BC01480001F6BD01480001F6BE01480001F6BF01480001F6C101480001F6C201480001F6C301480001F6C401480001F6C501480001F6CB01480001F6CD01480001F6CE01480001F6CF01480001F6D001480001F6D101480001F6D201480001F6D501480001F6D601480001F6D701480001F6DD01480001F6DE01480001F6DF01480001F6E001480001F6E101480001F6E201480001F6E301480001F6E401480001F6E501480001F6E901480001F6EB01480001F6EC01480001F6F001480001F6F301480001F6F401480001F6F501480001F6F601480001F6F701480001F6F801480001F6F901480001F6FA01480001F6FB01480001F6FC01480001F7E001480001F7E101480001F7E201480001F7E301480001F7E401480001F7E501480001F7E601480001F7E701480001F7E801480001F7E901480001F7EA01480001F7EB01480001F7F001480001F90D01480001F90E01480001F91001480001F91101480001F91201480001F91301480001F91401480001F91501480001F91601480001F91701480001F92001480001F92101480001F92201480001F92301480001F92401480001F92501480001F92701480001F92801480001F92901480001F92A01480001F92B01480001F92C01480001F92D01480001F92E01480001F92F01480001F93A01480001F93F01480001F94001480001F94101480001F94201480001F94301480001F94401480001F94501480001F94701480001F94801480001F94901480001F94A01480001F94B01480001F94C01480001F94D01480001F94E01480001F94F01480001F95001480001F95101480001F95201480001F95301480001F95401480001F95501480001F95601480001F95701480001F95801480001F95901480001F95A01480001F95B01480001F95C01480001F95D01480001F95E01480001F95F01480001F96001480001F96101480001F96201480001F96301480001F96401480001F96501480001F96601480001F96701480001F96801480001F96901480001F96A01480001F96B01480001F96C01480001F96D01480001F96E01480001F96F01480001F97001480001F97101480001F97201480001F97301480001F97401480001F97501480001F97601480001F97801480001F97901480001F97A01480001F97B01480001F97C01480001F97D01480001F97E01480001F97F01480001F98001480001F98101480001F98201480001F98301480001F98401480001F98501480001F98601480001F98701480001F98801480001F98901480001F98A01480001F98B01480001F98C01480001F98D01480001F98E01480001F98F01480001F99001480001F99101480001F99201480001F99301480001F99401480001F99501480001F99601480001F99701480001F99801480001F99901480001F99A01480001F99B01480001F99C01480001F99D01480001F99E01480001F99F01480001F9A001480001F9A101480001F9A201480001F9A301480001F9A401480001F9A501480001F9A601480001F9A701480001F9A801480001F9A901480001F9AA01480001F9AB01480001F9AC01480001F9AD01480001F9AE01480001F9AF01480001F9B001480001F9B101480001F9B201480001F9B301480001F9B401480001F9B701480001F9BA01480001F9BC01480001F9BD01480001F9BE01480001F9BF01480001F9C001480001F9C101480001F9C201480001F9C301480001F9C401480001F9C501480001F9C601480001F9C701480001F9C801480001F9C901480001F9CA01480001F9CB01480001F9CC01480001F9D001480001F9E001480001F9E101480001F9E201480001F9E301480001F9E401480001F9E501480001F9E601480001F9E701480001F9E801480001F9E901480001F9EA01480001F9EB01480001F9EC01480001F9ED01480001F9EE01480001F9EF01480001F9F001480001F9F101480001F9F201480001F9F301480001F9F401480001F9F501480001F9F601480001F9F701480001F9F801480001F9F901480001F9FA01480001F9FB01480001F9FC01480001F9FD01480001F9FE01480001F9FF01480001FA7001480001FA7101480001FA7201480001FA7301480001FA7401480001FA7801480001FA7901480001FA7A01480001FA7B01480001FA7C01480001FA8001480001FA8101480001FA8201480001FA8301480001FA8401480001FA8501480001FA8601480001FA9001480001FA9101480001FA9201480001FA9301480001FA9401480001FA9501480001FA9601480001FA9701480001FA9801480001FA9901480001FA9A01480001FA9B01480001FA9C01480001FA9D01480001FA9E01480001FA9F01480001FAA001480001FAA101480001FAA201480001FAA301480001FAA401480001FAA501480001FAA601480001FAA701480001FAA801480001FAA901480001FAAA01480001FAAB01480001FAAC01480001FAB001480001FAB101480001FAB201480001FAB301480001FAB401480001FAB501480001FAB601480001FAB701480001FAB801480001FAB901480001FABA01480001FAC001480001FAC101480001FAC201480001FAD001480001FAD101480001FAD201480001FAD301480001FAD401480001FAD501480001FAD601480001FAD701480001FAD801480001FAD901480001FAE001480001FAE101480001FAE201480001FAE301480001FAE401480001FAE501480001FAE601480001FAE701480000261D01610000270A01610000270B01610000270C01610000270D01610001F38501610001F3C201610001F3C701610001F44201610001F44301610001F44601610001F44701610001F44801610001F44901610001F44A01610001F44B01610001F44C01610001F44D01610001F44E01610001F44F01610001F45001610001F46601610001F46701610001F46A01610001F46B01610001F46C01610001F46D01610001F47201610001F47401610001F47501610001F47601610001F47801610001F47C01610001F48301610001F48501610001F48F01610001F49101610001F4AA01610001F57401610001F57A01610001F59001610001F59501610001F59601610001F64C01610001F64F01610001F6C001610001F6CC01610001F90C01610001F90F01610001F91801610001F91901610001F91A01610001F91B01610001F91C01610001F91D01610001F91E01610001F91F01610001F93001610001F93101610001F93201610001F93301610001F93401610001F93601610001F97701610001F9B501610001F9B601610001F9BB01610001F9D201610001F9D301610001F9D501610001FAC301610001FAC401610001FAC501610001FAF001610001FAF201610001FAF301610001FAF401610001FAF501610001FAF601616101F3FB01486101F3FC01486101F3FD01486101F3FE01486101F3FF01480000002300620000002A0062000000300062000000310062000000320062000000330062000000340062000000350062000000360062000000370062000000380062000000390062620020E30203

The chunks are 6 byte rules for state transitions which follow this tree:

Here is a test contract that I’ve set the above contract as the emoji parser and a BasicValidator contract that implements /^[a-z09-_.]+/$ as the validator.

filter(string name, bool ignoreFEOF) should parse any UTF8 string with emojis.
validate(string name) should validate any normalized name which uses valid emoji or obeys the regex above. Note: This will fail if emoji have FE0F (which isn’t normalized). This should validate 94% of registered names as-of today.

nick.eth · June 28, 2022, 9:14am

Outstanding! Do you have an estimate of what it would take to extend this to match your entire normalisation function, not just Emoji?

raffy · June 28, 2022, 10:42am

It depends on how IDNA and NFC are implemented. If you implement them verbatim, it will be very expensive to deploy like royalfork estimated. However, there’s probably a partial mapping that would give a lot of bang for the buck.

I think there is a possibility of using NFKD → Apply the small number of differences from IDNA 2003 → NFC, but I’d need to verify that this is the same thing as IDNA 2003 → NFD → NFC. You could avoid a whole lookup table with this approach.

For IDNA, we can pack 10 uint24 per storage slot for single char mappings. There are runs of “ABCabc” and “AaBbCc” for casefolding, and runs of “ABC”, “A?B?C?”, and “A??B??C??” that map to a constant. There’s a bunch of ideas in my github.

If, for the moment, we reject combining marks and composite characters (so NFC isn’t needed) then my emoji parser + an efficient IDNA mapping would be a string → string implementation. All the effort would go into efficiently compressing the mapping.

Otherwise, I think the most important contract is an NFD/NFC implementation.

For validation:

You can naively encode the valid output character set into a bitmap for 700 storage slots. Combine that with the emoji contract above + NFC quick check and that’s a gas efficient function for determining if a name is in the normalized form. I can demo this immediately once I have an NFC implementation.

I also think a few separate simple validators for each script would give a lot of coverage (and more could be added over time). However, this approach requires a mechanism for looping through the set of validators to find one which returns true for your name.

I mentioned in the other thread, you could store the address of validator on-chain for the name you verified, making all future validations only two SLOADs, node → validator address → is valid validator address, but this requires extra infrastructure.

nick.eth · June 28, 2022, 12:43pm

This seems to be a reasonable starting point. Are there existing domains that use combining marks or composite characters?

raffy · June 29, 2022, 12:13pm

5K names have combined characters (NFC != NFD.)
171 names have two combining marks that could be reordered.

Codepoints: 2.4K CM and 13K composed.

Maybe that is a good idea, just compress the IDNA part and disallow the codepoints that need NFC. That’s 99% of names unless I’m forgetting something.

@adraffy/ens-norm-research has an improved contract that only needs 360 storage slots and no longer requires the skipFE0F option, and the code that generates the state payload.

Emoji + IDNA 2003 - (Combining Marks, Compositions) can be done with 10KB contract code, 128 slots for Valid bitmap, 450 Slots for Mappings of [1-2 CP], 175 Slots for Mappings [3+ CP], 360 slots for Emoji.

[128 + 452 + 175 + 360] → 1115 slots → ~30M + contract gas so under 1 eth to deploy.

122810 valid, 4208 mapped, 9 ignored, 3046 emoji sequences.

Since the IDNA table is stable, a huge chunk of the valid and single character mappings can be compiled directly to code.

I think worse-case per character would be, ~15 comparisons (x < A), 10 range comparisions (x >= A && x <= B), 1 sload for valid, 1 sload for small mapping, 1 sload for big mapping, and then calling the emoji function which is (1 sload per codepoint.)

Edit: Here is my first working version: https://rinkeby.etherscan.io/address/0xdb68eb6f0ab93bc7059f2752d477b45f04457eb2#readContract

Cost 32M gas to deploy and populate.

Should work for every name except those with combining marks or characters that compose (like à).

Edit 2: I deployed a more gas efficient version that avoids using uint24[]. It also has a beautify() function.

I’m seeing about 50-100K gas per name.

Neal · July 3, 2022, 2:28am

Although I have little idea what you’re on about half the time here, your commitment to see this all the way through is admirable, @raffy

raffy · July 6, 2022, 8:50am

I got a version with NFC Quick Check, which means only disordered combining marks or decomposed characters will incorrectly fail to normalize.

https://rinkeby.etherscan.io/address/0x335be342669ae015d7a87eb7c632447f9218254b
43M gas

🤼🏻‍♂️Ⅷ👩🏽‍⚕️_$🇦ß.۱۲۳👨‍👩‍👦9️⃣.ѐ🏴󠁧󠁢󠁥󠁮󠁧󠁿.eth

nick.eth · July 6, 2022, 3:15pm

How will they be handled? Do they throw an error?

Roughly how much gas does normalize cost?

raffy · July 6, 2022, 6:52pm

NFC Quick Check failures throw NotNFC().
Disallowed throw InvalidCodepoint(uint256 cp).

🤼🏻‍♂️Ⅷ👩🏽‍⚕️_$🇦ß.۱۲۳👨‍👩‍👦9️⃣.ѐ🏴󠁧󠁢󠁥󠁮󠁧󠁿.eth → 180K gas
💩💩💩.eth → 50K gas
RAFFY.ETH → 60K gas
maybe-its-time-to-stop-registering-garbage-like-this-and-buy-premiums-from-secondary-instead-such-as-3-or-4-digits-or-clean-one-word-domains-like-soy.eth → 280K gas

~~An ASCII fastpath in-front of the emoji checker would probably save a lot of gas.~~ ~~Only saves 1 sload.~~

Edit: Actually, once both emoji and valid are known, a fast path bitmap can be stored in a single uint256, that returns true for any valid character that isn’t an emoji prefix. At the moment, that’s true for 81 of 0-255. That would make DNS names much cheaper if we need this kind of optimization.

raffy.eth → 29K gas
maybe-its-time-to-stop-registering-garbage-like-this-and-buy-premiums-from-secondary-instead-such-as-3-or-4-digits-or-clean-one-word-domains-like-soy.eth → 111K gas (save 170K)

Another question is whether the contract ultimately should be adjustable. I’m not sure yet the best way to encode the large tree of codepoint ranges → codepoints (I’m using a ~700 line function.)

nick.eth · July 7, 2022, 8:46am

This sounds reasonable to me. Is there a reason we would need to implement the full NFC, in this context? Rejecting invalid normalisations seems sufficient to me.

Not bad! We shouldn’t generally have to do these in a transaction, but it’s nice to know it’d be viable if we really had to. Do you think there’s much room for gas optimisation here?

I’d be in favor of immutable contracts that can be replaced as needed; this tends to save gas anyway, and makes verificaiton much easier.

raffy · July 8, 2022, 8:21am

We should have full NFC for when someone inputs decomposed characters.

Now that QC works, I think I’ll just implement the algorithm as-is in my next release, since it will be invoked rarely.

Good to know.

nick.eth · July 8, 2022, 9:00am

Can we not simply reject such names as invalid?

royalfork · July 29, 2022, 10:34pm

We’ll have to define “cheap”, but I believe Unicode adds new characters once a year-ish, so if each deployment is >$1k (or operationally complex with multiple transactions), I’d think that easy upgradeability should be a design requirement (ie: new code points can be set to “valid” by a single transaction).

I’d say yes, for the same reason that you’d also reject labels which contain minimally-qualified emoji.

What do we think about the following top-level API?

/**
 * @notice Compute namehash of domain after ENS normalization and
 *         validation.  Reverts if domain contains disallowed 
 *         codepoints, disordered combining marks, or decomposed characters.
 * @param domain Domain to namehash.
 * @return normalized Normalized domain, to be displayed to end-users.
 * @return node Namehash of domain.
 * @return charsets Bit array of unicode character sets present in the domain.
 */
function namehash(string memory domain) public view returns (string memory normalized, bytes32 node, uint256 charsets) {

// Unicode charsets
var (
	ASCII = 1 << iota
	Latin-1
	Greek
	Cryllic
	Emoji
	...
)

This includes a charsets return value, which tells the caller which charsets are present in the domain. I think this would make ENS much safer and inclusive for international users. For example, English sites could “flag as suspicious” any domain where charsets != ASCII & Emoji, and Chinese sites could flag charsets != Chinese & ASCII, etc.

Open Questions:

This function doesn’t really play well with the existing “UniversalResolver”. Should this function operate only on labels? Or should it have tighter integration with Universal Resolver? Either way, I think users should have an on-chain function which provides direct string -> (...helpful ENS info...) functionality.
I’m not wild about the namehash function name, since this returns more than just the namehash. Any suggestions for a better name?

@raffy That is some really impressive code you have there! Very well done

raffy · July 30, 2022, 1:01am

Here are some thoughts:

Assume max label length fits in 16 bits (or w/e). The low level normalize should be string → (string norm, string normNoEmoji, uint256 labelData) where:

normNoEmoji is the same string but each emoji sequence is zero’d
eg. (4 bytes) → [0,0,0,0]
each labelData is:
- 16 bit → label length
- 240 bits → bitset of active non-emoji codepoints shifted by 14
  (2^21/240 => 14 bits)

From that, it’s easy to compute the namehash, extract out any label, compute any label hash, or quickly determine which validation could apply. eg. the basic validator (DNS + Emoji) only runs if the bitset is 0x1. normNoEmoji avoids processing emoji again during validation.

The primary normalize function should be string → (string, hash) like you describe.

I think all the charset stuff should exist in the validator, whether that’s the same contract or a different contract, I’m not sure. Most validator checks are per-label, so given (label, bitset), you can efficiently check if its valid. IIRC, only check bidi requires a full-name check.

The low level validation function would be (string, bitset) → bool like:

function validate(string label, uint256 bitset) returns (bool) {
     // any 0 byte is a previously processed emoji
     if (bitset == 1) { // only codepoints [0, 0x4000)
         // check if basic
         // check if latin non-confusable
         // check if greek w/o wholescript confusable
         // etc...
     }
     return false;
}

Or more cleanly, the validation contract could have arbitrary functions key’d by bitset + nonce, so you could register/unregister any number of validation functions, and quickly check which ones could apply by intersecting the label bitset, etc.

The primary normalize + validation function would be string → (string norm, uint256 hash, bool valid) where:

given name
(norm, normNoEmoji, labelData) = normalize(name)
start = 0
hash = 0
valid = true
// possibly apply full-name check
for [len, bitset] of labelData.reverse()
    hash = keccak(hash + keccak(norm.slice(start, len)))
    valid &= validate(normNoEmoji.slice(start, len), bitset)
    start += 1 + len
return (norm, hash, valid)

If this throws, the name is invalid
If valid is false, the user should get some kind of warning that the name is potentially unsafe (where unsafe means one or more labels satisfied 0 approved validators.)
If valid isn’t needed, use the primary normalize function instead.

royalfork · July 30, 2022, 3:39am

I haven’t thought much about what the internal API should be, but this certainly looks reasonable. Just to clarify, is the motivation behind the emoji special case (norm vs normNoEmoji) to sort out the 0xfe0f issue? (ie: 0xfe0f must be present for “input/display normalization”, but be absent for namehash calculation?)

Would this also be part of the internal API? I don’t thing the nuance around valid should be exposed to end-users.

Why wouldn’t you also throw in this case?

I think the first step is to build general consensus around a top-level public API to be used by ecosystem developers. Once we have that, I’m pretty confident we can work backwards to nail down the low-level internals.

raffy · July 30, 2022, 4:13am

Validation only needs to check if the non-emoji part of the string obeys all the Unicode rules. But you need to run the full emoji logic to skip over them, so each validator would then need efficient access to the emoji logic. Whereas, allocating a little more memory and keeping a second copy with zero’d emoji bytes means each validators can just skip over 0’s and be fully ignorant of emoji.

This lets you construct the basic validator /[a-z0-9_-]+/ (and many others) very easily.

This is my opinion but I think normalizable-but-confusing names should still work. Especially for the headless situation, eg. aрe.eth confusable can normalize and hash, but valid would be false.

There exist creative/vanity names that will never satisfy even the most relaxed script-based rules simply because they use a single-character of a script that’s confusable in other contexts.

royalfork · July 31, 2022, 2:41am

Validity depends on context. Within an English-speaking context, aрe.eth should be flagged, but within a Russian-speaking context, something like овечкин8.eth shouldn’t be flagged. Since they both contain ASCII+Cryllic, I don’t think a simple yes/no validity boolean is enough to capture this distinction.

I don’t think we should dictate what combinations of character sets are or aren’t valid. Instead, we should provide an easy way for ecosystem developers to know what’s actually inside an input domain, and provide simple building blocks to construct custom validity rules/guidelines. I believe the (uint256 charsets) return value provides this.

Taking your example, the English-speaking ecosystem developer would only need to write valid = (charset == ASCII) to flag aрe.eth as invalid.

cthulu.eth · September 10, 2022, 2:27pm

I just wanted to check in and see if on-chain normalization is being worked on, or if there are any status updates as the last reply to this thread was in July.

raffy · September 10, 2022, 11:49pm

I wrote a Solidity Unicode NF implementation so we have all of the parts necessary.

I just need to integrate it into the previous contract and upload the payloads according to the latest spec and we should be 100% match.

Edit: This is really hacky and untested but appears to work just by gluing some pieces together. It follows the latest ENSIP rules. It has underscore, label extension (double hyphen), and combining mark (leading, emoji, adjacent) logic. Beautifier works too (missing the regional indicator separator logic.)
https://rinkeby.etherscan.io/address/0x1E321B0fdbbc022D959F5D9eB829f071cC375131#readContract

normalize() gives array of labels (unfortunately if you copy from etherscan periods will be commas)
normhash() goes straight from unnormalized to namehash

Deployment gas was 70M, up from 43M. Needs optimization but “RAFFY.eth” was under 70K gas.

There’s a question of what the API should be?

cthulu.eth · September 12, 2022, 12:44pm

That’s absolutely beautiful @raffy, thanks for the update and for your hard work!